justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 265279.88@dunegpschedd01.fnal.gov

Jobsub ID265279.88@dunegpschedd01.fnal.gov
Workflow ID11116
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-09 10:51:45
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-12-10 09:20:16
From worker nodeHostnamedunegli-48793-0-cmswn4069.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-09 16:47:38
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T235225Z_gen_001833_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time16h (59153s)
CPU time16h (58353s = 98%)
Max RSS bytes2554880000 (2436 MiB)
Outputting started2025-12-10 09:13:31
Output files
Finished2025-12-10 09:20:16
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

4496, 0.129089, 0.0383169, 
Output 6: 0.0521135, 0.0989725, 0.0982473, 0.750667, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 18330 at 10-Dec-2025 08:52:41 UTC
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  10-Dec-2025 09:13:00 UTC run: 5027 subRun: 0 event: 18330
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6889, 6915, 6837
minwire 0: 9
minwire 1: 1
minwire 2: 4
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0897193, 
Output 1: 0.0880767, 0.0977305, 0.141172, 0.67302, 
Output 2: 0.0368528, 0.157654, 0.791117, 0.014376, 
Output 3: 0.937187, 0.0513018, 0.00558023, 0.00593105, 
Output 4: 0.553346, 0.361793, 0.0718166, 0.013044, 
Output 5: 0.276047, 0.525505, 0.170264, 0.0281835, 
Output 6: 0.0786224, 0.159814, 0.123219, 0.638345, 

10-Dec-2025 09:13:01 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 09:13:01 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/bf/91/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T235225Z_gen_001833_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1216.72       5860.4        8658.18       6617.09       2149.35        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0133415     0.0294586     0.0699499     0.0254353     0.0152412       10     
reco:pandora:StandardPandora                            1215.64       5855.37       8638.15       6613.02       2146.12        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00107044     0.0793699     0.536673       0.01587      0.156315        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000609999    0.010202      0.0276131    0.00921885    0.00900849       10     
reco:pandoracalo:Calorimetry                          0.000240036    0.0113955     0.0631601    0.00180495     0.019277        10     
reco:pandorapid:Chi2ParticleID                        8.1766e-05    0.000958256   0.00405357    0.000234245   0.00125839       10     
reco:cvnmap:CVNMapper                                  0.0339989      0.17637      0.451028      0.135314      0.114279        10     
reco:cvneva:CVNEvaluator                               0.701376       3.54809       16.8982       2.59847       4.50651        10     
reco:energyrecnumu:EnergyReco                         0.00334271     0.136808      0.515266      0.023979      0.165237        10     
reco:energyrecnue:EnergyReco                          0.000810319    0.0104183     0.0305462    0.00677811    0.00919649       10     
reco:energyrecnc:EnergyReco                            0.0170216     0.103086      0.208923      0.082901      0.0751739       10     
[art]:TriggerResults:TriggerResultInserter            1.8665e-05    0.000171036   0.000521771   0.000151523   0.00013928       10     
end_path:out1:RootOutput                               5.75e-06     7.39125e-05   0.000260636   4.91235e-05   7.36834e-05      10     
end_path:out1:RootOutput(write)                        0.218096       0.91565       1.56122      0.847443      0.466068        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21677.9 MB
  Peak resident set size usage (VmHWM): 2554.88 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 48.4396 | hit correction : 1.41843
hit initial : 59.0778 | hit correction : 1.41848
hit initial : 73.6879 | hit correction : 1.42086
hit initial : 98.3902 | hit correction : 1.42446
hit initial : 203.224 | hit correction : 1.42452
hit initial : 95.8111 | hit correction : 1.42456
hit initial : 48.0502 | hit correction : 1.42269
hit initial : 140.739 | hit correction : 1.42258
hit initial : 147.253 | hit correction : 1.42224
hit initial : 32.9029 | hit correction : 1.42175
hit initial : 152.96 | hit correction : 1.42133
hit initial : 139.782 | hit correction : 1.42084
hit initial : 123.324 | hit correction : 1.41846
 *** Total Charge : 1939.07
 Looking at view: 1 | Number of hits in view: 13
hit initial : 170.16 | hit correction : 1.42428
hit initial : 148.985 | hit correction : 1.42471
hit initial : 70.6571 | hit correction : 1.42251
hit initial : 53.7149 | hit correction : 1.42493
hit initial : 38.5837 | hit correction : 1.42253
hit initial : 257.752 | hit correction : 1.42256
hit initial : 65.9755 | hit correction : 1.42244
hit initial : 148.043 | hit correction : 1.42152
hit initial : 264.343 | hit correction : 1.42091
hit initial : 194.234 | hit correction : 1.41848
hit initial : 51.0317 | hit correction : 1.41839
 *** Total Charge : 2081.02
 Looking at view: 2 | Number of hits in view: 11
Begin processing the 10th record. run: 5027 subRun: 0 event: 18330 at 10-Dec-2025 09:13:29 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  40
Added hit list length: 40 | Max was set to: -3.40282e+38
True Neutron Energy: 939.568
True Electron Energy: 58.5772
Completed MCParticle Loop
True Neutrino Energy: 62.049
 Size of all hits: 40
hit initial : 66.1272 | hit correction : 1.2742
hit initial : 99.6184 | hit correction : 1.27762
hit initial : 33.5059 | hit correction : 1.27692
hit initial : 30.6293 | hit correction : 1.27615
hit initial : 21.3542 | hit correction : 1.275
hit initial : 532.886 | hit correction : 1.27761
hit initial : 201.96 | hit correction : 1.27544
hit initial : 122.867 | hit correction : 1.27855
hit initial : 242.311 | hit correction : 1.27797
hit initial : 176.798 | hit correction : 1.27523
hit initial : 50.5624 | hit correction : 1.27801
hit initial : 106.181 | hit correction : 1.27524
hit initial : 281.124 | hit correction : 1.28038
hit initial : 156.074 | hit correction : 1.28051
hit initial : 104.246 | hit correction : 1.27928
hit initial : 556.099 | hit correction : 1.27936
hit initial : 165.834 | hit correction : 1.2794
 *** Total Charge : 3768.06
 Looking at view: 0 | Number of hits in view: 17
hit initial : 154.38 | hit correction : 1.28058
hit initial : 294.164 | hit correction : 1.2803
hit initial : 356.328 | hit correction : 1.27967
hit initial : 369.555 | hit correction : 1.27922
hit initial : 420.929 | hit correction : 1.27851
hit initial : 38.3109 | hit correction : 1.27768
hit initial : 487.539 | hit correction : 1.27793
hit initial : 41.5554 | hit correction : 1.27528
hit initial : 94.3001 | hit correction : 1.27781
hit initial : 152.896 | hit correction : 1.27708
hit initial : 118.492 | hit correction : 1.2753
hit initial : 92.3925 | hit correction : 1.27693
hit initial : 82.5104 | hit correction : 1.27609
hit initial : 123.846 | hit correction : 1.27538
hit initial : 29.8362 | hit correction : 1.27565
hit initial : 43.8346 | hit correction : 1.27489
hit initial : 66.9604 | hit correction : 1.27494
hit initial : 57.2959 | hit correction : 1.27273
hit initial : 160.721 | hit correction : 1.27271
hit initial : 40.7646 | hit correction : 1.2742
hit initial : 110.791 | hit correction : 1.2727
hit initial : 127.194 | hit correction : 1.27419
hit initial : 106.979 | hit correction : 1.27418
 *** Total Charge : 4562.39
 Looking at view: 1 | Number of hits in view: 23
10-Dec-2025 09:13:31 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0456615     0.0552389     0.0640149     0.0571918    0.00527812       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000338203   0.000496995   0.00104043    0.000422112   0.000203735      10     
end_path:ana:AnalyzeEvents         0.0451882     0.0545691     0.0634011     0.0567177    0.00533207       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19478 MB
  Peak resident set size usage (VmHWM): 1530.43 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 23:18:04 UTC       justIN version: 01.05.03