justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 265279.86@dunegpschedd01.fnal.gov

Jobsub ID265279.86@dunegpschedd01.fnal.gov
Workflow ID11116
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-09 10:51:45
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-12-10 09:21:21
From worker nodeHostnamedunegli-48793-0-cmswn4069.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-09 16:47:56
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T205358Z_gen_003388_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time16h (59200s)
CPU time16h (57817s = 97%)
Max RSS bytes2635284480 (2513 MiB)
Outputting started2025-12-10 09:14:37
Output files
Finished2025-12-10 09:21:21
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

54, 1653
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.113818, 
Output 1: 0.296053, 0.107859, 0.21574, 0.380348, 
Output 2: 0.0518153, 0.135062, 0.798061, 0.0150621, 
Output 3: 0.941984, 0.0518611, 0.00272593, 0.00342958, 
Output 4: 0.501108, 0.454782, 0.0388897, 0.00522037, 
Output 5: 0.335672, 0.465927, 0.151866, 0.0465351, 
Output 6: 0.110963, 0.197478, 0.130706, 0.560852, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 33880 at 10-Dec-2025 08:51:27 UTC
Boundary wire vector sizes: 6629, 6661, 6556
minwire 0: 54
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.114585, 
Output 1: 0.0718737, 0.04999, 0.101718, 0.776418, 
Output 2: 0.103511, 0.267152, 0.576465, 0.0528729, 
Output 3: 0.975186, 0.0141819, 0.00202865, 0.00860351, 
Output 4: 0.426571, 0.528819, 0.041358, 0.00325212, 
Output 5: 0.764133, 0.191963, 0.0325739, 0.0113299, 
Output 6: 0.0198121, 0.0426269, 0.0567482, 0.880813, 

10-Dec-2025 09:14:09 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 09:14:09 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/1f/81/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T205358Z_gen_003388_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1358.94       5870.47       8764.71       6994.74       2153.32        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00992578     0.0250196     0.0489792     0.0232294     0.0120421       10     
reco:pandora:StandardPandora                            1357.91       5810.7        8676.27       6924.41       2127.81        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00177408     0.152503      0.689288      0.109893      0.190746        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00115256     0.0151431     0.0685953    0.00793487     0.0190885       10     
reco:pandoracalo:Calorimetry                          0.000501293    0.0243815     0.196848      0.0027647     0.0577536       10     
reco:pandorapid:Chi2ParticleID                        7.6275e-05     0.0014284    0.00722894    0.000449689   0.00217459       10     
reco:cvnmap:CVNMapper                                  0.0201927       0.196       0.582298      0.148863      0.152307        10     
reco:cvneva:CVNEvaluator                               0.686946       3.62721       18.7292       2.30303       5.08634        10     
reco:energyrecnumu:EnergyReco                         0.00997145      13.3377       66.6557      0.0286416      26.6185        10     
reco:energyrecnue:EnergyReco                          0.00130048     0.0104996     0.0394539    0.00503029     0.0122496       10     
reco:energyrecnc:EnergyReco                           0.00479202      41.5374       74.3317       63.8812       34.0181        10     
[art]:TriggerResults:TriggerResultInserter            1.4227e-05    0.000170903   0.000558321   0.000142486   0.00014028       10     
end_path:out1:RootOutput                               7.444e-06    5.40999e-05   0.000198387   4.1274e-05    5.08771e-05      10     
end_path:out1:RootOutput(write)                        0.221059      0.837195       1.75939      0.789666      0.490287        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21683.3 MB
  Peak resident set size usage (VmHWM): 2635.28 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 77.8093 | hit correction : 1.07131
hit initial : 459.688 | hit correction : 1.06993
hit initial : 449.299 | hit correction : 1.06905
hit initial : 434.646 | hit correction : 1.06823
hit initial : 166.607 | hit correction : 1.07268
 *** Total Charge : 3226.74
 Looking at view: 0 | Number of hits in view: 10
hit initial : 40.7227 | hit correction : 1.07269
hit initial : 151.907 | hit correction : 1.07268
hit initial : 212.429 | hit correction : 1.0727
hit initial : 199.42 | hit correction : 1.0727
hit initial : 189.202 | hit correction : 1.07258
 *** Total Charge : 851.355
 Looking at view: 1 | Number of hits in view: 5
hit initial : 204.064 | hit correction : 1.07273
hit initial : 184.944 | hit correction : 1.07244
hit initial : 224.377 | hit correction : 1.07268
hit initial : 236.166 | hit correction : 1.07266
hit initial : 91.7765 | hit correction : 1.07268
 *** Total Charge : 1009.71
 Looking at view: 2 | Number of hits in view: 5
Begin processing the 9th record. run: 5026 subRun: 0 event: 33879 at 10-Dec-2025 09:14:33 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  13
Added hit list length: 13 | Max was set to: -3.40282e+38
This pfp has size:  34
Added hit list length: 34 | Max was set to: 13
This pfp has size:  8
True Neutron Energy: 941.122
True Electron Energy: 35.8141
Completed MCParticle Loop
True Neutrino Energy: 50.3261
 Size of all hits: 34
hit initial : 83.3336 | hit correction : 1.02296
hit initial : 138.073 | hit correction : 1.02171
hit initial : 328.027 | hit correction : 1.02294
hit initial : 136.41 | hit correction : 1.02294
hit initial : 56.5192 | hit correction : 1.022
hit initial : 47.5296 | hit correction : 1.02159
hit initial : 17.9753 | hit correction : 1.02159
 *** Total Charge : 826.09
 Looking at view: 0 | Number of hits in view: 7
hit initial : 53.2269 | hit correction : 1.02262
hit initial : 31.2747 | hit correction : 1.0226
hit initial : 68.8279 | hit correction : 1.02207
hit initial : 343.558 | hit correction : 1.02209
hit initial : 469.528 | hit correction : 1.02209
hit initial : 94.7172 | hit correction : 1.02219
hit initial : 177.994 | hit correction : 1.02292
hit initial : 81.9664 | hit correction : 1.02066
hit initial : 52.0934 | hit correction : 1.01981
hit initial : 117.666 | hit correction : 1.01902
hit initial : 320.928 | hit correction : 1.02294
hit initial : 34.8278 | hit correction : 1.02104
hit initial : 18.6587 | hit correction : 1.02058
hit initial : 19.6054 | hit correction : 1.02004
hit initial : 126.727 | hit correction : 1.01885
hit initial : 85.5511 | hit correction : 1.02299
hit initial : 23.6179 | hit correction : 1.02223
hit initial : 122.019 | hit correction : 1.02158
hit initial : 115.329 | hit correction : 1.01847
hit initial : 21.3515 | hit correction : 1.02219
hit initial : 31.0424 | hit correction : 1.01841
hit initial : 54.9921 | hit correction : 1.0216
hit initial : 30.6029 | hit correction : 1.02158
hit initial : 72.9943 | hit correction : 1.02162
 *** Total Charge : 2624.63
 Looking at view: 1 | Number of hits in view: 24
hit initial : 268.192 | hit correction : 1.02284
hit initial : 412.494 | hit correction : 1.02298
hit initial : 269.705 | hit correction : 1.02229
 *** Total Charge : 972.006
 Looking at view: 2 | Number of hits in view: 3
Begin processing the 10th record. run: 5026 subRun: 0 event: 33880 at 10-Dec-2025 09:14:35 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.569
True Electron Energy: 8.89144
Completed MCParticle Loop
True Neutrino Energy: 12.178
 Size of all hits: 0
10-Dec-2025 09:14:37 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0456277     0.0575946     0.0672474     0.0584806     0.0066358       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000347771   0.000489248   0.000938894   0.000418565   0.000173477      10     
end_path:ana:AnalyzeEvents         0.0450945     0.0569331     0.0665144     0.0579303    0.00655548       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19474.7 MB
  Peak resident set size usage (VmHWM): 1526.71 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 23:27:05 UTC       justIN version: 01.05.03