Jobsub ID 292861.7@dunegpschedd02.fnal.gov
| Jobsub ID | 292861.7@dunegpschedd02.fnal.gov |
| Workflow ID | 12170 |
| Stage ID | 1 |
| User name | lwhite86@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 3600 (1 hours) |
| Submitted time | 2026-01-21 15:04:52 |
| Site | NL_SURFsara |
| Entry | DUNE_SurfSARA_arc03 |
| Last heartbeat | 2026-01-21 15:12:11 |
| From worker node | Hostname | wn-da-10.gina.surf.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | finished |
| Started | 2026-01-21 15:05:21 |
| Input files | fardet-hd:nu_dune10kt_1x2x6_1109_539_20230827T003332Z_gen_g4_detsim_hitreco__20240109T225720Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 6m (381s) |
| CPU time | 5m (357s = 93%) |
| Max RSS bytes | 1539551232 (1468 MiB) |
| Outputting started | 2026-01-21 15:11:43 |
| Output files | |
| Finished | 2026-01-21 15:12:11 |
| Saved logs | justin-logs:292861.7-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
event: 53974 at 21-Jan-2026 16:10:31 CET
TrackPID: 2212, 0.000161823, 0.00351197, 0.995722, 0.000603775
TrackPID: 13, 0.995997, 0.00169052, 0.00220505, 0.00010719
Begin processing the 75th record. run: 1109 subRun: 1 event: 53975 at 21-Jan-2026 16:10:33 CET
TrackPID: 13, 0.938924, 0.0385178, 0.0213671, 0.00119143
TrackPID: -13, 0.999901, 7.64786e-05, 1.66456e-05, 5.65802e-06
Begin processing the 76th record. run: 1109 subRun: 1 event: 53976 at 21-Jan-2026 16:10:36 CET
TrackPID: 2212, 0.000241066, 0.00327308, 0.995671, 0.000814456
TrackPID: 13, 0.223317, 0.675279, 0.0954861, 0.00591822
Begin processing the 77th record. run: 1109 subRun: 1 event: 53977 at 21-Jan-2026 16:10:38 CET
TrackPID: 22, 0.0347135, 0.854195, 0.0999561, 0.0111349
TrackPID: -211, 0.031196, 0.856338, 0.0939525, 0.018514
TrackPID: 2212, 0.000228527, 0.00346436, 0.99525, 0.00105703
Begin processing the 78th record. run: 1109 subRun: 1 event: 53978 at 21-Jan-2026 16:10:41 CET
Begin processing the 79th record. run: 1109 subRun: 1 event: 53979 at 21-Jan-2026 16:10:43 CET
TrackPID: 2212, 0.000344306, 0.0020309, 0.996915, 0.000710065
Begin processing the 80th record. run: 1109 subRun: 1 event: 53980 at 21-Jan-2026 16:10:45 CET
TrackPID: 13, 0.172991, 0.693777, 0.129834, 0.00339846
TrackPID: 211, 0.558842, 0.394392, 0.0377185, 0.00904765
TrackPID: 2212, 0.000347717, 0.00176396, 0.997025, 0.000862967
TrackPID: 13, 0.927948, 0.0607063, 0.0100969, 0.00124845
TrackPID: -211, 0.0150734, 0.197323, 0.781259, 0.00634514
TrackPID: 211, 0.0279833, 0.355875, 0.609215, 0.00692664
Begin processing the 81st record. run: 1109 subRun: 1 event: 53981 at 21-Jan-2026 16:10:50 CET
TrackPID: 2212, 0.000152934, 0.000972962, 0.998412, 0.000461954
TrackPID: 13, 0.970452, 0.0240649, 0.004715, 0.00076748
Begin processing the 82nd record. run: 1109 subRun: 1 event: 53982 at 21-Jan-2026 16:10:52 CET
TrackPID: 2212, 0.000360203, 0.00806347, 0.99037, 0.00120615
TrackPID: 211, 0.0983125, 0.692295, 0.184959, 0.0244331
TrackPID: 2212, 0.0528139, 0.763026, 0.169498, 0.0146621
TrackPID: 2212, 0.000108501, 0.00448087, 0.994787, 0.000623292
TrackPID: 2212, 0.711021, 0.221776, 0.0587553, 0.00844788
TrackPID: 2212, 0.000759588, 0.012734, 0.985597, 0.000909296
TrackPID: 2212, 0.000432326, 0.00669821, 0.99172, 0.00114929
TrackPID: 13, 0.195646, 0.36864, 0.42704, 0.00867312
TrackPID: 211, 0.0108496, 0.176543, 0.807907, 0.00470018
Begin processing the 83rd record. run: 1109 subRun: 1 event: 53983 at 21-Jan-2026 16:10:58 CET
TrackPID: 211, 0.0945605, 0.803627, 0.0981149, 0.00369781
TrackPID: 2212, 0.00049334, 0.0556603, 0.939119, 0.00472773
TrackPID: 211, 0.0474123, 0.747411, 0.196275, 0.00890156
TrackPID: 2212, 0.0118158, 0.227301, 0.757582, 0.00330164
TrackPID: 13, 0.96049, 0.0222572, 0.0166386, 0.000614423
Begin processing the 84th record. run: 1109 subRun: 1 event: 53984 at 21-Jan-2026 16:11:00 CET
TrackPID: 211, 0.134667, 0.701351, 0.157996, 0.00598523
TrackPID: -211, 0.108096, 0.706403, 0.162126, 0.0233747
TrackPID: 211, 0.000469313, 0.00918082, 0.988472, 0.00187841
TrackPID: 2212, 0.000255608, 0.00223023, 0.996709, 0.000805378
TrackPID: 2212, 0.000182378, 0.00146843, 0.997804, 0.000545604
Begin processing the 85th record. run: 1109 subRun: 1 event: 53985 at 21-Jan-2026 16:11:03 CET
Begin processing the 86th record. run: 1109 subRun: 1 event: 53986 at 21-Jan-2026 16:11:05 CET
TrackPID: 13, 0.371117, 0.502324, 0.119488, 0.00707154
TrackPID: 2212, 0.000413857, 0.00521033, 0.992819, 0.0015564
Begin processing the 87th record. run: 1109 subRun: 1 event: 53987 at 21-Jan-2026 16:11:07 CET
TrackPID: 13, 0.972473, 0.0192365, 0.00772743, 0.000562848
Begin processing the 88th record. run: 1109 subRun: 1 event: 53988 at 21-Jan-2026 16:11:10 CET
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
in function: GetVolumeIdToHitListMap
in file: /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
in function: Run
in file: /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 89th record. run: 1109 subRun: 1 event: 53989 at 21-Jan-2026 16:11:10 CET
TrackPID: 2212, 0.00024383, 0.00177525, 0.997375, 0.000605496
TrackPID: 13, 0.997634, 0.00153739, 0.000771959, 5.63155e-05
TrackPID: 211, 0.0249428, 0.682355, 0.281216, 0.0114861
Begin processing the 90th record. run: 1109 subRun: 1 event: 53990 at 21-Jan-2026 16:11:12 CET
TrackPID: 13, 0.99726, 0.00204396, 0.000637837, 5.86655e-05
TrackPID: 2212, 0.000965867, 0.0027448, 0.995119, 0.00117054
Begin processing the 91st record. run: 1109 subRun: 1 event: 53991 at 21-Jan-2026 16:11:15 CET
TrackPID: 2212, 0.000184057, 0.00150919, 0.997804, 0.000502567
TrackPID: 13, 0.386457, 0.455007, 0.149134, 0.00940257
TrackPID: 22, 0.152162, 0.666295, 0.176495, 0.00504777
TrackPID: 22, 0.147075, 0.686469, 0.159717, 0.00673925
TrackPID: -211, 0.29768, 0.540869, 0.158514, 0.00293662
Begin processing the 92nd record. run: 1109 subRun: 1 event: 53992 at 21-Jan-2026 16:11:18 CET
TrackPID: 211, 0.482946, 0.366212, 0.142729, 0.0081129
Begin processing the 93rd record. run: 1109 subRun: 1 event: 53993 at 21-Jan-2026 16:11:20 CET
TrackPID: 13, 0.128715, 0.413465, 0.439325, 0.018495
TrackPID: 22, 0.0417074, 0.384078, 0.567252, 0.00696317
TrackPID: 22, 0.15808, 0.63848, 0.19764, 0.00579967
Begin processing the 94th record. run: 1109 subRun: 1 event: 53994 at 21-Jan-2026 16:11:23 CET
Begin processing the 95th record. run: 1109 subRun: 1 event: 53995 at 21-Jan-2026 16:11:25 CET
TrackPID: 13, 0.999032, 0.000684028, 0.000257904, 2.61892e-05
Begin processing the 96th record. run: 1109 subRun: 1 event: 53996 at 21-Jan-2026 16:11:27 CET
Begin processing the 97th record. run: 1109 subRun: 1 event: 53997 at 21-Jan-2026 16:11:29 CET
TrackPID: 2212, 0.116829, 0.451506, 0.427253, 0.00441147
TrackPID: 2212, 0.000252014, 0.00243923, 0.996497, 0.000811976
TrackPID: -211, 0.112308, 0.480045, 0.399128, 0.00851868
Begin processing the 98th record. run: 1109 subRun: 1 event: 53998 at 21-Jan-2026 16:11:32 CET
TrackPID: 2212, 0.000203139, 0.00305551, 0.995908, 0.000833836
Begin processing the 99th record. run: 1109 subRun: 1 event: 53999 at 21-Jan-2026 16:11:34 CET
TrackPID: -211, 0.166124, 0.666032, 0.161293, 0.00655137
TrackPID: 211, 0.00556535, 0.143708, 0.846556, 0.00417091
Begin processing the 100th record. run: 1109 subRun: 1 event: 54000 at 21-Jan-2026 16:11:36 CET
TrackPID: 2212, 0.000297459, 0.00359418, 0.995106, 0.00100275
TrackPID: 2212, 0.000140191, 0.00156253, 0.997671, 0.000626472
TrackPID: 13, 0.895015, 0.0860186, 0.0174861, 0.00148074
TrackPID: 2212, 0.000938759, 0.00679904, 0.990956, 0.00130586
21-Jan-2026 16:11:40 CET Closed output file "nu_dune10kt_1x2x6_1109_539_20230827T003332Z_gen_g4_detsim_hitreco__20240109T225720Z_reco2_reco2.root"
21-Jan-2026 16:11:40 CET Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/8a/b7/nu_dune10kt_1x2x6_1109_539_20230827T003332Z_gen_g4_detsim_hitreco__20240109T225720Z_reco2.root"
================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 0.0415786 2.97401 18.0299 2.36661 2.34325 100
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00543565 0.0316046 0.085992 0.0227312 0.0225748 100
reco:pandora2:StandardPandora 0.0198008 2.93993 18.0114 2.34086 2.34352 100
[art]:TriggerResults:TriggerResultInserter 1.3796e-05 4.79449e-05 0.000302196 3.687e-05 3.60172e-05 100
end_path:out1:RootOutput 4.589e-06 7.99811e-06 3.2091e-05 6.958e-06 3.73527e-06 100
end_path:out1:RootOutput(write) 0.000374353 0.00208135 0.0247398 0.000697883 0.00360886 100
================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 2519.58 MB
Peak resident set size usage (VmHWM): 1539.55 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 424
-rw-r--r--. 1 dune009 dune 210 Jan 21 16:05 all-input-dids.txt
-rw-r--r--. 1 dune009 dune 0 Jan 21 16:05 debugprod.log
-rw-r--r--. 1 dune009 dune 59497 Jan 21 16:11 jobscript.log
-rw-r--r--. 1 dune009 dune 186 Jan 21 16:11 justin-processed-pfns.txt
drwxr-xr-x. 4 dune009 dune 60 Jan 21 16:05 larpandoracontent
-rw-r--r--. 1 dune009 dune 359114 Jan 21 16:11 nu_dune10kt_1x2x6_1109_539_20230827T003332Z_gen_g4_detsim_hitreco__20240109T225720Z_reco2_reco2.root
-rw-r--r--. 1 dune009 dune 519 Jan 21 16:11 reco2_hist.root 