Jobsub ID 292861.12@dunegpschedd02.fnal.gov
| Jobsub ID | 292861.12@dunegpschedd02.fnal.gov |
| Workflow ID | 12170 |
| Stage ID | 1 |
| User name | lwhite86@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 3600 (1 hours) |
| Submitted time | 2026-01-21 15:04:52 |
| Site | UK_QMUL |
| Entry | DUNE_UK_London_QMUL_arcce03 |
| Last heartbeat | 2026-01-21 15:14:00 |
| From worker node | Hostname | cn019.htc.esc.qmul |
| cpuinfo | Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | finished |
| Started | 2026-01-21 15:05:39 |
| Input files | fardet-hd:nu_dune10kt_1x2x6_1062_281_20230823T120818Z_gen_g4_detsim_hitreco__20240113T054159Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 5m (343s) |
| CPU time | 5m (305s = 88%) |
| Max RSS bytes | 1547472896 (1475 MiB) |
| Outputting started | 2026-01-21 15:11:24 |
| Output files | |
| Finished | 2026-01-21 15:14:00 |
| Saved logs | justin-logs:292861.12-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
68, 0.155488, 0.0557783, 0.00546538
Begin processing the 72nd record. run: 1062 subRun: 1 event: 28172 at 21-Jan-2026 15:10:15 GMT
TrackPID: 2212, 0.0211037, 0.386732, 0.586439, 0.00572504
TrackPID: 13, 0.999089, 0.000645063, 0.000216903, 4.93693e-05
Begin processing the 73rd record. run: 1062 subRun: 1 event: 28173 at 21-Jan-2026 15:10:18 GMT
TrackPID: 13, 0.998514, 0.0011733, 0.000277758, 3.5094e-05
TrackPID: 2212, 0.00020344, 0.00757645, 0.991512, 0.0007083
Begin processing the 74th record. run: 1062 subRun: 1 event: 28174 at 21-Jan-2026 15:10:20 GMT
Begin processing the 75th record. run: 1062 subRun: 1 event: 28175 at 21-Jan-2026 15:10:21 GMT
TrackPID: 2212, 0.00578178, 0.116952, 0.874753, 0.00251327
TrackPID: 22, 0.0339405, 0.302303, 0.659792, 0.0039643
Begin processing the 76th record. run: 1062 subRun: 1 event: 28176 at 21-Jan-2026 15:10:23 GMT
TrackPID: 211, 0.0319882, 0.225986, 0.73354, 0.00848625
TrackPID: 13, 0.338123, 0.478059, 0.174131, 0.00968628
TrackPID: -211, 0.0192144, 0.325824, 0.649491, 0.00547096
TrackPID: 2212, 0.000149476, 0.00231184, 0.996766, 0.000772513
Begin processing the 77th record. run: 1062 subRun: 1 event: 28177 at 21-Jan-2026 15:10:26 GMT
TrackPID: 13, 0.961057, 0.0282455, 0.00972052, 0.000977003
TrackPID: 321, 0.00353979, 0.223675, 0.760077, 0.0127083
TrackPID: 2212, 0.0721523, 0.785723, 0.133439, 0.00868554
TrackPID: 2212, 7.45505e-05, 0.00125579, 0.99821, 0.00045921
TrackPID: -211, 0.168667, 0.666597, 0.159458, 0.00527764
TrackPID: 2212, 0.000236949, 0.0603307, 0.929075, 0.0103573
TrackPID: 2212, 0.021055, 0.716996, 0.250556, 0.011393
Begin processing the 78th record. run: 1062 subRun: 1 event: 28178 at 21-Jan-2026 15:10:28 GMT
Begin processing the 79th record. run: 1062 subRun: 1 event: 28179 at 21-Jan-2026 15:10:30 GMT
Begin processing the 80th record. run: 1062 subRun: 1 event: 28180 at 21-Jan-2026 15:10:31 GMT
TrackPID: 2212, 0.000157632, 0.00608574, 0.992982, 0.000774791
TrackPID: -211, 0.0935448, 0.621113, 0.274809, 0.0105333
Begin processing the 81st record. run: 1062 subRun: 1 event: 28181 at 21-Jan-2026 15:10:33 GMT
TrackPID: 2212, 0.000695808, 0.019148, 0.979081, 0.00107489
TrackPID: 13, 0.0289338, 0.127773, 0.833855, 0.00943822
TrackPID: -211, 0.0482824, 0.758209, 0.17909, 0.0144191
TrackPID: 211, 0.0357047, 0.859217, 0.0889893, 0.0160889
Begin processing the 82nd record. run: 1062 subRun: 1 event: 28182 at 21-Jan-2026 15:10:35 GMT
TrackPID: 13, 0.326086, 0.2148, 0.450364, 0.00875008
TrackPID: -13, 0.0354042, 0.845579, 0.112411, 0.0066063
TrackPID: 321, 0.00465092, 0.400267, 0.583974, 0.0111084
TrackPID: 2212, 0.00092572, 0.00762099, 0.989093, 0.00235981
Begin processing the 83rd record. run: 1062 subRun: 1 event: 28183 at 21-Jan-2026 15:10:38 GMT
Begin processing the 84th record. run: 1062 subRun: 1 event: 28184 at 21-Jan-2026 15:10:39 GMT
TrackPID: 211, 0.0143893, 0.246944, 0.734751, 0.00391526
Begin processing the 85th record. run: 1062 subRun: 1 event: 28185 at 21-Jan-2026 15:10:42 GMT
PfoList empty... moving on.
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 86th record. run: 1062 subRun: 1 event: 28186 at 21-Jan-2026 15:10:44 GMT
TrackPID: 2212, 0.000137043, 0.00250894, 0.996708, 0.000645929
TrackPID: 2212, 5.41128e-05, 0.00191404, 0.997606, 0.000425869
TrackPID: 13, 0.104021, 0.177105, 0.706454, 0.0124197
TrackPID: 22, 0.0951358, 0.725758, 0.174, 0.00510653
TrackPID: 2212, 0.000156698, 0.0412287, 0.955577, 0.00303762
TrackPID: 2212, 0.000221637, 0.00499036, 0.993558, 0.00123043
Begin processing the 87th record. run: 1062 subRun: 1 event: 28187 at 21-Jan-2026 15:10:46 GMT
PandoraContentApi::GetList(*this, m_trackPfoListName, pTrackPfoList) return STATUS_CODE_NOT_INITIALIZED
in function: GetAllTrackFeatures
in file: /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoradlcontent/LArTrackShowerId/DlTrackCharacterisationAlgorithm.cc line#: 223
Begin processing the 88th record. run: 1062 subRun: 1 event: 28188 at 21-Jan-2026 15:10:48 GMT
TrackPID: 13, 0.995275, 0.00391584, 0.000721793, 8.7372e-05
TrackPID: 13, 0.311636, 0.586322, 0.0957044, 0.00633794
Begin processing the 89th record. run: 1062 subRun: 1 event: 28189 at 21-Jan-2026 15:10:50 GMT
TrackPID: 13, 0.931475, 0.0543636, 0.0130886, 0.00107292
Begin processing the 90th record. run: 1062 subRun: 1 event: 28190 at 21-Jan-2026 15:10:51 GMT
TrackPID: 2212, 0.0102869, 0.0864707, 0.896886, 0.00635603
TrackPID: 2212, 0.000293289, 0.00184326, 0.997176, 0.000687201
Begin processing the 91st record. run: 1062 subRun: 1 event: 28191 at 21-Jan-2026 15:10:56 GMT
Begin processing the 92nd record. run: 1062 subRun: 1 event: 28192 at 21-Jan-2026 15:10:59 GMT
TrackPID: 22, 0.0442877, 0.413255, 0.53578, 0.0066768
TrackPID: 211, 0.850861, 0.12164, 0.0254498, 0.00204945
TrackPID: 2212, 0.000425343, 0.00702501, 0.991488, 0.00106175
TrackPID: 13, 0.974076, 0.0204485, 0.00503576, 0.000440104
Begin processing the 93rd record. run: 1062 subRun: 1 event: 28193 at 21-Jan-2026 15:11:05 GMT
TrackPID: 22, 0.261151, 0.618826, 0.11719, 0.00283346
TrackPID: 22, 0.16092, 0.639693, 0.191636, 0.00775089
TrackPID: -13, 0.117128, 0.693144, 0.17763, 0.0120988
TrackPID: 2212, 0.000185077, 0.00147615, 0.997546, 0.000792522
Begin processing the 94th record. run: 1062 subRun: 1 event: 28194 at 21-Jan-2026 15:11:07 GMT
TrackPID: 13, 0.999162, 0.000705248, 9.77969e-05, 3.49838e-05
TrackPID: 13, 0.122227, 0.667081, 0.204566, 0.00612575
Begin processing the 95th record. run: 1062 subRun: 1 event: 28195 at 21-Jan-2026 15:11:09 GMT
TrackPID: 211, 0.0311509, 0.919774, 0.0444966, 0.00457803
TrackPID: 13, 0.528863, 0.375698, 0.0893286, 0.00611079
TrackPID: 2212, 0.00079532, 0.0141872, 0.983968, 0.00104898
Begin processing the 96th record. run: 1062 subRun: 1 event: 28196 at 21-Jan-2026 15:11:11 GMT
TrackPID: 211, 0.000552595, 0.0762279, 0.915299, 0.00792035
TrackPID: 211, 0.00880206, 0.574519, 0.397309, 0.0193699
TrackPID: 211, 0.0147559, 0.357627, 0.61663, 0.0109875
TrackPID: 13, 0.0678217, 0.253867, 0.668437, 0.00987398
Begin processing the 97th record. run: 1062 subRun: 1 event: 28197 at 21-Jan-2026 15:11:13 GMT
TrackPID: 2212, 0.000184208, 0.00568803, 0.993462, 0.000665745
TrackPID: 2212, 0.00049895, 0.0152357, 0.983359, 0.000905914
Begin processing the 98th record. run: 1062 subRun: 1 event: 28198 at 21-Jan-2026 15:11:15 GMT
TrackPID: 2212, 0.00845128, 0.0867709, 0.899332, 0.00544548
Begin processing the 99th record. run: 1062 subRun: 1 event: 28199 at 21-Jan-2026 15:11:16 GMT
TrackPID: 13, 0.200533, 0.699657, 0.0952678, 0.00454202
TrackPID: 13, 0.999831, 0.000120686, 3.64199e-05, 1.1762e-05
TrackPID: 13, 0.313594, 0.537546, 0.146119, 0.00274101
Begin processing the 100th record. run: 1062 subRun: 1 event: 28200 at 21-Jan-2026 15:11:19 GMT
TrackPID: 13, 0.111394, 0.647941, 0.23336, 0.00730471
TrackPID: 2212, 0.00031405, 0.00368904, 0.995218, 0.000778339
21-Jan-2026 15:11:21 GMT Closed output file "nu_dune10kt_1x2x6_1062_281_20230823T120818Z_gen_g4_detsim_hitreco__20240113T054159Z_reco2_reco2.root"
21-Jan-2026 15:11:21 GMT Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/9f/e7/nu_dune10kt_1x2x6_1062_281_20230823T120818Z_gen_g4_detsim_hitreco__20240113T054159Z_reco2.root"
================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 1.31717 2.60328 25.0466 1.93268 2.74365 100
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.016843 0.0298462 0.0375872 0.0306826 0.00539651 100
reco:pandora2:StandardPandora 1.29716 2.57251 25.0213 1.90134 2.74392 100
[art]:TriggerResults:TriggerResultInserter 1.3476e-05 2.62237e-05 8.2774e-05 2.2909e-05 1.09405e-05 100
end_path:out1:RootOutput 2.944e-06 3.72612e-06 2.5777e-05 3.438e-06 2.24808e-06 100
end_path:out1:RootOutput(write) 0.000198943 0.000712868 0.00344176 0.000423215 0.000703248 100
================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 2529.73 MB
Peak resident set size usage (VmHWM): 1547.47 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 428
-rw-r--r-- 1 pildune32 pildune 210 Jan 21 15:05 all-input-dids.txt
-rw-r--r-- 1 pildune32 pildune 0 Jan 21 15:06 debugprod.log
-rw-r--r-- 1 pildune32 pildune 59828 Jan 21 15:11 jobscript.log
-rw-r--r-- 1 pildune32 pildune 184 Jan 21 15:11 justin-processed-pfns.txt
drwxr-xr-x 4 pildune32 pildune 48 Jan 21 15:05 larpandoracontent
-rw-r--r-- 1 pildune32 pildune 363241 Jan 21 15:11 nu_dune10kt_1x2x6_1062_281_20230823T120818Z_gen_g4_detsim_hitreco__20240113T054159Z_reco2_reco2.root
-rw-r--r-- 1 pildune32 pildune 519 Jan 21 15:11 reco2_hist.root 