Jobsub ID 226702.124@dunegpschedd02.fnal.gov
| Jobsub ID | 226702.124@dunegpschedd02.fnal.gov |
| Workflow ID | 8797 |
| Stage ID | 1 |
| User name | lwhite86@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 3600 (1 hours) |
| Submitted time | 2025-10-09 17:09:01 |
| Site | US_FNAL-T1 |
| Entry | CMSHTPC_T1_US_FNAL_condce_opp1_whole |
| Last heartbeat | 2025-10-09 17:35:40 |
| From worker node | Hostname | dunegli-46902-0-cmswn5023.fnal.gov |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | finished |
| Started | 2025-10-09 17:26:17 |
| Input files | fardet-hd:nu_dune10kt_1x2x6_1108_153_20230826T213337Z_gen_g4_detsim_hitreco__20240222T225439Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 9m (555s) |
| CPU time | 9m (544s = 98%) |
| Max RSS bytes | 1536897024 (1465 MiB) |
| Outputting started | 2025-10-09 17:35:33 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/lwhite86/fnal/08797/1/001/trainingFile_nu_dune10kt_1x2x6_1108_153_20230826T213337Z_gen_g4_detsim_hitreco__20240222T225439Z_reco2.root
|
| Finished | 2025-10-09 17:35:40 |
| Saved logs | justin-logs:226702.124-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
472
26
Begin processing the 51st record. run: 1108 subRun: 1 event: 15351 at 09-Oct-2025 17:27:58 UTC
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 52nd record. run: 1108 subRun: 1 event: 15352 at 09-Oct-2025 17:27:59 UTC
701
2197
320
72
153
Begin processing the 53rd record. run: 1108 subRun: 1 event: 15353 at 09-Oct-2025 17:28:07 UTC
32
Begin processing the 54th record. run: 1108 subRun: 1 event: 15354 at 09-Oct-2025 17:28:08 UTC
58
995
4092
Begin processing the 55th record. run: 1108 subRun: 1 event: 15355 at 09-Oct-2025 17:28:09 UTC
2236
125
Begin processing the 56th record. run: 1108 subRun: 1 event: 15356 at 09-Oct-2025 17:28:11 UTC
953
Begin processing the 57th record. run: 1108 subRun: 1 event: 15357 at 09-Oct-2025 17:28:11 UTC
3304
50
Begin processing the 58th record. run: 1108 subRun: 1 event: 15358 at 09-Oct-2025 17:28:12 UTC
3307
190
Begin processing the 59th record. run: 1108 subRun: 1 event: 15359 at 09-Oct-2025 17:28:14 UTC
2045
Begin processing the 60th record. run: 1108 subRun: 1 event: 15360 at 09-Oct-2025 17:28:15 UTC
419
994
173
364
25
Begin processing the 61st record. run: 1108 subRun: 1 event: 15361 at 09-Oct-2025 17:28:16 UTC
Begin processing the 62nd record. run: 1108 subRun: 1 event: 15362 at 09-Oct-2025 17:28:17 UTC
117
Begin processing the 63rd record. run: 1108 subRun: 1 event: 15363 at 09-Oct-2025 17:28:18 UTC
891
326
237
Begin processing the 64th record. run: 1108 subRun: 1 event: 15364 at 09-Oct-2025 17:28:19 UTC
604
110
41
55
Begin processing the 65th record. run: 1108 subRun: 1 event: 15365 at 09-Oct-2025 17:28:20 UTC
Begin processing the 66th record. run: 1108 subRun: 1 event: 15366 at 09-Oct-2025 17:28:20 UTC
403
60
38
Begin processing the 67th record. run: 1108 subRun: 1 event: 15367 at 09-Oct-2025 17:28:21 UTC
2498
169
48
Begin processing the 68th record. run: 1108 subRun: 1 event: 15368 at 09-Oct-2025 17:28:22 UTC
97
634
5480
174
93
41
Begin processing the 69th record. run: 1108 subRun: 1 event: 15369 at 09-Oct-2025 17:28:24 UTC
106
Begin processing the 70th record. run: 1108 subRun: 1 event: 15370 at 09-Oct-2025 17:28:25 UTC
1281
71
Begin processing the 71st record. run: 1108 subRun: 1 event: 15371 at 09-Oct-2025 17:28:26 UTC
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 72nd record. run: 1108 subRun: 1 event: 15372 at 09-Oct-2025 17:28:27 UTC
54
Begin processing the 73rd record. run: 1108 subRun: 1 event: 15373 at 09-Oct-2025 17:28:28 UTC
117
4904
36
Begin processing the 74th record. run: 1108 subRun: 1 event: 15374 at 09-Oct-2025 17:28:29 UTC
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 75th record. run: 1108 subRun: 1 event: 15375 at 09-Oct-2025 17:28:30 UTC
246
1265
152
Begin processing the 76th record. run: 1108 subRun: 1 event: 15376 at 09-Oct-2025 17:28:31 UTC
PandoraContentApi::GetList(*this, m_trackPfoListName, pTrackPfoList) return STATUS_CODE_NOT_INITIALIZED
in function: GetAllTrackFeatures
in file: /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoradlcontent/LArTrackShowerId/DlTrackCharacterisationAlgorithm.cc line#: 155
Begin processing the 77th record. run: 1108 subRun: 1 event: 15377 at 09-Oct-2025 17:28:32 UTC
268
204
59
Begin processing the 78th record. run: 1108 subRun: 1 event: 15378 at 09-Oct-2025 17:28:33 UTC
110
657
26
Begin processing the 79th record. run: 1108 subRun: 1 event: 15379 at 09-Oct-2025 17:28:34 UTC
69
Begin processing the 80th record. run: 1108 subRun: 1 event: 15380 at 09-Oct-2025 17:28:35 UTC
1878
Begin processing the 81st record. run: 1108 subRun: 1 event: 15381 at 09-Oct-2025 17:28:36 UTC
2803
867
1700
194
536
42
32
170
313
63
Begin processing the 82nd record. run: 1108 subRun: 1 event: 15382 at 09-Oct-2025 17:28:45 UTC
157
224
1686
240
52
87
75
64
314
186
Begin processing the 83rd record. run: 1108 subRun: 1 event: 15383 at 09-Oct-2025 17:28:46 UTC
777
140
73
Begin processing the 84th record. run: 1108 subRun: 1 event: 15384 at 09-Oct-2025 17:28:47 UTC
1940
Begin processing the 85th record. run: 1108 subRun: 1 event: 15385 at 09-Oct-2025 17:28:48 UTC
1899
435
549
Begin processing the 86th record. run: 1108 subRun: 1 event: 15386 at 09-Oct-2025 17:28:49 UTC
1765
244
37
39
166
Begin processing the 87th record. run: 1108 subRun: 1 event: 15387 at 09-Oct-2025 17:28:50 UTC
1673
620
358
217
43
Begin processing the 88th record. run: 1108 subRun: 1 event: 15388 at 09-Oct-2025 17:28:51 UTC
175
1374
223
Begin processing the 89th record. run: 1108 subRun: 1 event: 15389 at 09-Oct-2025 17:28:52 UTC
38
Begin processing the 90th record. run: 1108 subRun: 1 event: 15390 at 09-Oct-2025 17:28:53 UTC
564
519
254
436
368
937
32
176
214
431
126
52
216
36
124
188
509
Begin processing the 91st record. run: 1108 subRun: 1 event: 15391 at 09-Oct-2025 17:35:24 UTC
237
3623
638
750
122
54
109
155
74
60
Begin processing the 92nd record. run: 1108 subRun: 1 event: 15392 at 09-Oct-2025 17:35:25 UTC
Begin processing the 93rd record. run: 1108 subRun: 1 event: 15393 at 09-Oct-2025 17:35:26 UTC
330
Begin processing the 94th record. run: 1108 subRun: 1 event: 15394 at 09-Oct-2025 17:35:26 UTC
45
Begin processing the 95th record. run: 1108 subRun: 1 event: 15395 at 09-Oct-2025 17:35:27 UTC
249
Begin processing the 96th record. run: 1108 subRun: 1 event: 15396 at 09-Oct-2025 17:35:28 UTC
2751
30
Begin processing the 97th record. run: 1108 subRun: 1 event: 15397 at 09-Oct-2025 17:35:29 UTC
78
Begin processing the 98th record. run: 1108 subRun: 1 event: 15398 at 09-Oct-2025 17:35:30 UTC
222
Begin processing the 99th record. run: 1108 subRun: 1 event: 15399 at 09-Oct-2025 17:35:30 UTC
813
Begin processing the 100th record. run: 1108 subRun: 1 event: 15400 at 09-Oct-2025 17:35:31 UTC
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
09-Oct-2025 17:35:32 UTC Closed output file "nu_dune10kt_1x2x6_1108_153_20230826T213337Z_gen_g4_detsim_hitreco__20240222T225439Z_reco2_reco2.root"
09-Oct-2025 17:35:32 UTC Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/d6/18/nu_dune10kt_1x2x6_1108_153_20230826T213337Z_gen_g4_detsim_hitreco__20240222T225439Z_reco2.root"
================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 0.00942597 5.15712 389.039 0.975591 38.6055 100
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00109691 0.0121956 0.0392585 0.0135226 0.0078047 100
reco:pandora2:StandardPandora 0.00807839 5.14432 389.025 0.949862 38.6054 100
[art]:TriggerResults:TriggerResultInserter 9.488e-06 1.67688e-05 4.6449e-05 1.42015e-05 7.07105e-06 100
end_path:out1:RootOutput 2.535e-06 4.54055e-06 1.8485e-05 4.188e-06 2.08518e-06 100
end_path:out1:RootOutput(write) 0.000118435 0.000415372 0.00219029 0.000251251 0.000402241 100
================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 2548.84 MB
Peak resident set size usage (VmHWM): 1536.9 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 1084
-rw-r--r-- 1 dunegli fnalgrid 210 Oct 9 17:26 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid 0 Oct 9 17:26 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid 47248 Oct 9 17:35 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid 181 Oct 9 17:35 justin-processed-pfns.txt
drwxr-xr-x 4 dunegli fnalgrid 60 Oct 9 17:26 larpandoracontent
-rw-r--r-- 1 dunegli fnalgrid 355086 Oct 9 17:35 nu_dune10kt_1x2x6_1108_153_20230826T213337Z_gen_g4_detsim_hitreco__20240222T225439Z_reco2_reco2.root
-rw-r--r-- 1 dunegli fnalgrid 519 Oct 9 17:35 reco2_hist.root
-rw-r--r-- 1 dunegli fnalgrid 688779 Oct 9 17:35 trainingFile_nu_dune10kt_1x2x6_1108_153_20230826T213337Z_gen_g4_detsim_hitreco__20240222T225439Z_reco2.root 