Jobsub ID 226591.69@dunegpschedd02.fnal.gov
| Jobsub ID | 226591.69@dunegpschedd02.fnal.gov |
| Workflow ID | 8772 |
| Stage ID | 1 |
| User name | lwhite86@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 3600 (1 hours) |
| Submitted time | 2025-10-09 10:24:38 |
| Site | US_UCSD |
| Entry | CMSHTPC_T2_US_UCSD_gw7 |
| Last heartbeat | 2025-10-09 10:54:19 |
| From worker node | Hostname | mh-7763-16.t2.ucsd.edu |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | finished |
| Started | 2025-10-09 10:27:40 |
| Input files | fardet-hd:nu_dune10kt_1x2x6_1431_658_20230828T033401Z_gen_g4_detsim_hitreco__20240223T002215Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 26m (1581s) |
| CPU time | 4m (272s = 17%) |
| Max RSS bytes | 1181007872 (1126 MiB) |
| Outputting started | 2025-10-09 10:54:02 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/lwhite86/fnal/08772/1/001/trainingFile_nu_dune10kt_1x2x6_1431_658_20230828T033401Z_gen_g4_detsim_hitreco__20240223T002215Z_reco2.root
|
| Finished | 2025-10-09 10:54:19 |
| Saved logs | justin-logs:226591.69-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ODE_NOT_INITIALIZED
in function: GetAllTrackFeatures
in file: /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoradlcontent/LArTrackShowerId/DlTrackCharacterisationAlgorithm.cc line#: 155
Begin processing the 49th record. run: 1431 subRun: 1 event: 65849 at 09-Oct-2025 03:41:02 PDT
3651
67
Begin processing the 50th record. run: 1431 subRun: 1 event: 65850 at 09-Oct-2025 03:41:16 PDT
804
841
4113
467
470
50
71
258
682
Begin processing the 51st record. run: 1431 subRun: 1 event: 65851 at 09-Oct-2025 03:41:43 PDT
2303
677
33
Begin processing the 52nd record. run: 1431 subRun: 1 event: 65852 at 09-Oct-2025 03:41:58 PDT
46
4097
6721
35
Begin processing the 53rd record. run: 1431 subRun: 1 event: 65853 at 09-Oct-2025 03:42:24 PDT
656
132
776
57
49
Begin processing the 54th record. run: 1431 subRun: 1 event: 65854 at 09-Oct-2025 03:42:37 PDT
343
307
Begin processing the 55th record. run: 1431 subRun: 1 event: 65855 at 09-Oct-2025 03:42:44 PDT
3919
57
Begin processing the 56th record. run: 1431 subRun: 1 event: 65856 at 09-Oct-2025 03:42:51 PDT
Begin processing the 57th record. run: 1431 subRun: 1 event: 65857 at 09-Oct-2025 03:42:54 PDT
5866
31
Begin processing the 58th record. run: 1431 subRun: 1 event: 65858 at 09-Oct-2025 03:43:03 PDT
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 59th record. run: 1431 subRun: 1 event: 65859 at 09-Oct-2025 03:43:08 PDT
1318
95
135
145
42
28
Begin processing the 60th record. run: 1431 subRun: 1 event: 65860 at 09-Oct-2025 03:43:21 PDT
964
Begin processing the 61st record. run: 1431 subRun: 1 event: 65861 at 09-Oct-2025 03:43:25 PDT
37
Begin processing the 62nd record. run: 1431 subRun: 1 event: 65862 at 09-Oct-2025 03:43:29 PDT
115
608
169
115
59
78
Begin processing the 63rd record. run: 1431 subRun: 1 event: 65863 at 09-Oct-2025 03:43:45 PDT
573
238
Begin processing the 64th record. run: 1431 subRun: 1 event: 65864 at 09-Oct-2025 03:43:51 PDT
569
Begin processing the 65th record. run: 1431 subRun: 1 event: 65865 at 09-Oct-2025 03:43:56 PDT
278
76
114
Begin processing the 66th record. run: 1431 subRun: 1 event: 65866 at 09-Oct-2025 03:44:17 PDT
410
4545
290
98
Begin processing the 67th record. run: 1431 subRun: 1 event: 65867 at 09-Oct-2025 03:44:36 PDT
3362
144
75
78
287
Begin processing the 68th record. run: 1431 subRun: 1 event: 65868 at 09-Oct-2025 03:44:48 PDT
51
4549
87
55
Begin processing the 69th record. run: 1431 subRun: 1 event: 65869 at 09-Oct-2025 03:44:59 PDT
975
666
398
157
Begin processing the 70th record. run: 1431 subRun: 1 event: 65870 at 09-Oct-2025 03:45:08 PDT
47
Begin processing the 71st record. run: 1431 subRun: 1 event: 65871 at 09-Oct-2025 03:45:33 PDT
6119
336
Begin processing the 72nd record. run: 1431 subRun: 1 event: 65872 at 09-Oct-2025 03:45:53 PDT
Begin processing the 73rd record. run: 1431 subRun: 1 event: 65873 at 09-Oct-2025 03:45:59 PDT
868
Begin processing the 74th record. run: 1431 subRun: 1 event: 65874 at 09-Oct-2025 03:46:07 PDT
3570
48
Begin processing the 75th record. run: 1431 subRun: 1 event: 65875 at 09-Oct-2025 03:46:22 PDT
Begin processing the 76th record. run: 1431 subRun: 1 event: 65876 at 09-Oct-2025 03:46:26 PDT
102
1134
3305
1121
136
120
141
29
Begin processing the 77th record. run: 1431 subRun: 1 event: 65877 at 09-Oct-2025 03:47:28 PDT
597
7405
511
64
49
Begin processing the 78th record. run: 1431 subRun: 1 event: 65878 at 09-Oct-2025 03:47:46 PDT
2777
518
27
Begin processing the 79th record. run: 1431 subRun: 1 event: 65879 at 09-Oct-2025 03:47:56 PDT
250
127
119
220
Begin processing the 80th record. run: 1431 subRun: 1 event: 65880 at 09-Oct-2025 03:48:06 PDT
782
2562
462
Begin processing the 81st record. run: 1431 subRun: 1 event: 65881 at 09-Oct-2025 03:48:23 PDT
1840
Begin processing the 82nd record. run: 1431 subRun: 1 event: 65882 at 09-Oct-2025 03:48:30 PDT
1349
139
117
Begin processing the 83rd record. run: 1431 subRun: 1 event: 65883 at 09-Oct-2025 03:48:46 PDT
149
511
46
1135
Begin processing the 84th record. run: 1431 subRun: 1 event: 65884 at 09-Oct-2025 03:48:59 PDT
1211
161
64
27
Begin processing the 85th record. run: 1431 subRun: 1 event: 65885 at 09-Oct-2025 03:49:06 PDT
63
283
3479
112
57
Begin processing the 86th record. run: 1431 subRun: 1 event: 65886 at 09-Oct-2025 03:49:17 PDT
493
814
117
Begin processing the 87th record. run: 1431 subRun: 1 event: 65887 at 09-Oct-2025 03:49:30 PDT
26
Begin processing the 88th record. run: 1431 subRun: 1 event: 65888 at 09-Oct-2025 03:49:38 PDT
1682
531
157
Begin processing the 89th record. run: 1431 subRun: 1 event: 65889 at 09-Oct-2025 03:49:51 PDT
63
55
109
Begin processing the 90th record. run: 1431 subRun: 1 event: 65890 at 09-Oct-2025 03:50:01 PDT
3152
33
Begin processing the 91st record. run: 1431 subRun: 1 event: 65891 at 09-Oct-2025 03:50:15 PDT
75
122
121
39
139
130
26
203
Begin processing the 92nd record. run: 1431 subRun: 1 event: 65892 at 09-Oct-2025 03:50:31 PDT
32
2591
576
37
Begin processing the 93rd record. run: 1431 subRun: 1 event: 65893 at 09-Oct-2025 03:50:37 PDT
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
in function: GetVolumeIdToHitListMap
in file: /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
in function: Run
in file: /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 94th record. run: 1431 subRun: 1 event: 65894 at 09-Oct-2025 03:50:55 PDT
114
200
187
170
Begin processing the 95th record. run: 1431 subRun: 1 event: 65895 at 09-Oct-2025 03:51:20 PDT
182
1140
523
103
34
128
Begin processing the 96th record. run: 1431 subRun: 1 event: 65896 at 09-Oct-2025 03:51:35 PDT
1989
347
32
Begin processing the 97th record. run: 1431 subRun: 1 event: 65897 at 09-Oct-2025 03:51:49 PDT
356
359
Begin processing the 98th record. run: 1431 subRun: 1 event: 65898 at 09-Oct-2025 03:51:54 PDT
101
99
57
91
Begin processing the 99th record. run: 1431 subRun: 1 event: 65899 at 09-Oct-2025 03:52:00 PDT
1861
146
865
477
41
26
670
367
49
Begin processing the 100th record. run: 1431 subRun: 1 event: 65900 at 09-Oct-2025 03:53:57 PDT
80
26
28
09-Oct-2025 03:54:01 PDT Closed output file "nu_dune10kt_1x2x6_1431_658_20230828T033401Z_gen_g4_detsim_hitreco__20240223T002215Z_reco2_reco2.root"
09-Oct-2025 03:54:01 PDT Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/5a/1e/nu_dune10kt_1x2x6_1431_658_20230828T033401Z_gen_g4_detsim_hitreco__20240223T002215Z_reco2.root"
================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 0.438561 3.12276 25.0088 2.19941 3.54272 100
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0627828 0.138715 0.608176 0.108858 0.0986937 100
reco:pandora2:StandardPandora 0.325953 2.98329 24.7149 2.0595 3.53327 100
[art]:TriggerResults:TriggerResultInserter 1.025e-05 1.87573e-05 0.000112169 1.408e-05 1.52712e-05 100
end_path:out1:RootOutput 2.04e-06 3.24907e-06 2.859e-05 2.505e-06 3.19499e-06 100
end_path:out1:RootOutput(write) 0.000150198 0.000548814 0.00289274 0.000294247 0.00056231 100
================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 2248.18 MB
Peak resident set size usage (VmHWM): 1181.01 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 1136
-rw-r--r--. 1 cuser cuser 210 Oct 9 03:27 all-input-dids.txt
-rw-r--r--. 1 cuser cuser 0 Oct 9 03:27 debugprod.log
-rw-r--r--. 1 cuser cuser 44992 Oct 9 03:54 jobscript.log
-rw-r--r--. 1 cuser cuser 181 Oct 9 03:54 justin-processed-pfns.txt
drwxr-xr-x. 4 cuser cuser 60 Oct 9 03:27 larpandoracontent
-rw-r--r--. 1 cuser cuser 355039 Oct 9 03:54 nu_dune10kt_1x2x6_1431_658_20230828T033401Z_gen_g4_detsim_hitreco__20240223T002215Z_reco2_reco2.root
-rw-r--r--. 1 cuser cuser 519 Oct 9 03:54 reco2_hist.root
-rw-r--r--. 1 cuser cuser 746434 Oct 9 03:54 trainingFile_nu_dune10kt_1x2x6_1431_658_20230828T033401Z_gen_g4_detsim_hitreco__20240223T002215Z_reco2.root 