Jobsub ID 232164.37@dunegpschedd01.fnal.gov

Jobsub ID	232164.37@dunegpschedd01.fnal.gov
Workflow ID	8797
Stage ID	1
User name	lwhite86@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	3600 (1 hours)
Submitted time	2025-10-09 16:38:59
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-10-09 16:56:01
From worker node	Hostname	dunegli-46902-0-cmswn5023.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-09 16:51:13
Input files	fardet-hd:nue_dune10kt_1x2x6_1435_999_20230828T105807Z_gen_g4_detsim_hitreco__20240221T171513Z_reco2.root
Jobscript	Exit code	0
	Real time	4m (259s)
	CPU time	4m (250s = 96%)
	Max RSS bytes	1331654656 (1269 MiB)
Outputting started	2025-10-09 16:55:33
Output files
Finished	2025-10-09 16:56:01
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

e/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 50th record. run: 1435 subRun: 1 event: 99950 at 09-Oct-2025 16:53:12 UTC
420
366
45
140
1514
66
59
543
408
48
52
Begin processing the 51st record. run: 1435 subRun: 1 event: 99951 at 09-Oct-2025 16:53:18 UTC
60
Begin processing the 52nd record. run: 1435 subRun: 1 event: 99952 at 09-Oct-2025 16:53:19 UTC
177
282
62
Begin processing the 53rd record. run: 1435 subRun: 1 event: 99953 at 09-Oct-2025 16:53:20 UTC
406
Begin processing the 54th record. run: 1435 subRun: 1 event: 99954 at 09-Oct-2025 16:53:22 UTC
106
545
Begin processing the 55th record. run: 1435 subRun: 1 event: 99955 at 09-Oct-2025 16:53:23 UTC
Begin processing the 56th record. run: 1435 subRun: 1 event: 99956 at 09-Oct-2025 16:53:24 UTC
542
202
198
Begin processing the 57th record. run: 1435 subRun: 1 event: 99957 at 09-Oct-2025 16:53:25 UTC
225
174
1789
34
52
106
711
260
Begin processing the 58th record. run: 1435 subRun: 1 event: 99958 at 09-Oct-2025 16:53:47 UTC
560
Begin processing the 59th record. run: 1435 subRun: 1 event: 99959 at 09-Oct-2025 16:53:48 UTC
87
45
Begin processing the 60th record. run: 1435 subRun: 1 event: 99960 at 09-Oct-2025 16:53:49 UTC
51
Begin processing the 61st record. run: 1435 subRun: 1 event: 99961 at 09-Oct-2025 16:53:49 UTC
65
71
Begin processing the 62nd record. run: 1435 subRun: 1 event: 99962 at 09-Oct-2025 16:53:50 UTC
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 63rd record. run: 1435 subRun: 1 event: 99963 at 09-Oct-2025 16:53:53 UTC
45
127
Begin processing the 64th record. run: 1435 subRun: 1 event: 99964 at 09-Oct-2025 16:53:54 UTC
67
103
Begin processing the 65th record. run: 1435 subRun: 1 event: 99965 at 09-Oct-2025 16:53:55 UTC
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 66th record. run: 1435 subRun: 1 event: 99966 at 09-Oct-2025 16:53:56 UTC
Begin processing the 67th record. run: 1435 subRun: 1 event: 99967 at 09-Oct-2025 16:53:57 UTC
Begin processing the 68th record. run: 1435 subRun: 1 event: 99968 at 09-Oct-2025 16:54:01 UTC
121
Begin processing the 69th record. run: 1435 subRun: 1 event: 99969 at 09-Oct-2025 16:54:02 UTC
170
226
33
Begin processing the 70th record. run: 1435 subRun: 1 event: 99970 at 09-Oct-2025 16:54:03 UTC
30
Begin processing the 71st record. run: 1435 subRun: 1 event: 99971 at 09-Oct-2025 16:54:04 UTC
25
Begin processing the 72nd record. run: 1435 subRun: 1 event: 99972 at 09-Oct-2025 16:54:05 UTC
29
Begin processing the 73rd record. run: 1435 subRun: 1 event: 99973 at 09-Oct-2025 16:54:05 UTC
491
114
174
168
52
46
Begin processing the 74th record. run: 1435 subRun: 1 event: 99974 at 09-Oct-2025 16:54:07 UTC
581
Begin processing the 75th record. run: 1435 subRun: 1 event: 99975 at 09-Oct-2025 16:54:08 UTC
50
Begin processing the 76th record. run: 1435 subRun: 1 event: 99976 at 09-Oct-2025 16:54:09 UTC
32
50
Begin processing the 77th record. run: 1435 subRun: 1 event: 99977 at 09-Oct-2025 16:54:10 UTC
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 78th record. run: 1435 subRun: 1 event: 99978 at 09-Oct-2025 16:54:11 UTC
146
518
158
82
Begin processing the 79th record. run: 1435 subRun: 1 event: 99979 at 09-Oct-2025 16:54:12 UTC
704
45
80
Begin processing the 80th record. run: 1435 subRun: 1 event: 99980 at 09-Oct-2025 16:54:13 UTC
222
178
86
Begin processing the 81st record. run: 1435 subRun: 1 event: 99981 at 09-Oct-2025 16:54:14 UTC
462
159
33
104
Begin processing the 82nd record. run: 1435 subRun: 1 event: 99982 at 09-Oct-2025 16:54:41 UTC
1069
39
74
139
Begin processing the 83rd record. run: 1435 subRun: 1 event: 99983 at 09-Oct-2025 16:54:42 UTC
324
99
182
93
Begin processing the 84th record. run: 1435 subRun: 1 event: 99984 at 09-Oct-2025 16:54:44 UTC
521
93
53
312
67
Begin processing the 85th record. run: 1435 subRun: 1 event: 99985 at 09-Oct-2025 16:54:45 UTC
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 86th record. run: 1435 subRun: 1 event: 99986 at 09-Oct-2025 16:54:47 UTC
519
45
Begin processing the 87th record. run: 1435 subRun: 1 event: 99987 at 09-Oct-2025 16:54:48 UTC
Begin processing the 88th record. run: 1435 subRun: 1 event: 99988 at 09-Oct-2025 16:54:49 UTC
33
170
52
Begin processing the 89th record. run: 1435 subRun: 1 event: 99989 at 09-Oct-2025 16:54:50 UTC
148
92
162
55
Begin processing the 90th record. run: 1435 subRun: 1 event: 99990 at 09-Oct-2025 16:54:51 UTC
45
61
Begin processing the 91st record. run: 1435 subRun: 1 event: 99991 at 09-Oct-2025 16:54:52 UTC
115
31
72
922
63
Begin processing the 92nd record. run: 1435 subRun: 1 event: 99992 at 09-Oct-2025 16:54:53 UTC
78
249
211
921
118
361
319
28
330
530
135
28
42
Begin processing the 93rd record. run: 1435 subRun: 1 event: 99993 at 09-Oct-2025 16:55:15 UTC
140
320
228
168
213
41
36
107
Begin processing the 94th record. run: 1435 subRun: 1 event: 99994 at 09-Oct-2025 16:55:22 UTC
84
147
104
87
55
Begin processing the 95th record. run: 1435 subRun: 1 event: 99995 at 09-Oct-2025 16:55:22 UTC
29
271
Begin processing the 96th record. run: 1435 subRun: 1 event: 99996 at 09-Oct-2025 16:55:25 UTC
70
92
117
Begin processing the 97th record. run: 1435 subRun: 1 event: 99997 at 09-Oct-2025 16:55:28 UTC
Begin processing the 98th record. run: 1435 subRun: 1 event: 99998 at 09-Oct-2025 16:55:29 UTC
218
112
110
124
48
34
Begin processing the 99th record. run: 1435 subRun: 1 event: 99999 at 09-Oct-2025 16:55:30 UTC
345
Begin processing the 100th record. run: 1435 subRun: 1 event: 100000 at 09-Oct-2025 16:55:31 UTC
144
09-Oct-2025 16:55:32 UTC  Closed output file "nue_dune10kt_1x2x6_1435_999_20230828T105807Z_gen_g4_detsim_hitreco__20240221T171513Z_reco2_reco2.root"
09-Oct-2025 16:55:32 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/b9/36/nue_dune10kt_1x2x6_1435_999_20230828T105807Z_gen_g4_detsim_hitreco__20240221T171513Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0239763      2.1872        26.2111       0.97235       4.43816        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00146736     0.0146433      0.29038      0.0132162     0.0285369       100    
reco:pandora2:StandardPandora                  0.0147952      2.17189       26.1922      0.954254       4.43888        100    
[art]:TriggerResults:TriggerResultInserter     9.297e-06    1.78898e-05   5.8511e-05    1.52135e-05   8.48679e-06      100    
end_path:out1:RootOutput                       1.913e-06    3.64167e-06   1.3315e-05     3.421e-06    1.31486e-06      100    
end_path:out1:RootOutput(write)               0.000141048   0.000470853   0.00154083    0.000289444   0.000369699      100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2333.07 MB
  Peak resident set size usage (VmHWM): 1331.65 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 924
-rw-r--r-- 1 dunegli fnalgrid    212 Oct  9 16:51 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid      0 Oct  9 16:51 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid  47550 Oct  9 16:55 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid    182 Oct  9 16:55 justin-processed-pfns.txt
drwxr-xr-x 4 dunegli fnalgrid     60 Oct  9 16:51 larpandoracontent
-rw-r--r-- 1 dunegli fnalgrid 359107 Oct  9 16:55 nue_dune10kt_1x2x6_1435_999_20230828T105807Z_gen_g4_detsim_hitreco__20240221T171513Z_reco2_reco2.root
-rw-r--r-- 1 dunegli fnalgrid    519 Oct  9 16:55 reco2_hist.root
-rw-r--r-- 1 dunegli fnalgrid 520917 Oct  9 16:55 trainingFile_nue_dune10kt_1x2x6_1435_999_20230828T105807Z_gen_g4_detsim_hitreco__20240221T171513Z_reco2.root