Jobsub ID 42003.0@dunegpschedd01.fnal.gov

Jobsub ID	42003.0@dunegpschedd01.fnal.gov
Workflow ID	2246
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-07 00:43:20
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-09-07 06:59:02
From worker node	Hostname	dunegli-6909907-0-fnpc19118.fnal.gov
	cpuinfo	AMD EPYC 7502 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-07 00:44:49
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T111354Z_gen_002527_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	6h (22432s)
	CPU time	6h (22232s = 99%)
	Max RSS bytes	3065442304 (2923 MiB)
Outputting started	2025-09-07 06:58:42
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/02246/1/001/mydetsimreco1reco2_20250907T004506Z.root
Finished	2025-09-07 06:59:02
Saved logs	justin-logs:42003.0-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

5th record. run: 5059 subRun: 0 event: 25265 at 07-Sep-2025 02:43:30 UTC
Begin processing the 6th record. run: 5059 subRun: 0 event: 25266 at 07-Sep-2025 02:43:35 UTC
Begin processing the 7th record. run: 5059 subRun: 0 event: 25267 at 07-Sep-2025 02:43:40 UTC
Begin processing the 8th record. run: 5059 subRun: 0 event: 25268 at 07-Sep-2025 02:43:46 UTC
Begin processing the 9th record. run: 5059 subRun: 0 event: 25269 at 07-Sep-2025 02:43:51 UTC
Begin processing the 10th record. run: 5059 subRun: 0 event: 25270 at 07-Sep-2025 02:43:57 UTC
07-Sep-2025 02:44:03 UTC  Closed output file "matt_reco1.root"
07-Sep-2025 02:44:03 UTC  Closed input file "matt_detsim.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      2.27273       2.33717       2.49695       2.30288      0.0672117       10     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000362092   0.000436636   0.000595574   0.000400158   7.91085e-05      10     
reco:gaushit:GausHitFinder                     0.289596      0.299154      0.312592      0.297546     0.00716764       10     
reco:ophit10ppm:OpHitFinder                     1.8624        1.91621       2.06065       1.88543      0.059247        10     
[art]:TriggerResults:TriggerResultInserter     1.299e-05    1.9528e-05     5.141e-05     1.569e-05    1.08908e-05      10     
end_path:out1:RootOutput                       3.79e-06     7.0611e-06     2.572e-05     4.22e-06      6.652e-06       10     
end_path:out1:RootOutput(write)                0.116744      0.121064      0.126043      0.120043     0.00261336       10     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20211.4 MB
  Peak resident set size usage (VmHWM): 2219.25 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6702, 6562, 6598
minwire 0: 42
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1544, 2043
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.10198, 
Output 1: 0.0408171, 0.0765614, 0.127213, 0.755408, 
Output 2: 0.0324871, 0.205978, 0.740324, 0.0212104, 
Output 3: 0.942409, 0.0446883, 0.00671657, 0.00618627, 
Output 4: 0.369092, 0.488972, 0.128153, 0.0137831, 
Output 5: 0.562574, 0.309659, 0.106531, 0.0212349, 
Output 6: 0.0250941, 0.067652, 0.0799914, 0.827263, 

Begin processing the 9th record. run: 5059 subRun: 0 event: 25269 at 07-Sep-2025 06:07:44 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6603, 6613, 6533
minwire 0: 35
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2000, 2499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0812094, 
Output 1: 0.0254732, 0.0473625, 0.0976372, 0.829527, 
Output 2: 0.0537425, 0.203189, 0.714123, 0.0289448, 
Output 3: 0.95763, 0.0285836, 0.00508227, 0.00870447, 
Output 4: 0.493843, 0.425305, 0.0715124, 0.00933978, 
Output 5: 0.688464, 0.250891, 0.0494873, 0.0111581, 
Output 6: 0.00936472, 0.0358306, 0.0516782, 0.903126, 

Begin processing the 10th record. run: 5059 subRun: 0 event: 25270 at 07-Sep-2025 06:33:07 UTC
Boundary wire vector sizes: 6725, 6739, 6542
minwire 0: 18
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2029, 2528
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.135957, 
Output 1: 0.0786462, 0.109302, 0.170436, 0.641616, 
Output 2: 0.0467981, 0.276072, 0.636435, 0.0406949, 
Output 3: 0.939339, 0.0342335, 0.0106168, 0.0158102, 
Output 4: 0.527866, 0.394313, 0.0700056, 0.00781621, 
Output 5: 0.59697, 0.309094, 0.0787115, 0.0152243, 
Output 6: 0.0190493, 0.046167, 0.063076, 0.871708, 

07-Sep-2025 06:58:41 UTC  Closed output file "mydetsimreco1reco2_20250907T004506Z.root"
07-Sep-2025 06:58:41 UTC  Closed input file "matt_reco1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1443.3        1520.19        1600         1525.36       45.9553        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000624635   0.00358231    0.00516531    0.00428714    0.00144743       10     
reco:pandora:StandardPandora                            1441.95       1518.73       1598.68       1524.1        46.0769        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00204682     0.0195075     0.0696944     0.0156998     0.0189009       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00117999    0.00462384     0.0106151    0.00510336    0.00285631       10     
reco:pandoracalo:Calorimetry                          0.000242551   0.00228381     0.0175155    0.000392553   0.00511462       10     
reco:pandorapid:Chi2ParticleID                         7.61e-05     0.000190498   0.000997107   8.6881e-05    0.000272459      10     
reco:cvnmap:CVNMapper                                  0.0239186     0.0283789     0.0381714     0.0248969    0.00539803       10     
reco:cvneva:CVNEvaluator                               0.851022       1.03211       2.37184      0.879767      0.447681        10     
reco:energyrecnumu:EnergyReco                          0.0044329     0.0281673     0.0572481     0.0195836     0.0206985       10     
reco:energyrecnue:EnergyReco                          0.000919107    0.0023245    0.00732376     0.0011358     0.0024924       10     
reco:energyrecnc:EnergyReco                            0.0071641     0.0351208     0.0605502     0.0493913     0.0229035       10     
[art]:TriggerResults:TriggerResultInserter             1.466e-05    2.20612e-05   7.4581e-05    1.63005e-05   1.75444e-05      10     
end_path:out1:RootOutput                               8.37e-06     1.10561e-05    2.684e-05     9.46e-06     5.29262e-06      10     
end_path:out1:RootOutput(write)                        0.272599      0.302007       0.34834      0.304594      0.0206968       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21765 MB
  Peak resident set size usage (VmHWM): 3065.44 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0