Jobsub ID 35296.1@dunegpschedd02.fnal.gov

Jobsub ID	35296.1@dunegpschedd02.fnal.gov
Workflow ID	2246
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-07 02:01:25
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-09-07 05:38:50
From worker node	Hostname	dunegli-6916305-0-fnpc23051.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-07 02:02:19
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250125T230528Z_gen_002786_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (12973s)
	CPU time	3h (12840s = 98%)
	Max RSS bytes	3088228352 (2945 MiB)
Outputting started	2025-09-07 05:38:32
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/02246/1/001/mydetsimreco1reco2_20250907T020223Z.root
Finished	2025-09-07 05:38:50
Saved logs	justin-logs:35296.1-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e 5th record. run: 4901 subRun: 0 event: 27855 at 07-Sep-2025 03:38:26 UTC
Begin processing the 6th record. run: 4901 subRun: 0 event: 27856 at 07-Sep-2025 03:38:29 UTC
Begin processing the 7th record. run: 4901 subRun: 0 event: 27857 at 07-Sep-2025 03:38:33 UTC
Begin processing the 8th record. run: 4901 subRun: 0 event: 27858 at 07-Sep-2025 03:38:37 UTC
Begin processing the 9th record. run: 4901 subRun: 0 event: 27859 at 07-Sep-2025 03:38:40 UTC
Begin processing the 10th record. run: 4901 subRun: 0 event: 27860 at 07-Sep-2025 03:38:44 UTC
07-Sep-2025 03:38:48 UTC  Closed output file "matt_reco1.root"
07-Sep-2025 03:38:48 UTC  Closed input file "matt_detsim.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.76031       1.79464       1.93798       1.77839      0.0500348       10     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000373868   0.000605678   0.00112312    0.000468088   0.000280033      10     
reco:gaushit:GausHitFinder                     0.199283      0.207214      0.219788      0.206475     0.00543243       10     
reco:ophit10ppm:OpHitFinder                     1.46914       1.49994       1.62837        1.483       0.045703        10     
[art]:TriggerResults:TriggerResultInserter    1.4949e-05    2.17841e-05   6.3352e-05    1.7514e-05    1.38883e-05      10     
end_path:out1:RootOutput                       3.296e-06    7.2588e-06    3.2813e-05     4.418e-06    8.52966e-06      10     
end_path:out1:RootOutput(write)                0.0845376     0.0865354     0.0884359     0.0866845    0.00116196       10     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20211.6 MB
  Peak resident set size usage (VmHWM): 2222.57 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
Begin processing the 8th record. run: 4901 subRun: 0 event: 27858 at 07-Sep-2025 05:03:38 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6547, 6531, 6577
minwire 0: 44
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 527, 1026
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.10522, 
Output 1: 0.0723024, 0.0824388, 0.156633, 0.688626, 
Output 2: 0.0428429, 0.203594, 0.7294, 0.0241629, 
Output 3: 0.946525, 0.0403612, 0.00613411, 0.00697934, 
Output 4: 0.45699, 0.455563, 0.0783665, 0.00908079, 
Output 5: 0.534325, 0.35264, 0.0939397, 0.0190952, 
Output 6: 0.0281627, 0.0790525, 0.0854857, 0.807299, 

Begin processing the 9th record. run: 4901 subRun: 0 event: 27859 at 07-Sep-2025 05:15:19 UTC
Boundary wire vector sizes: 6689, 6658, 6476
minwire 0: 7
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2084, 2583
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0971519, 
Output 1: 0.0470744, 0.0682766, 0.124254, 0.760395, 
Output 2: 0.0773461, 0.205506, 0.691708, 0.0254397, 
Output 3: 0.937039, 0.0496924, 0.00533409, 0.00793441, 
Output 4: 0.309016, 0.605574, 0.077379, 0.00803076, 
Output 5: 0.405464, 0.492358, 0.0867774, 0.0154007, 
Output 6: 0.0442574, 0.115802, 0.111281, 0.72866, 

Begin processing the 10th record. run: 4901 subRun: 0 event: 27860 at 07-Sep-2025 05:26:52 UTC
Boundary wire vector sizes: 6852, 6848, 6687
minwire 0: 34
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0952682, 
Output 1: 0.0454364, 0.0680451, 0.0816522, 0.804866, 
Output 2: 0.1044, 0.287409, 0.584601, 0.0235895, 
Output 3: 0.956268, 0.0359259, 0.0024484, 0.00535722, 
Output 4: 0.362622, 0.573245, 0.0587355, 0.00539725, 
Output 5: 0.394859, 0.556469, 0.0437231, 0.00494857, 
Output 6: 0.0240075, 0.0801715, 0.0966077, 0.799213, 

07-Sep-2025 05:38:31 UTC  Closed output file "mydetsimreco1reco2_20250907T020223Z.root"
07-Sep-2025 05:38:31 UTC  Closed input file "matt_reco1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              670.067       712.424       758.521       701.379       30.8098        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000464541   0.00408984    0.00550868    0.00479388    0.00173266       10     
reco:pandora:StandardPandora                            669.065       711.226       757.402       700.318       30.949         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00022429     0.0181332     0.0708194     0.0126246     0.0205174       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000199292   0.00403413     0.0151547    0.00305156    0.00414463       10     
reco:pandoracalo:Calorimetry                          0.000147603   0.00561822     0.0295714    0.000315806    0.0102972       10     
reco:pandorapid:Chi2ParticleID                        3.9136e-05    0.000205269   0.00103865    8.73075e-05   0.000288008      10     
reco:cvnmap:CVNMapper                                  0.0201038     0.0243344     0.0338816     0.0243355    0.00367901       10     
reco:cvneva:CVNEvaluator                               0.711006      0.877988       1.94097      0.768477      0.355271        10     
reco:energyrecnumu:EnergyReco                         0.00165539     0.0279858     0.0380528     0.0336418     0.0127736       10     
reco:energyrecnue:EnergyReco                          0.000814662   0.00268333     0.0054342    0.00107967    0.00205418       10     
reco:energyrecnc:EnergyReco                           0.00478297     0.0153438     0.0475635    0.00526511     0.0160557       10     
[art]:TriggerResults:TriggerResultInserter            1.1582e-05    2.54759e-05   0.000115402   1.53795e-05   3.00737e-05      10     
end_path:out1:RootOutput                               7.454e-06    1.08908e-05   3.5969e-05     8.226e-06    8.37271e-06      10     
end_path:out1:RootOutput(write)                        0.203277      0.216943      0.230563      0.218541      0.0091192       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21763.6 MB
  Peak resident set size usage (VmHWM): 3088.23 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0