Jobsub ID 35296.0@dunegpschedd02.fnal.gov

Jobsub ID	35296.0@dunegpschedd02.fnal.gov
Workflow ID	2246
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-07 02:01:25
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-09-07 09:05:19
From worker node	Hostname	dunegli-6909907-0-fnpc19118.fnal.gov
	cpuinfo	AMD EPYC 7502 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-07 02:02:08
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T100629Z_gen_001107_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (25371s)
	CPU time	6h (25121s = 99%)
	Max RSS bytes	3065307136 (2923 MiB)
Outputting started	2025-09-07 09:05:00
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/02246/1/001/mydetsimreco1reco2_20250907T020225Z.root
Finished	2025-09-07 09:05:19
Saved logs	justin-logs:35296.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t: 11064 at 07-Sep-2025 04:14:42 UTC
Begin processing the 5th record. run: 5059 subRun: 0 event: 11065 at 07-Sep-2025 04:14:48 UTC
Begin processing the 6th record. run: 5059 subRun: 0 event: 11066 at 07-Sep-2025 04:14:54 UTC
Begin processing the 7th record. run: 5059 subRun: 0 event: 11067 at 07-Sep-2025 04:15:00 UTC
Begin processing the 8th record. run: 5059 subRun: 0 event: 11068 at 07-Sep-2025 04:15:07 UTC
Begin processing the 9th record. run: 5059 subRun: 0 event: 11069 at 07-Sep-2025 04:15:13 UTC
Begin processing the 10th record. run: 5059 subRun: 0 event: 11070 at 07-Sep-2025 04:15:19 UTC
07-Sep-2025 04:15:26 UTC  Closed output file "matt_reco1.root"
07-Sep-2025 04:15:26 UTC  Closed input file "matt_detsim.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      2.41716       2.49928       2.70155       2.46786      0.0911386       10     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000484123   0.000821483   0.00129149    0.000701611   0.000303768      10     
reco:gaushit:GausHitFinder                     0.299045      0.321103      0.403888      0.312784      0.0288214       10     
reco:ophit10ppm:OpHitFinder                     1.98932       2.04208       2.22216       2.01739      0.0681206       10     
[art]:TriggerResults:TriggerResultInserter     1.735e-05    3.0243e-05     5.059e-05     3.068e-05    1.07312e-05      10     
end_path:out1:RootOutput                       4.29e-06     9.0251e-06     2.444e-05     8.38e-06     5.56816e-06      10     
end_path:out1:RootOutput(write)                0.127815      0.134866      0.144957      0.134661     0.00457028       10     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20211.5 MB
  Peak resident set size usage (VmHWM): 2221 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6672, 6591, 6530
minwire 0: 10
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2290, 2789
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.116389, 
Output 1: 0.1802, 0.114541, 0.245146, 0.460113, 
Output 2: 0.0516998, 0.0707456, 0.867855, 0.00969902, 
Output 3: 0.960872, 0.0366988, 0.00102395, 0.00140564, 
Output 4: 0.411281, 0.535731, 0.0451307, 0.00785749, 
Output 5: 0.162318, 0.467015, 0.259657, 0.11101, 
Output 6: 0.195016, 0.307832, 0.142918, 0.354235, 

Begin processing the 9th record. run: 5059 subRun: 0 event: 11069 at 07-Sep-2025 08:09:16 UTC
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  07-Sep-2025 08:35:10 UTC run: 5059 subRun: 0 event: 11069
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  07-Sep-2025 08:35:10 UTC run: 5059 subRun: 0 event: 11069
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6667, 6657, 6596
minwire 0: 18
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 907, 1406
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.100904, 
Output 1: 0.0843969, 0.0630761, 0.135977, 0.71655, 
Output 2: 0.0450495, 0.173487, 0.766355, 0.0151078, 
Output 3: 0.964783, 0.0300865, 0.00208915, 0.00304177, 
Output 4: 0.369942, 0.549909, 0.0727241, 0.00742452, 
Output 5: 0.447525, 0.441362, 0.0935651, 0.0175483, 
Output 6: 0.0279745, 0.0919364, 0.0953738, 0.784715, 

Begin processing the 10th record. run: 5059 subRun: 0 event: 11070 at 07-Sep-2025 08:35:11 UTC
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  07-Sep-2025 09:04:57 UTC run: 5059 subRun: 0 event: 11070
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  07-Sep-2025 09:04:57 UTC run: 5059 subRun: 0 event: 11070
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  07-Sep-2025 09:04:57 UTC run: 5059 subRun: 0 event: 11070
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6707, 6745, 6561
minwire 0: 30
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2052, 2551
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.100165, 
Output 1: 0.048556, 0.0877136, 0.0902833, 0.773447, 
Output 2: 0.064992, 0.253745, 0.650647, 0.0306167, 
Output 3: 0.969239, 0.0182985, 0.00276157, 0.00970053, 
Output 4: 0.497764, 0.445052, 0.0515744, 0.0056092, 
Output 5: 0.552202, 0.363835, 0.0689791, 0.0149846, 
Output 6: 0.0232487, 0.0563334, 0.0712698, 0.849148, 

07-Sep-2025 09:04:59 UTC  Closed output file "mydetsimreco1reco2_20250907T020225Z.root"
07-Sep-2025 09:04:59 UTC  Closed input file "matt_reco1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1552.82       1728.89       2090.5        1719.45       137.177        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00200753    0.00392011    0.00502458    0.00413028    0.000990472      10     
reco:pandora:StandardPandora                            1551.57       1727.37       2086.84       1718.16       136.549        10     
reco:pandoraTrack:LArPandoraTrackCreation              0.001972      0.0252467     0.128601     0.00922751     0.0368296       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00185866     0.0051591     0.0139498    0.00461081     0.0032399       10     
reco:pandoracalo:Calorimetry                          0.000225971   0.00626919     0.0287214    0.000387808    0.0108832       10     
reco:pandorapid:Chi2ParticleID                        7.0241e-05    0.000341627   0.00231644    9.0171e-05    0.000661294      10     
reco:cvnmap:CVNMapper                                  0.0285324      0.03301      0.0543808     0.0309531    0.00731711       10     
reco:cvneva:CVNEvaluator                               0.864305       1.09072       3.00664      0.879594      0.638692        10     
reco:energyrecnumu:EnergyReco                         0.00423465     0.0241237     0.0522309     0.0114632     0.0195383       10     
reco:energyrecnue:EnergyReco                          0.000941327   0.00309257    0.00755337    0.00126855    0.00278921       10     
reco:energyrecnc:EnergyReco                           0.00696617     0.037823      0.0683768     0.047774      0.0209437       10     
[art]:TriggerResults:TriggerResultInserter             1.49e-05     2.51202e-05   0.000100171   1.6355e-05    2.50855e-05      10     
end_path:out1:RootOutput                               6.191e-06    1.02904e-05   3.8531e-05     6.845e-06    9.45611e-06      10     
end_path:out1:RootOutput(write)                        0.275154      0.291796      0.344691      0.289218      0.0184684       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21766.2 MB
  Peak resident set size usage (VmHWM): 3065.31 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0