Jobsub ID 265279.27@dunegpschedd01.fnal.gov

Jobsub ID	265279.27@dunegpschedd01.fnal.gov
Workflow ID	11116
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-12-09 10:51:45
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-12-10 09:07:09
From worker node	Hostname	dunegli-48793-0-cmswn4069.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-12-09 16:29:04
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T233912Z_gen_004839_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	16h (59482s)
	CPU time	16h (58554s = 98%)
	Max RSS bytes	2522284032 (2405 MiB)
Outputting started	2025-12-10 09:00:26
Output files
Finished	2025-12-10 09:07:09
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

sifier summary: 
Output 0: 0.154306, 
Output 1: 0.0817051, 0.190377, 0.152308, 0.57561, 
Output 2: 0.0472235, 0.280631, 0.636125, 0.0360209, 
Output 3: 0.953545, 0.0257131, 0.00634846, 0.0143939, 
Output 4: 0.532001, 0.404033, 0.0576033, 0.00636256, 
Output 5: 0.421492, 0.42325, 0.129379, 0.0258789, 
Output 6: 0.0290188, 0.067557, 0.0850201, 0.818404, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 48390 at 10-Dec-2025 08:14:28 UTC
Boundary wire vector sizes: 6639, 6597, 6662
minwire 0: 44
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 67, 566
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.106264, 
Output 1: 0.104217, 0.0971518, 0.212736, 0.585895, 
Output 2: 0.0361886, 0.130198, 0.819713, 0.0138998, 
Output 3: 0.956649, 0.0325235, 0.00436232, 0.00646518, 
Output 4: 0.45944, 0.471917, 0.0601108, 0.00853251, 
Output 5: 0.279507, 0.552867, 0.143238, 0.0243886, 
Output 6: 0.0493343, 0.125667, 0.104806, 0.720192, 

10-Dec-2025 08:59:51 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 08:59:51 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/f3/fd/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T233912Z_gen_004839_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2716.42       5894.26       8346.78       6317.81        1565          10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00849941     0.0379438     0.159279      0.024493      0.0412058       10     
reco:pandora:StandardPandora                            2715.07       5878.35       8216.59       6313.66       1544.7         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000676255    0.101822      0.609241      0.0363946     0.174955        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00154797     0.0235116     0.0834328    0.00598082     0.0285582       10     
reco:pandoracalo:Calorimetry                          0.00027369     0.0761135     0.329256      0.0529941     0.0958357       10     
reco:pandorapid:Chi2ParticleID                        8.6915e-05    0.00156186    0.00701996    0.000291995   0.00222351       10     
reco:cvnmap:CVNMapper                                  0.0577591     0.278573      0.909435      0.204175      0.236913        10     
reco:cvneva:CVNEvaluator                               0.835969       14.2988       126.235       2.13692       37.316         10     
reco:energyrecnumu:EnergyReco                          0.0437537     0.208778      0.654707      0.146112      0.193103        10     
reco:energyrecnue:EnergyReco                          0.00405967     0.0146214     0.0299401     0.0129334    0.00907416       10     
reco:energyrecnc:EnergyReco                           0.00514122     0.0633214     0.246223      0.0220913     0.0896852       10     
[art]:TriggerResults:TriggerResultInserter             2.079e-05    0.00015644    0.000733974   9.27715e-05   0.000198997      10     
end_path:out1:RootOutput                               8.156e-06    0.000163165   0.00097903    3.71155e-05   0.000284149      10     
end_path:out1:RootOutput(write)                         0.25393      0.795086       1.63419       0.70776       0.36497        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21688.8 MB
  Peak resident set size usage (VmHWM): 2522.28 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 115.275 | hit correction : 1.14201
hit initial : 217.173 | hit correction : 1.1404
hit initial : 257.329 | hit correction : 1.14169
hit initial : 113.445 | hit correction : 1.14085
hit initial : 71.3411 | hit correction : 1.13634
hit initial : 402.783 | hit correction : 1.13973
hit initial : 1993.51 | hit correction : 1.13918
hit initial : 52.9943 | hit correction : 1.13779
hit initial : 1129.8 | hit correction : 1.13667
 *** Total Charge : 6451.41
 Looking at view: 2 | Number of hits in view: 17
Begin processing the 5th record. run: 5026 subRun: 0 event: 48385 at 10-Dec-2025 09:00:13 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 1.57103
Completed MCParticle Loop
True Neutrino Energy: 4.85482
 Size of all hits: 0
Begin processing the 6th record. run: 5026 subRun: 0 event: 48386 at 10-Dec-2025 09:00:15 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  17
Added hit list length: 17 | Max was set to: -3.40282e+38
True Neutron Energy: 939.567
True Neutron Energy: 939.719
True Electron Energy: 22.0211
Completed MCParticle Loop
True Neutrino Energy: 30.9812
 Size of all hits: 17
hit initial : 228.236 | hit correction : 1.37818
hit initial : 24.8097 | hit correction : 1.38102
hit initial : 125.683 | hit correction : 1.3792
hit initial : 259.577 | hit correction : 1.37912
hit initial : 272.882 | hit correction : 1.37893
hit initial : 282.501 | hit correction : 1.37854
 *** Total Charge : 1645.87
 Looking at view: 0 | Number of hits in view: 6
hit initial : 71.3403 | hit correction : 1.37926
hit initial : 175.374 | hit correction : 1.37919
hit initial : 201.114 | hit correction : 1.37908
hit initial : 177.477 | hit correction : 1.37892
hit initial : 163.339 | hit correction : 1.37871
hit initial : 157.98 | hit correction : 1.37844
 *** Total Charge : 1305.31
 Looking at view: 1 | Number of hits in view: 6
hit initial : 101.138 | hit correction : 1.37807
hit initial : 269.809 | hit correction : 1.37824
hit initial : 184.725 | hit correction : 1.37922
hit initial : 327.618 | hit correction : 1.37905
hit initial : 287.043 | hit correction : 1.37876
 *** Total Charge : 1613.58
 Looking at view: 2 | Number of hits in view: 5
Begin processing the 7th record. run: 5026 subRun: 0 event: 48387 at 10-Dec-2025 09:00:17 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
True Electron Energy: 14.6147
Completed MCParticle Loop
True Neutrino Energy: 17.8991
 Size of all hits: 0
Begin processing the 8th record. run: 5026 subRun: 0 event: 48388 at 10-Dec-2025 09:00:19 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.627
True Electron Energy: 39.1684
Completed MCParticle Loop
True Neutrino Energy: 56.4747
 Size of all hits: 0
Begin processing the 9th record. run: 5026 subRun: 0 event: 48389 at 10-Dec-2025 09:00:21 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 3.35902
Completed MCParticle Loop
True Neutrino Energy: 10.0356
 Size of all hits: 0
Begin processing the 10th record. run: 5026 subRun: 0 event: 48390 at 10-Dec-2025 09:00:24 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.576
True Electron Energy: 31.911
Completed MCParticle Loop
True Neutrino Energy: 37.3068
 Size of all hits: 0
10-Dec-2025 09:00:26 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0502289     0.065243      0.0917091     0.0587228     0.0141013       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000596483   0.00205939     0.0120151    0.000821486   0.00334235       10     
end_path:ana:AnalyzeEvents         0.0494249     0.0629055     0.0804871     0.0576845     0.0120978       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19475.9 MB
  Peak resident set size usage (VmHWM): 1528.46 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0