Jobsub ID 267037.0@dunegpschedd01.fnal.gov

Jobsub ID	267037.0@dunegpschedd01.fnal.gov
Workflow ID	11115
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-12-11 21:52:38
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-12-11 23:39:36
From worker node	Hostname	dunegli-8033740-0-fnpc22020.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-12-11 21:54:04
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T013152Z_gen_003112_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	1h (6319s)
	CPU time	1h (6265s = 99%)
	Max RSS bytes	2958241792 (2821 MiB)
Outputting started	2025-12-11 23:39:24
Output files
Finished	2025-12-11 23:39:36
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

.697898, 0.0310143, 
Output 3: 0.96798, 0.0199209, 0.0034464, 0.00865282, 
Output 4: 0.545809, 0.403594, 0.0454635, 0.00513414, 
Output 5: 0.318681, 0.457971, 0.183424, 0.0399238, 
Output 6: 0.0327501, 0.0806946, 0.0916349, 0.79492, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 31120 at 11-Dec-2025 23:28:20 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
Boundary wire vector sizes: 6868, 6753, 6659
minwire 0: 19
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.174002, 
Output 1: 0.0887261, 0.0624649, 0.151099, 0.69771, 
Output 2: 0.0345482, 0.153819, 0.792421, 0.0192117, 
Output 3: 0.979191, 0.0110019, 0.00177971, 0.0080273, 
Output 4: 0.236016, 0.676848, 0.0803348, 0.00680076, 
Output 5: 0.272615, 0.460756, 0.213704, 0.0529258, 
Output 6: 0.0258693, 0.061711, 0.0793911, 0.833029, 

11-Dec-2025 23:38:54 UTC  Closed output file "matt_reco2.root"
11-Dec-2025 23:38:54 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/d4/60/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T013152Z_gen_003112_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              614.472       622.925       630.787       624.359       5.48539        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00103812    0.00704948     0.0274651    0.00344786    0.00821856       10     
reco:pandora:StandardPandora                            613.478       621.793       629.788       623.169       5.41214        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000204686   0.00678376     0.0483748    0.00249148     0.013911        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000171373   0.00468085     0.0220147    0.00201133     0.0063461       10     
reco:pandoracalo:Calorimetry                          0.000112783   0.00336597     0.015176     0.000350642    0.0058059       10     
reco:pandorapid:Chi2ParticleID                         4.202e-05    0.000197758   0.000749504   6.2102e-05    0.000219886      10     
reco:cvnmap:CVNMapper                                  0.0162156     0.0185692     0.0268692     0.0170471     0.0034225       10     
reco:cvneva:CVNEvaluator                               0.678025      0.804252       1.87706      0.684982      0.357623        10     
reco:energyrecnumu:EnergyReco                         0.00134317     0.0474109     0.0662127     0.0542295     0.019863        10     
reco:energyrecnue:EnergyReco                          0.000812072   0.00282744    0.00504714    0.00282845     0.0019023       10     
reco:energyrecnc:EnergyReco                           0.00453614     0.0162942     0.0716997    0.00462834     0.0236715       10     
[art]:TriggerResults:TriggerResultInserter             8.046e-06    1.6486e-05     6.887e-05    1.0695e-05    1.76094e-05      10     
end_path:out1:RootOutput                               2.875e-06    7.0472e-06    2.2473e-05     4.659e-06    5.62042e-06      10     
end_path:out1:RootOutput(write)                        0.200261      0.219543      0.244089      0.218349      0.0100269       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21691.9 MB
  Peak resident set size usage (VmHWM): 2958.24 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 22.3716 | hit correction : 1.14718
hit initial : 83.4933 | hit correction : 1.14714
hit initial : 61.1927 | hit correction : 1.14712
hit initial : 48.7047 | hit correction : 1.14645
hit initial : 133.102 | hit correction : 1.14651
hit initial : 235.928 | hit correction : 1.14679
hit initial : 463.685 | hit correction : 1.14666
hit initial : 224.137 | hit correction : 1.14687
hit initial : 233.268 | hit correction : 1.1469
hit initial : 233.867 | hit correction : 1.1469
hit initial : 215.133 | hit correction : 1.14693
hit initial : 208.398 | hit correction : 1.14704
hit initial : 219.598 | hit correction : 1.14722
hit initial : 220.861 | hit correction : 1.14738
hit initial : 216.832 | hit correction : 1.1475
hit initial : 222.569 | hit correction : 1.14755
hit initial : 225.188 | hit correction : 1.14759
hit initial : 213.598 | hit correction : 1.14768
hit initial : 194.956 | hit correction : 1.14776
hit initial : 197.884 | hit correction : 1.14778
hit initial : 219.584 | hit correction : 1.14774
hit initial : 244.625 | hit correction : 1.14765
hit initial : 239.473 | hit correction : 1.14753
hit initial : 143.992 | hit correction : 1.14745
hit initial : 31.168 | hit correction : 1.14742
 *** Total Charge : 5453.5
 Looking at view: 1 | Number of hits in view: 25
hit initial : 36.797 | hit correction : 1.14718
hit initial : 254.514 | hit correction : 1.14718
hit initial : 165.135 | hit correction : 1.14709
hit initial : 138.755 | hit correction : 1.14694
hit initial : 144.059 | hit correction : 1.14675
hit initial : 203.909 | hit correction : 1.14648
hit initial : 204.507 | hit correction : 1.14648
hit initial : 135.881 | hit correction : 1.14644
hit initial : 154.578 | hit correction : 1.14669
hit initial : 164.934 | hit correction : 1.14653
hit initial : 148.84 | hit correction : 1.14684
hit initial : 155.818 | hit correction : 1.14688
hit initial : 186.758 | hit correction : 1.14688
hit initial : 167.837 | hit correction : 1.1469
hit initial : 129.912 | hit correction : 1.1469
hit initial : 141.657 | hit correction : 1.14689
hit initial : 139.818 | hit correction : 1.14692
hit initial : 139.426 | hit correction : 1.14699
hit initial : 141.721 | hit correction : 1.14712
hit initial : 169.013 | hit correction : 1.14725
hit initial : 144.454 | hit correction : 1.14738
hit initial : 126.576 | hit correction : 1.14748
hit initial : 132.218 | hit correction : 1.14753
hit initial : 123.833 | hit correction : 1.14755
hit initial : 128.642 | hit correction : 1.14755
hit initial : 147.078 | hit correction : 1.14757
hit initial : 152.572 | hit correction : 1.14765
hit initial : 145.046 | hit correction : 1.14772
hit initial : 137.146 | hit correction : 1.14778
hit initial : 136.029 | hit correction : 1.14781
hit initial : 125.858 | hit correction : 1.1478
 *** Total Charge : 5303.42
 Looking at view: 2 | Number of hits in view: 31
Begin processing the 8th record. run: 5027 subRun: 0 event: 31118 at 11-Dec-2025 23:39:18 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 32.542
Completed MCParticle Loop
True Neutrino Energy: 37.9659
 Size of all hits: 0
Begin processing the 9th record. run: 5027 subRun: 0 event: 31119 at 11-Dec-2025 23:39:19 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
True Electron Energy: 59.278
Completed MCParticle Loop
True Neutrino Energy: 63.1172
 Size of all hits: 0
Begin processing the 10th record. run: 5027 subRun: 0 event: 31120 at 11-Dec-2025 23:39:21 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.577
True Electron Energy: 3.914
Completed MCParticle Loop
True Neutrino Energy: 7.63851
 Size of all hits: 0
11-Dec-2025 23:39:23 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.045413      0.0563623     0.106358      0.0482758     0.0174352       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000296349   0.000441677   0.000746278   0.000354635   0.000167819      10     
end_path:ana:AnalyzeEvents         0.0450201     0.0557847     0.105137      0.0475253     0.0172416       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19477.5 MB
  Peak resident set size usage (VmHWM): 1540.51 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0