Jobsub ID 259479.17@dunegpschedd02.fnal.gov

Jobsub ID	259479.17@dunegpschedd02.fnal.gov
Workflow ID	11111
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-12-09 10:51:44
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-12-10 09:19:39
From worker node	Hostname	dunegli-48793-0-cmswn4069.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-12-09 16:18:36
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T222033Z_gen_003619_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	16h (60857s)
	CPU time	16h (59357s = 97%)
	Max RSS bytes	2485600256 (2370 MiB)
Outputting started	2025-12-10 09:12:54
Output files
Finished	2025-12-10 09:19:39
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

t min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0993316, 
Output 1: 0.0606068, 0.0748817, 0.141444, 0.723067, 
Output 2: 0.055253, 0.225002, 0.693749, 0.0259962, 
Output 3: 0.937075, 0.0498578, 0.0059237, 0.00714384, 
Output 4: 0.282398, 0.618434, 0.0917589, 0.007409, 
Output 5: 0.60322, 0.31496, 0.0670831, 0.0147375, 
Output 6: 0.0329177, 0.0861135, 0.0926207, 0.788348, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 36190 at 10-Dec-2025 08:43:04 UTC
Boundary wire vector sizes: 6696, 6624, 6714
minwire 0: 18
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1450, 1949
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.108516, 
Output 1: 0.0681615, 0.071496, 0.154589, 0.705754, 
Output 2: 0.0293326, 0.147544, 0.805568, 0.0175553, 
Output 3: 0.964076, 0.0236855, 0.00416243, 0.00807657, 
Output 4: 0.335124, 0.555316, 0.0983765, 0.0111838, 
Output 5: 0.495957, 0.336632, 0.13159, 0.0358207, 
Output 6: 0.0283976, 0.078512, 0.0851221, 0.807968, 

10-Dec-2025 09:12:26 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 09:12:26 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/d2/75/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T222033Z_gen_003619_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1758.93       6034.61       8704.24       6713.29       1942.77        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0103028     0.0238979     0.0400098     0.0265287    0.00859247       10     
reco:pandora:StandardPandora                            1757.9        5960.35       8510.79       6643.35       1910.64        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00168682     0.0821069     0.658875     0.00762758     0.194618        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00151602     0.0183794     0.109885     0.00999778     0.0309232       10     
reco:pandoracalo:Calorimetry                          0.000455035    0.0786801     0.207187      0.0593871     0.0724887       10     
reco:pandorapid:Chi2ParticleID                        7.6325e-05    0.00200264     0.015916     0.000404168   0.00465344       10     
reco:cvnmap:CVNMapper                                  0.0204567     0.316959       1.45179      0.197653      0.396278        10     
reco:cvneva:CVNEvaluator                               0.691029       12.9708       113.19        2.22113       33.413         10     
reco:energyrecnumu:EnergyReco                         0.00935538      45.6845       68.4337       63.9041       29.7178        10     
reco:energyrecnue:EnergyReco                          0.000971446    0.0209622     0.0335536     0.0252323     0.0107083       10     
reco:energyrecnc:EnergyReco                           0.00512107      14.098        74.4908      0.0286236      28.203         10     
[art]:TriggerResults:TriggerResultInserter            1.2714e-05    0.000194887   0.000701143   0.000162138   0.000178136      10     
end_path:out1:RootOutput                               6.182e-06    7.80324e-05   0.000256417    6.294e-05    6.69902e-05      10     
end_path:out1:RootOutput(write)                         0.22066      0.957492       2.37605      0.893878      0.563733        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21680.3 MB
  Peak resident set size usage (VmHWM): 2485.6 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 6.32781
Completed MCParticle Loop
True Neutrino Energy: 9.61139
 Size of all hits: 0
Begin processing the 6th record. run: 5026 subRun: 0 event: 36186 at 10-Dec-2025 09:12:44 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 941.371
True Neutron Energy: 939.88
True Neutron Energy: 939.605
True Electron Energy: 5.85425
Completed MCParticle Loop
True Neutrino Energy: 9.58018
 Size of all hits: 0
Begin processing the 7th record. run: 5026 subRun: 0 event: 36187 at 10-Dec-2025 09:12:46 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 56.0517
Completed MCParticle Loop
True Neutrino Energy: 59.4804
 Size of all hits: 0
Begin processing the 8th record. run: 5026 subRun: 0 event: 36188 at 10-Dec-2025 09:12:48 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 25.0039
Completed MCParticle Loop
True Neutrino Energy: 29.8436
 Size of all hits: 0
Begin processing the 9th record. run: 5026 subRun: 0 event: 36189 at 10-Dec-2025 09:12:50 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  27
Added hit list length: 27 | Max was set to: -3.40282e+38
True Neutron Energy: 944.785
True Neutron Energy: 940.03
True Electron Energy: 47.5146
Completed MCParticle Loop
True Neutrino Energy: 67.039
 Size of all hits: 27
hit initial : 77.2255 | hit correction : 1.16895
hit initial : 294.666 | hit correction : 1.16906
hit initial : 341.859 | hit correction : 1.16934
hit initial : 334.82 | hit correction : 1.17003
hit initial : 325.174 | hit correction : 1.17078
hit initial : 208.879 | hit correction : 1.17133
hit initial : 240.139 | hit correction : 1.17237
hit initial : 529.755 | hit correction : 1.17326
hit initial : 114.753 | hit correction : 1.17434
hit initial : 611.618 | hit correction : 1.17441
hit initial : 449.077 | hit correction : 1.17571
 *** Total Charge : 4135.73
 Looking at view: 0 | Number of hits in view: 11
hit initial : 49.1766 | hit correction : 1.17636
hit initial : 459.525 | hit correction : 1.17101
hit initial : 463.966 | hit correction : 1.17001
hit initial : 39.5656 | hit correction : 1.16911
hit initial : 386.014 | hit correction : 1.16914
hit initial : 165.36 | hit correction : 1.169
hit initial : 49.5349 | hit correction : 1.171
hit initial : 331.362 | hit correction : 1.17241
hit initial : 28.1158 | hit correction : 1.1736
hit initial : 553.199 | hit correction : 1.17382
hit initial : 35.8954 | hit correction : 1.1742
hit initial : 60.697 | hit correction : 1.17565
 *** Total Charge : 3072.03
 Looking at view: 1 | Number of hits in view: 12
hit initial : 696.663 | hit correction : 1.16941
hit initial : 1857.84 | hit correction : 1.17281
hit initial : 1059.49 | hit correction : 1.17357
hit initial : 101.751 | hit correction : 1.17562
 *** Total Charge : 4356.56
 Looking at view: 2 | Number of hits in view: 4
Begin processing the 10th record. run: 5026 subRun: 0 event: 36190 at 10-Dec-2025 09:12:51 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.57
True Electron Energy: 12.3202
Completed MCParticle Loop
True Neutrino Energy: 17.7047
 Size of all hits: 0
10-Dec-2025 09:12:53 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0463952     0.0551829     0.0833833     0.0488549     0.0124069       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000365795   0.000496279   0.000864974   0.000437121   0.000159409      10     
end_path:ana:AnalyzeEvents         0.0457468     0.0544912     0.0819111     0.0482814     0.0122244       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19470.9 MB
  Peak resident set size usage (VmHWM): 1527.11 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0