Jobsub ID 229902.1@dunegpschedd01.fnal.gov

Jobsub ID	229902.1@dunegpschedd01.fnal.gov
Workflow ID	8430
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-04 12:58:25
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-10-04 19:01:52
From worker node	Hostname	dunegli-7273006-0-fnpc17106.fnal.gov
	cpuinfo	Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-04 12:59:40
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T005453Z_gen_002561_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	6h (21713s)
	CPU time	5h (21534s = 99%)
	Max RSS bytes	3075895296 (2933 MiB)
Outputting started	2025-10-04 19:01:34
Output files
Finished	2025-10-04 19:01:52
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/b1/35/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T005453Z_gen_002561_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max wires due to vertex determination failure: 1364, 1863
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.10317, 
Output 1: 0.0874765, 0.0759962, 0.165282, 0.671245, 
Output 2: 0.0752804, 0.199343, 0.700675, 0.0247018, 
Output 3: 0.963639, 0.0292162, 0.00279111, 0.00435337, 
Output 4: 0.462626, 0.474479, 0.0561036, 0.00679105, 
Output 5: 0.591347, 0.354012, 0.0454459, 0.00919539, 
Output 6: 0.0294476, 0.0868517, 0.0863106, 0.79739, 

Begin processing the 8th record. run: 5028 subRun: 0 event: 25608 at 04-Oct-2025 17:15:02 UTC
Boundary wire vector sizes: 6789, 6778, 6729
minwire 0: 37
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.106971, 
Output 1: 0.185833, 0.0639031, 0.228286, 0.521978, 
Output 2: 0.0310432, 0.0812986, 0.876943, 0.0107154, 
Output 3: 0.961383, 0.0313404, 0.00287827, 0.00439833, 
Output 4: 0.466697, 0.459092, 0.0623989, 0.0118124, 
Output 5: 0.262471, 0.448123, 0.221085, 0.0683217, 
Output 6: 0.108237, 0.183338, 0.117277, 0.591148, 

Begin processing the 9th record. run: 5028 subRun: 0 event: 25609 at 04-Oct-2025 17:52:03 UTC
Boundary wire vector sizes: 6907, 6735, 6638
minwire 0: 50
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 468, 967
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.174389, 
Output 1: 0.0852017, 0.102859, 0.135462, 0.676477, 
Output 2: 0.0565512, 0.290724, 0.616543, 0.0361817, 
Output 3: 0.972805, 0.0171048, 0.00314941, 0.00694104, 
Output 4: 0.498002, 0.433604, 0.0622201, 0.00617342, 
Output 5: 0.387091, 0.480725, 0.11475, 0.0174344, 
Output 6: 0.0257684, 0.0662859, 0.0835079, 0.824438, 

Begin processing the 10th record. run: 5028 subRun: 0 event: 25610 at 04-Oct-2025 18:26:08 UTC
Error: A CaloHitList is empty
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  04-Oct-2025 19:01:25 UTC run: 5028 subRun: 0 event: 25610
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6559, 6569, 6626
minwire 0: 58
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 176, 675
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.106727, 
Output 1: 0.0618064, 0.18198, 0.168829, 0.587385, 
Output 2: 0.0829728, 0.270825, 0.599929, 0.0462733, 
Output 3: 0.948818, 0.0354576, 0.00629437, 0.00942999, 
Output 4: 0.73544, 0.23429, 0.0263325, 0.00393703, 
Output 5: 0.672828, 0.273966, 0.0431752, 0.0100305, 
Output 6: 0.0197843, 0.0531112, 0.0636781, 0.863426, 

04-Oct-2025 19:01:30 UTC  Closed output file "mydetsimreco1reco2_20251004T125951Z.root"
04-Oct-2025 19:01:30 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/b1/35/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T005453Z_gen_002561_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1995.85       2152.4        2284.01       2174.34       87.3515        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00208539    0.00882974     0.0189484    0.00750181     0.0047626       10     
reco:pandora:StandardPandora                            1993.48       2149.39       2281.45       2171.84       87.7498        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00283947     0.0457954     0.199352      0.0414791     0.0541306       10     
reco:pandoraShower:LArPandoraModularShowerCreation     0.0021071    0.00780965     0.0308589    0.00617647    0.00790559       10     
reco:pandoracalo:Calorimetry                          0.000571269   0.00533161     0.0400775    0.000718468    0.0117549       10     
reco:pandorapid:Chi2ParticleID                        0.000131669   0.000467994   0.00319384    0.000162833   0.000908799      10     
reco:cvnmap:CVNMapper                                  0.0549704     0.0732501     0.118099      0.0719703     0.0164943       10     
reco:cvneva:CVNEvaluator                                1.38701       1.9901        6.54829       1.5051        1.52129        10     
reco:energyrecnumu:EnergyReco                         0.00838123     0.0944983     0.152616      0.127256      0.0595821       10     
reco:energyrecnue:EnergyReco                           0.0014832    0.00382122      0.01484     0.00239963    0.00383005       10     
reco:energyrecnc:EnergyReco                           0.00982599     0.0697078     0.168209      0.0153671     0.0687811       10     
[art]:TriggerResults:TriggerResultInserter            1.7681e-05    4.18664e-05   0.000103272   3.10655e-05    2.527e-05       10     
end_path:out1:RootOutput                               8.337e-06    1.42282e-05   4.0031e-05     1.062e-05    9.00906e-06      10     
end_path:out1:RootOutput(write)                        0.669546      0.714657      0.855429      0.698987      0.0503901       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14258 MB
  Peak resident set size usage (VmHWM): 3075.9 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0