Jobsub ID 224046.0@dunegpschedd02.fnal.gov

Jobsub ID	224046.0@dunegpschedd02.fnal.gov
Workflow ID	8430
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-04 01:49:50
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-10-04 03:36:04
From worker node	Hostname	dunegli-46731-0-cmswn4001.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-04 01:50:27
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T005153Z_gen_004586_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	1h (6323s)
	CPU time	1h (6243s = 98%)
	Max RSS bytes	3043086336 (2902 MiB)
Outputting started	2025-10-04 03:35:51
Output files
Finished	2025-10-04 03:36:04
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/9f/37/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T005153Z_gen_004586_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 75, 574
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.128972, 
Output 1: 0.105536, 0.22337, 0.159126, 0.511967, 
Output 2: 0.0617583, 0.250331, 0.642599, 0.0453125, 
Output 3: 0.969686, 0.0149746, 0.00361453, 0.011725, 
Output 4: 0.840419, 0.145182, 0.0122645, 0.00213428, 
Output 5: 0.526147, 0.351224, 0.0958031, 0.0268263, 
Output 6: 0.0215714, 0.0527796, 0.0634173, 0.862232, 

Begin processing the 8th record. run: 4901 subRun: 0 event: 45858 at 04-Oct-2025 03:05:29 UTC
Boundary wire vector sizes: 6502, 6479, 6516
minwire 0: 46
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2235, 2734
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0851767, 
Output 1: 0.080584, 0.0461626, 0.142497, 0.730756, 
Output 2: 0.0437633, 0.143473, 0.791831, 0.020933, 
Output 3: 0.951339, 0.0267023, 0.00643526, 0.0155239, 
Output 4: 0.571207, 0.361619, 0.0561152, 0.0110582, 
Output 5: 0.492786, 0.39436, 0.0924801, 0.0203741, 
Output 6: 0.0260181, 0.0628631, 0.0680659, 0.843053, 

Begin processing the 9th record. run: 4901 subRun: 0 event: 45859 at 04-Oct-2025 03:15:30 UTC
Boundary wire vector sizes: 6710, 6623, 6672
minwire 0: 54
minwire 1: 1
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.100058, 
Output 1: 0.0559126, 0.111195, 0.225182, 0.60771, 
Output 2: 0.0534325, 0.196503, 0.72078, 0.0292844, 
Output 3: 0.948661, 0.0325186, 0.00641538, 0.0124053, 
Output 4: 0.325797, 0.618945, 0.0515992, 0.00365833, 
Output 5: 0.693759, 0.25478, 0.0411766, 0.0102844, 
Output 6: 0.0115547, 0.0363952, 0.049551, 0.902499, 

Begin processing the 10th record. run: 4901 subRun: 0 event: 45860 at 04-Oct-2025 03:25:36 UTC
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  04-Oct-2025 03:35:49 UTC run: 4901 subRun: 0 event: 45860
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6665, 6671, 6758
minwire 0: 18
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.183921, 
Output 1: 0.0888336, 0.132612, 0.169931, 0.608623, 
Output 2: 0.0349581, 0.202248, 0.728181, 0.0346123, 
Output 3: 0.969821, 0.0103129, 0.00359941, 0.0162671, 
Output 4: 0.45897, 0.47859, 0.0567929, 0.00564655, 
Output 5: 0.324643, 0.387901, 0.225059, 0.0623962, 
Output 6: 0.0301538, 0.0548524, 0.0697882, 0.845205, 

04-Oct-2025 03:35:50 UTC  Closed output file "mydetsimreco1reco2_20251004T015031Z.root"
04-Oct-2025 03:35:50 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/9f/37/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T005153Z_gen_004586_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              598.736       624.093       644.184       626.278       15.4329        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00117139    0.00648689     0.0231398     0.0042197     0.0065458       10     
reco:pandora:StandardPandora                            597.771       622.929        643.2        625.271       15.3438        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00022694     0.0130001     0.0563117    0.00855373     0.0160006       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000197384   0.00272752     0.0113054    0.00127926    0.00318044       10     
reco:pandoracalo:Calorimetry                          0.000136839   0.000790555   0.00349938    0.000292619   0.00107924       10     
reco:pandorapid:Chi2ParticleID                         4.215e-05    0.000142233   0.000810687   7.15755e-05   0.000223226      10     
reco:cvnmap:CVNMapper                                  0.0162413     0.0221463     0.0387701     0.0211912    0.00654924       10     
reco:cvneva:CVNEvaluator                               0.670543      0.813287       1.97847      0.680885       0.38874        10     
reco:energyrecnumu:EnergyReco                         0.00480967     0.073728      0.423007      0.0502286     0.118972        10     
reco:energyrecnue:EnergyReco                          0.000745092   0.00170937    0.00470751    0.000876141    0.0014951       10     
reco:energyrecnc:EnergyReco                           0.00448136     0.0237629     0.0609824    0.00488237     0.0240627       10     
[art]:TriggerResults:TriggerResultInserter             7.484e-06    1.65374e-05   7.4311e-05    1.00295e-05   1.9344e-05       10     
end_path:out1:RootOutput                               4.889e-06    9.1043e-06    2.4397e-05    6.7535e-06    5.99811e-06      10     
end_path:out1:RootOutput(write)                        0.189356      0.205313      0.237337      0.200797      0.0125293       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21784.3 MB
  Peak resident set size usage (VmHWM): 3043.09 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0