Jobsub ID 229872.1@dunegpschedd01.fnal.gov

Jobsub ID	229872.1@dunegpschedd01.fnal.gov
Workflow ID	8431
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-04 12:00:22
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-10-04 17:56:22
From worker node	Hostname	dunegli-7273006-0-fnpc17106.fnal.gov
	cpuinfo	Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-04 12:02:14
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T155141Z_gen_000223_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (21232s)
	CPU time	5h (21053s = 99%)
	Max RSS bytes	3051614208 (2910 MiB)
Outputting started	2025-10-04 17:56:07
Output files
Finished	2025-10-04 17:56:22
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/c8/58/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T155141Z_gen_000223_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Output 5: 0.671069, 0.236685, 0.0668822, 0.0253632, 
Output 6: 0.0108448, 0.0370308, 0.0513983, 0.900726, 

Begin processing the 8th record. run: 5025 subRun: 0 event: 2228 at 04-Oct-2025 16:11:13 UTC
Boundary wire vector sizes: 6465, 6512, 6400
minwire 0: 56
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1291, 1790
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.112465, 
Output 1: 0.045897, 0.225167, 0.188707, 0.540229, 
Output 2: 0.0377935, 0.20052, 0.734448, 0.0272381, 
Output 3: 0.939412, 0.0396437, 0.00917303, 0.0117715, 
Output 4: 0.550284, 0.381973, 0.0599634, 0.0077802, 
Output 5: 0.470221, 0.372517, 0.127939, 0.0293232, 
Output 6: 0.0300364, 0.0778173, 0.0835788, 0.808568, 

Begin processing the 9th record. run: 5025 subRun: 0 event: 2229 at 04-Oct-2025 16:47:27 UTC
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  04-Oct-2025 17:22:57 UTC run: 5025 subRun: 0 event: 2229
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  04-Oct-2025 17:22:57 UTC run: 5025 subRun: 0 event: 2229
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  04-Oct-2025 17:22:57 UTC run: 5025 subRun: 0 event: 2229
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6544, 6411, 6488
minwire 0: 39
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 527, 1026
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0396774, 
Output 1: 0.0234514, 0.0408954, 0.0483715, 0.887282, 
Output 2: 0.0586099, 0.179703, 0.741219, 0.0204676, 
Output 3: 0.936496, 0.0512043, 0.00621541, 0.00608408, 
Output 4: 0.956099, 0.0323945, 0.00705666, 0.00444977, 
Output 5: 0.402894, 0.491513, 0.0933461, 0.0122467, 
Output 6: 0.0131714, 0.0614158, 0.067779, 0.857634, 

Begin processing the 10th record. run: 5025 subRun: 0 event: 2230 at 04-Oct-2025 17:23:00 UTC
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  04-Oct-2025 17:56:01 UTC run: 5025 subRun: 0 event: 2230
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6949, 6914, 6830
minwire 0: 50
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 820, 1319
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.081882, 
Output 1: 0.0384509, 0.0562025, 0.195241, 0.710106, 
Output 2: 0.142837, 0.228682, 0.573307, 0.0551738, 
Output 3: 0.909639, 0.068843, 0.0107074, 0.0108105, 
Output 4: 0.23748, 0.700165, 0.0585207, 0.00383432, 
Output 5: 0.827365, 0.156295, 0.0128956, 0.0034443, 
Output 6: 0.0270107, 0.0742503, 0.0738361, 0.824903, 

04-Oct-2025 17:56:05 UTC  Closed output file "mydetsimreco1reco2_20251004T120224Z.root"
04-Oct-2025 17:56:05 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/c8/58/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T155141Z_gen_000223_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1889.35       2104.13       2219.58       2146.67       111.484        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0022334     0.0104776     0.0223185    0.00769354    0.00654154       10     
reco:pandora:StandardPandora                            1882.1        2101.31       2217.32       2144.26       112.467        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00201587     0.0418523     0.144001      0.0391486     0.0369321       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00170904    0.00807297     0.0325346    0.00449563    0.00919772       10     
reco:pandoracalo:Calorimetry                          0.000370842   0.00119741    0.00646696    0.000602284   0.00176473       10     
reco:pandorapid:Chi2ParticleID                        0.000114589   0.000428354   0.00279406    0.000163564   0.000789275      10     
reco:cvnmap:CVNMapper                                  0.0500981     0.0670838     0.0873384     0.0655499     0.0105114       10     
reco:cvneva:CVNEvaluator                               0.981624       1.80235       5.88014       1.35362       1.3673         10     
reco:energyrecnumu:EnergyReco                         0.00796333     0.127854      0.474866      0.110391      0.141019        10     
reco:energyrecnue:EnergyReco                          0.00157459    0.00225503    0.00372376    0.00214879    0.000538999      10     
reco:energyrecnc:EnergyReco                           0.00953795     0.0818785     0.345676      0.0146749     0.102486        10     
[art]:TriggerResults:TriggerResultInserter            1.9208e-05    3.49694e-05   0.000105831   2.6255e-05    2.48369e-05      10     
end_path:out1:RootOutput                               7.746e-06    1.43976e-05   3.3531e-05    1.04665e-05   7.86472e-06      10     
end_path:out1:RootOutput(write)                         0.39379      0.674516      0.789966       0.67095      0.108377        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14258.1 MB
  Peak resident set size usage (VmHWM): 3051.61 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0