Jobsub ID 224330.0@dunegpschedd02.fnal.gov

Jobsub ID	224330.0@dunegpschedd02.fnal.gov
Workflow ID	8431
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-04 11:40:21
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-10-04 18:08:08
From worker node	Hostname	dunegli-7270795-0-fnpc17112.fnal.gov
	cpuinfo	Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-04 11:41:05
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T161602Z_gen_000576_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	6h (23211s)
	CPU time	6h (22928s = 98%)
	Max RSS bytes	3058212864 (2916 MiB)
Outputting started	2025-10-04 18:07:56
Output files
Finished	2025-10-04 18:08:08
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/dd/f0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T161602Z_gen_000576_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 0: 30
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 233, 732
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.137124, 
Output 1: 0.0608522, 0.0350806, 0.129454, 0.774613, 
Output 2: 0.0722725, 0.208903, 0.684469, 0.0343547, 
Output 3: 0.97368, 0.0183326, 0.00253025, 0.00545702, 
Output 4: 0.31953, 0.605601, 0.0690276, 0.00584107, 
Output 5: 0.518989, 0.384826, 0.0796878, 0.0164973, 
Output 6: 0.0192725, 0.0588218, 0.0723438, 0.849562, 

Begin processing the 8th record. run: 4992 subRun: 0 event: 5758 at 04-Oct-2025 16:24:30 UTC
Boundary wire vector sizes: 6731, 6607, 6707
minwire 0: 28
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 144, 643
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.135698, 
Output 1: 0.102894, 0.120123, 0.142909, 0.634074, 
Output 2: 0.0648605, 0.180904, 0.736824, 0.0174108, 
Output 3: 0.977968, 0.0174372, 0.00110956, 0.00348512, 
Output 4: 0.495029, 0.452728, 0.0457851, 0.00645821, 
Output 5: 0.310346, 0.557028, 0.110251, 0.0223752, 
Output 6: 0.0315744, 0.0890181, 0.0973142, 0.782093, 

Begin processing the 9th record. run: 4992 subRun: 0 event: 5759 at 04-Oct-2025 17:04:45 UTC
Boundary wire vector sizes: 6563, 6522, 6556
minwire 0: 32
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0950773, 
Output 1: 0.16709, 0.0885947, 0.132749, 0.611567, 
Output 2: 0.0565746, 0.15132, 0.779666, 0.0124393, 
Output 3: 0.977298, 0.0190296, 0.000959061, 0.00271322, 
Output 4: 0.404202, 0.543247, 0.0468959, 0.0056556, 
Output 5: 0.379771, 0.510012, 0.0889096, 0.0213066, 
Output 6: 0.0689286, 0.187258, 0.13667, 0.607143, 

Begin processing the 10th record. run: 4992 subRun: 0 event: 5760 at 04-Oct-2025 17:39:59 UTC
Boundary wire vector sizes: 6710, 6626, 6624
minwire 0: 32
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2228, 2727
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.110298, 
Output 1: 0.0551399, 0.0494628, 0.16829, 0.727107, 
Output 2: 0.078152, 0.202087, 0.692003, 0.0277581, 
Output 3: 0.945984, 0.0474524, 0.00364691, 0.00291639, 
Output 4: 0.32476, 0.582241, 0.085113, 0.007886, 
Output 5: 0.631069, 0.311365, 0.0480795, 0.00948608, 
Output 6: 0.0237798, 0.0761262, 0.0815588, 0.818535, 

04-Oct-2025 18:07:54 UTC  Closed output file "mydetsimreco1reco2_20251004T114111Z.root"
04-Oct-2025 18:07:54 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/dd/f0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T161602Z_gen_000576_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1668.29       2300.46       2569.1        2356.45       239.684        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00364253    0.00811155     0.0137808     0.0075443    0.00300323       10     
reco:pandora:StandardPandora                            1666.29       2297.45       2566.64       2353.17       239.458        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00561221     0.0379618      0.1569       0.0336838     0.0421783       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000578324    0.0074265     0.0278029    0.00578001    0.00729348       10     
reco:pandoracalo:Calorimetry                          0.00043006     0.0103586     0.0457102    0.00199334     0.0154581       10     
reco:pandorapid:Chi2ParticleID                        0.000102653   0.000541729   0.00249242    0.000200499   0.000752997      10     
reco:cvnmap:CVNMapper                                  0.0609713     0.0768553     0.0992507     0.0723866     0.0135001       10     
reco:cvneva:CVNEvaluator                                1.08956       1.98083       6.88038       1.48229       1.63819        10     
reco:energyrecnumu:EnergyReco                          0.0172892     0.0942795     0.177088      0.101788      0.0582941       10     
reco:energyrecnue:EnergyReco                          0.00155405    0.00854347     0.0227855    0.00348954    0.00772459       10     
reco:energyrecnc:EnergyReco                           0.00953582     0.0631254     0.160621      0.0215261     0.058465        10     
[art]:TriggerResults:TriggerResultInserter            1.8326e-05    3.97427e-05   0.000132078   3.02695e-05   3.12351e-05      10     
end_path:out1:RootOutput                               9.751e-06    1.56645e-05    3.805e-05    1.2229e-05    8.20775e-06      10     
end_path:out1:RootOutput(write)                         0.39262      0.727504      0.891748      0.745175      0.131055        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14236.4 MB
  Peak resident set size usage (VmHWM): 3058.21 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0