Jobsub ID 229872.0@dunegpschedd01.fnal.gov

Jobsub ID	229872.0@dunegpschedd01.fnal.gov
Workflow ID	8431
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-04 12:00:22
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-10-04 17:46:41
From worker node	Hostname	dunegli-7270795-0-fnpc17112.fnal.gov
	cpuinfo	Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-04 12:01:34
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T023924Z_gen_001729_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (20693s)
	CPU time	5h (20498s = 99%)
	Max RSS bytes	3056123904 (2914 MiB)
Outputting started	2025-10-04 17:46:27
Output files
Finished	2025-10-04 17:46:41
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/99/22/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T023924Z_gen_001729_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6621, 6643, 6560
minwire 0: 60
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2147, 2646
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.130926, 
Output 1: 0.0501674, 0.0901877, 0.0599449, 0.7997, 
Output 2: 0.0764104, 0.349585, 0.542843, 0.0311616, 
Output 3: 0.979092, 0.0138267, 0.00149056, 0.0055911, 
Output 4: 0.729644, 0.232253, 0.0331387, 0.00496434, 
Output 5: 0.365452, 0.543629, 0.0813567, 0.00956289, 
Output 6: 0.0161913, 0.0399419, 0.0628075, 0.881059, 

Begin processing the 8th record. run: 5019 subRun: 0 event: 17288 at 04-Oct-2025 16:06:16 UTC
Boundary wire vector sizes: 6508, 6455, 6394
minwire 0: 21
minwire 1: 1
minwire 2: 3
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.148963, 
Output 1: 0.114374, 0.122786, 0.112399, 0.65044, 
Output 2: 0.0434958, 0.243262, 0.686692, 0.0265498, 
Output 3: 0.975143, 0.0118047, 0.00219761, 0.0108546, 
Output 4: 0.582913, 0.374234, 0.0384015, 0.00445167, 
Output 5: 0.291517, 0.504992, 0.170037, 0.0334539, 
Output 6: 0.0273177, 0.0620137, 0.0801386, 0.83053, 

Begin processing the 9th record. run: 5019 subRun: 0 event: 17289 at 04-Oct-2025 16:40:17 UTC
Boundary wire vector sizes: 6528, 6577, 6580
minwire 0: 38
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 902, 1401
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0931058, 
Output 1: 0.0534634, 0.200306, 0.163138, 0.583093, 
Output 2: 0.0955145, 0.25689, 0.611306, 0.0362897, 
Output 3: 0.964542, 0.0270896, 0.00265386, 0.00571419, 
Output 4: 0.584886, 0.393003, 0.020429, 0.001682, 
Output 5: 0.635861, 0.319891, 0.0360094, 0.00823861, 
Output 6: 0.0159877, 0.0530766, 0.0631321, 0.867804, 

Begin processing the 10th record. run: 5019 subRun: 0 event: 17290 at 04-Oct-2025 17:13:35 UTC
Boundary wire vector sizes: 6930, 6750, 6782
minwire 0: 56
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0821725, 
Output 1: 0.0836581, 0.0509531, 0.158701, 0.706688, 
Output 2: 0.0732822, 0.138607, 0.769339, 0.0187721, 
Output 3: 0.945615, 0.0475351, 0.00323705, 0.0036126, 
Output 4: 0.372782, 0.542859, 0.0740553, 0.0103038, 
Output 5: 0.510977, 0.394755, 0.0753547, 0.0189126, 
Output 6: 0.0876129, 0.172624, 0.117231, 0.622533, 

04-Oct-2025 17:46:26 UTC  Closed output file "mydetsimreco1reco2_20251004T120142Z.root"
04-Oct-2025 17:46:26 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/99/22/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T023924Z_gen_001729_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1660.52       2049.3        2523.06       2028.6        202.887        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00369692    0.00735414     0.0151406    0.00650309    0.00307991       10     
reco:pandora:StandardPandora                            1658.05       2046.76        2516         2026.61       201.826        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00325644     0.0245347     0.0438552     0.0260989     0.0133453       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000603642   0.00737497     0.036835     0.00513388     0.0100927       10     
reco:pandoracalo:Calorimetry                          0.000574025   0.00627468     0.0363942    0.000833049    0.0106119       10     
reco:pandorapid:Chi2ParticleID                        0.000118417   0.000538277   0.00357572    0.000184202   0.00101548       10     
reco:cvnmap:CVNMapper                                  0.058375      0.0635404     0.0924908     0.0604133     0.0097745       10     
reco:cvneva:CVNEvaluator                               0.938395       1.71321       5.78118       1.38261       1.37034        10     
reco:energyrecnumu:EnergyReco                          0.0082111     0.072464      0.133429      0.0925342     0.0479884       10     
reco:energyrecnue:EnergyReco                          0.00167727    0.00424876     0.0141271    0.00223401     0.0040714       10     
reco:energyrecnc:EnergyReco                            0.0100857     0.0700824     0.136939      0.0851904     0.0481795       10     
[art]:TriggerResults:TriggerResultInserter            2.0151e-05    3.58851e-05   0.000127933   2.3789e-05    3.10803e-05      10     
end_path:out1:RootOutput                               7.892e-06    1.25557e-05   4.3892e-05    9.5095e-06    1.04817e-05      10     
end_path:out1:RootOutput(write)                        0.398561      0.568838      0.863585      0.532639      0.163565        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14272.2 MB
  Peak resident set size usage (VmHWM): 3056.12 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0