Jobsub ID 224330.2@dunegpschedd02.fnal.gov

Jobsub ID	224330.2@dunegpschedd02.fnal.gov
Workflow ID	8431
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-04 11:40:21
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-10-04 17:31:20
From worker node	Hostname	dunegli-7270795-0-fnpc17112.fnal.gov
	cpuinfo	Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-04 11:41:13
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T091001Z_gen_002235_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (20991s)
	CPU time	5h (20707s = 98%)
	Max RSS bytes	3073265664 (2930 MiB)
Outputting started	2025-10-04 17:31:04
Output files
Finished	2025-10-04 17:31:20
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/50/1b/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T091001Z_gen_002235_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 1: 1
minwire 2: 3
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 406, 905
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.089465, 
Output 1: 0.0817015, 0.0661094, 0.104177, 0.748012, 
Output 2: 0.0563727, 0.31669, 0.576019, 0.050918, 
Output 3: 0.943363, 0.0251516, 0.00858966, 0.0228954, 
Output 4: 0.624344, 0.32976, 0.0415714, 0.0043243, 
Output 5: 0.821711, 0.144517, 0.0265695, 0.00720226, 
Output 6: 0.0112526, 0.0294418, 0.0455883, 0.913717, 

Begin processing the 8th record. run: 4991 subRun: 0 event: 22348 at 04-Oct-2025 15:48:01 UTC
Boundary wire vector sizes: 6732, 6650, 6559
minwire 0: 30
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 783, 1282
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.126915, 
Output 1: 0.120474, 0.0718153, 0.141842, 0.665869, 
Output 2: 0.0689296, 0.16552, 0.729327, 0.0362232, 
Output 3: 0.980009, 0.0104716, 0.00176466, 0.0077546, 
Output 4: 0.682979, 0.293695, 0.0202176, 0.00310763, 
Output 5: 0.365637, 0.425181, 0.157084, 0.0520982, 
Output 6: 0.0396411, 0.0895807, 0.0851335, 0.785645, 

Begin processing the 9th record. run: 4991 subRun: 0 event: 22349 at 04-Oct-2025 16:26:08 UTC
Boundary wire vector sizes: 6813, 6826, 6793
minwire 0: 24
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1629, 2128
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.101489, 
Output 1: 0.0500797, 0.0488255, 0.0825307, 0.818564, 
Output 2: 0.0927529, 0.248176, 0.625751, 0.0333203, 
Output 3: 0.962271, 0.0293637, 0.002847, 0.00551806, 
Output 4: 0.670338, 0.279042, 0.0426944, 0.00792597, 
Output 5: 0.538354, 0.387441, 0.0624273, 0.0117778, 
Output 6: 0.0161271, 0.0506531, 0.0684586, 0.864761, 

Begin processing the 10th record. run: 4991 subRun: 0 event: 22350 at 04-Oct-2025 16:58:54 UTC
Boundary wire vector sizes: 6439, 6375, 6524
minwire 0: 66
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1872, 2371
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.116547, 
Output 1: 0.0765342, 0.0651792, 0.117992, 0.740295, 
Output 2: 0.0385532, 0.187915, 0.751068, 0.0224637, 
Output 3: 0.967307, 0.0242322, 0.00293601, 0.00552451, 
Output 4: 0.357703, 0.552661, 0.0819534, 0.00768239, 
Output 5: 0.434121, 0.360177, 0.161958, 0.0437444, 
Output 6: 0.0409964, 0.0988907, 0.101237, 0.758876, 

04-Oct-2025 17:31:02 UTC  Closed output file "mydetsimreco1reco2_20251004T114121Z.root"
04-Oct-2025 17:31:02 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/50/1b/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T091001Z_gen_002235_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1901.83       2080.98       2281.1        2103.36       134.001        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00206844     0.009039      0.0382054    0.00646775    0.00983393       10     
reco:pandora:StandardPandora                            1899.56       2078.44       2278.92       2099.32       134.013        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000601247    0.0428291     0.120586      0.0432298     0.0309856       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00120674    0.00631848     0.021032     0.00530044    0.00547601       10     
reco:pandoracalo:Calorimetry                          0.000321464   0.00108458    0.00414525    0.000692926   0.00108585       10     
reco:pandorapid:Chi2ParticleID                        0.000115174   0.000388494   0.00155407     0.0001869    0.000436044      10     
reco:cvnmap:CVNMapper                                  0.0530617     0.0704392     0.120189       0.07074      0.0192093       10     
reco:cvneva:CVNEvaluator                               0.932502       1.61875       4.59578       1.41141       1.01192        10     
reco:energyrecnumu:EnergyReco                         0.00272428     0.0855178     0.174131      0.097645      0.0636116       10     
reco:energyrecnue:EnergyReco                          0.00153954     0.0022887    0.00341849    0.00232585    0.000600107      10     
reco:energyrecnc:EnergyReco                            0.0100724     0.0583894     0.134248      0.0149925     0.056452        10     
[art]:TriggerResults:TriggerResultInserter            1.8602e-05    3.4764e-05    8.4183e-05    2.4627e-05    1.92499e-05      10     
end_path:out1:RootOutput                               8.191e-06    1.41091e-05   2.7381e-05    1.11135e-05   6.79239e-06      10     
end_path:out1:RootOutput(write)                        0.402423      0.634414       0.84881      0.678415      0.140055        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14272.4 MB
  Peak resident set size usage (VmHWM): 3073.27 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0