Jobsub ID 224372.1@dunegpschedd02.fnal.gov

Jobsub ID	224372.1@dunegpschedd02.fnal.gov
Workflow ID	8430
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-04 13:14:26
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-10-04 19:12:34
From worker node	Hostname	dunegli-7273006-0-fnpc17106.fnal.gov
	cpuinfo	Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-04 13:15:08
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T093435Z_gen_003576_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (21430s)
	CPU time	5h (21242s = 99%)
	Max RSS bytes	3062980608 (2921 MiB)
Outputting started	2025-10-04 19:12:19
Output files
Finished	2025-10-04 19:12:34
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/51/f0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T093435Z_gen_003576_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6715, 6797, 6666
minwire 0: 36
minwire 1: 1
minwire 2: 3
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.143154, 
Output 1: 0.0707252, 0.0621101, 0.113708, 0.753457, 
Output 2: 0.0653721, 0.281376, 0.621003, 0.0322488, 
Output 3: 0.973812, 0.0190241, 0.00201175, 0.00515241, 
Output 4: 0.300585, 0.640673, 0.0556464, 0.00309636, 
Output 5: 0.474853, 0.424756, 0.0852926, 0.0150976, 
Output 6: 0.0164031, 0.0489066, 0.0697854, 0.864905, 

Begin processing the 8th record. run: 5021 subRun: 0 event: 35758 at 04-Oct-2025 17:22:43 UTC
Boundary wire vector sizes: 6761, 6749, 6654
minwire 0: 26
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.114, 
Output 1: 0.0788589, 0.0936897, 0.195098, 0.632353, 
Output 2: 0.0416532, 0.158319, 0.78197, 0.0180581, 
Output 3: 0.964706, 0.0270201, 0.0036804, 0.00459309, 
Output 4: 0.565001, 0.355119, 0.0681247, 0.0117551, 
Output 5: 0.45213, 0.436085, 0.0953731, 0.0164124, 
Output 6: 0.0336966, 0.0820354, 0.0797236, 0.804544, 

Begin processing the 9th record. run: 5021 subRun: 0 event: 35759 at 04-Oct-2025 17:58:53 UTC
Boundary wire vector sizes: 6675, 6586, 6535
minwire 0: 25
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.115829, 
Output 1: 0.302842, 0.0856095, 0.210545, 0.401004, 
Output 2: 0.0356015, 0.065228, 0.89048, 0.00869003, 
Output 3: 0.977353, 0.0191785, 0.000944514, 0.00252365, 
Output 4: 0.48133, 0.466938, 0.0428344, 0.00889775, 
Output 5: 0.202951, 0.41523, 0.25382, 0.127999, 
Output 6: 0.1996, 0.271509, 0.131875, 0.397016, 

Begin processing the 10th record. run: 5021 subRun: 0 event: 35760 at 04-Oct-2025 18:35:53 UTC
Boundary wire vector sizes: 7074, 7129, 6913
minwire 0: 12
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.143125, 
Output 1: 0.0725167, 0.0903999, 0.193838, 0.643246, 
Output 2: 0.0686316, 0.136693, 0.777802, 0.0168728, 
Output 3: 0.972435, 0.0248083, 0.00113249, 0.00162393, 
Output 4: 0.250301, 0.660515, 0.0807288, 0.00845565, 
Output 5: 0.338705, 0.50194, 0.127493, 0.031862, 
Output 6: 0.0695934, 0.172427, 0.131161, 0.626818, 

04-Oct-2025 19:12:17 UTC  Closed output file "mydetsimreco1reco2_20251004T131524Z.root"
04-Oct-2025 19:12:17 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/51/f0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T093435Z_gen_003576_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1820.05       2124.17       2284.92       2160.31       120.41         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00210105    0.00927762     0.0270401    0.00681615     0.0070375       10     
reco:pandora:StandardPandora                            1817.8        2121.46       2278.38       2157.96       119.835        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00780525     0.0364628     0.160965      0.0193588     0.0434839       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00218128    0.00681384     0.021449     0.00582245     0.0053703       10     
reco:pandoracalo:Calorimetry                          0.000515397   0.00867447     0.0382122    0.000891797    0.014729        10     
reco:pandorapid:Chi2ParticleID                        0.000146566   0.000426302   0.00239006    0.00019754    0.000657408      10     
reco:cvnmap:CVNMapper                                   0.05737      0.0689963     0.100214      0.0649766     0.0125862       10     
reco:cvneva:CVNEvaluator                               0.947151       1.75535       5.16572       1.40721       1.14787        10     
reco:energyrecnumu:EnergyReco                          0.0149011     0.0814876     0.144475      0.0839114     0.0515395       10     
reco:energyrecnue:EnergyReco                          0.00166665    0.00471608     0.0151537    0.00234523    0.00494869       10     
reco:energyrecnc:EnergyReco                            0.010157      0.0733501     0.147318      0.0660341     0.0602164       10     
[art]:TriggerResults:TriggerResultInserter            1.9812e-05    4.23997e-05   0.000120194   3.3603e-05    2.70675e-05      10     
end_path:out1:RootOutput                               9.591e-06    1.55512e-05   3.2211e-05    1.31235e-05   6.53272e-06      10     
end_path:out1:RootOutput(write)                        0.396205      0.668161      0.873525      0.675731       0.11288        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14260.7 MB
  Peak resident set size usage (VmHWM): 3062.98 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0