Jobsub ID 224405.0@dunegpschedd02.fnal.gov

Jobsub ID	224405.0@dunegpschedd02.fnal.gov
Workflow ID	8430
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-04 16:18:36
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-10-04 21:49:55
From worker node	Hostname	dunegli-7273006-0-fnpc17106.fnal.gov
	cpuinfo	Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-04 16:20:34
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250115T225751Z_gen_000749_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (19745s)
	CPU time	5h (19588s = 99%)
	Max RSS bytes	3056959488 (2915 MiB)
Outputting started	2025-10-04 21:49:40
Output files
Finished	2025-10-04 21:49:55
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/08/c6/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250115T225751Z_gen_000749_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6887, 6583, 6569
minwire 0: 14
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2218, 2717
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.137291, 
Output 1: 0.196404, 0.1028, 0.374399, 0.326398, 
Output 2: 0.0382094, 0.0663604, 0.88361, 0.0118206, 
Output 3: 0.946987, 0.0449789, 0.00402565, 0.00400803, 
Output 4: 0.384224, 0.519604, 0.0785647, 0.0176074, 
Output 5: 0.292959, 0.418266, 0.202749, 0.0860258, 
Output 6: 0.171103, 0.268142, 0.132287, 0.428469, 

Begin processing the 9th record. run: 4694 subRun: 0 event: 7489 at 04-Oct-2025 20:46:25 UTC
Boundary wire vector sizes: 6719, 6539, 6558
minwire 0: 55
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 3, 502
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.158673, 
Output 1: 0.0673393, 0.133452, 0.20926, 0.589948, 
Output 2: 0.0351795, 0.157101, 0.782553, 0.0251667, 
Output 3: 0.971679, 0.0190739, 0.00338801, 0.00585936, 
Output 4: 0.404085, 0.517485, 0.0710558, 0.00737411, 
Output 5: 0.322418, 0.393233, 0.22296, 0.0613892, 
Output 6: 0.0440761, 0.101281, 0.0969497, 0.757693, 

Begin processing the 10th record. run: 4694 subRun: 0 event: 7490 at 04-Oct-2025 21:18:50 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 7009, 6760, 6687
minwire 0: 11
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1837, 2336
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.135768, 
Output 1: 0.0538167, 0.0638302, 0.121598, 0.760755, 
Output 2: 0.0460635, 0.146017, 0.787405, 0.0205139, 
Output 3: 0.980235, 0.0130943, 0.0014227, 0.00524767, 
Output 4: 0.28285, 0.641612, 0.068983, 0.00655589, 
Output 5: 0.301025, 0.439039, 0.200517, 0.0594183, 
Output 6: 0.0363386, 0.0897201, 0.0957539, 0.778187, 

04-Oct-2025 21:49:37 UTC  Closed output file "mydetsimreco1reco2_20251004T162049Z.root"
04-Oct-2025 21:49:37 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/08/c6/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250115T225751Z_gen_000749_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1803.8        1955.19       2096.17       1966.84       92.4632        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0015257    0.00787292     0.0159791    0.00637559    0.00420372       10     
reco:pandora:StandardPandora                            1801.25       1952.42       2090.16       1964.56       91.978         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000861193    0.0270426     0.141611      0.0115478     0.0405772       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00129426    0.00682484     0.027691     0.00558637    0.00722661       10     
reco:pandoracalo:Calorimetry                          0.000325579    0.0062175     0.0320431    0.000667161    0.0102117       10     
reco:pandorapid:Chi2ParticleID                        9.3477e-05    0.000403747   0.00226511    0.00019417    0.000623685      10     
reco:cvnmap:CVNMapper                                  0.0500695     0.0838245     0.171681      0.0678202     0.0356881       10     
reco:cvneva:CVNEvaluator                                1.33655       1.76837       4.71682       1.44613      0.987003        10     
reco:energyrecnumu:EnergyReco                         0.00866416     0.0923076     0.152287      0.123004      0.0569225       10     
reco:energyrecnue:EnergyReco                          0.00184827    0.00585228     0.0152351    0.00250233    0.00548812       10     
reco:energyrecnc:EnergyReco                            0.0133894     0.0598799     0.153733      0.0150286     0.0567606       10     
[art]:TriggerResults:TriggerResultInserter            1.7935e-05    4.52212e-05   0.000121188   3.0509e-05    3.55468e-05      10     
end_path:out1:RootOutput                               9.408e-06    1.69576e-05   4.7552e-05    1.1914e-05    1.09421e-05      10     
end_path:out1:RootOutput(write)                        0.621382      0.710244      0.850528       0.70789      0.0761109       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14264.2 MB
  Peak resident set size usage (VmHWM): 3056.96 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0