Jobsub ID 229606.1@dunegpschedd01.fnal.gov

Jobsub ID	229606.1@dunegpschedd01.fnal.gov
Workflow ID	8431
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-04 01:46:33
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-10-04 03:35:20
From worker node	Hostname	dunegli-46731-0-cmswn4001.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-04 01:48:39
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250125T004335Z_gen_000422_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	1h (6389s)
	CPU time	1h (6299s = 98%)
	Max RSS bytes	3025223680 (2885 MiB)
Outputting started	2025-10-04 03:35:09
Output files
Finished	2025-10-04 03:35:20
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7c/6f/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250125T004335Z_gen_000422_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Output 3: 0.986582, 0.00705162, 0.000585601, 0.00578095, 
Output 4: 0.339479, 0.616458, 0.0398414, 0.00422171, 
Output 5: 0.418427, 0.447736, 0.0977497, 0.0360874, 
Output 6: 0.0364516, 0.0758479, 0.0848063, 0.802894, 

Begin processing the 9th record. run: 4881 subRun: 0 event: 4219 at 04-Oct-2025 03:14:43 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6425, 6280, 6350
minwire 0: 17
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1135, 1634
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.124586, 
Output 1: 0.0655828, 0.106324, 0.128145, 0.699948, 
Output 2: 0.0202882, 0.155602, 0.811857, 0.0122521, 
Output 3: 0.96829, 0.0142257, 0.00350791, 0.0139768, 
Output 4: 0.308619, 0.568281, 0.110595, 0.0125059, 
Output 5: 0.265672, 0.501306, 0.198792, 0.0342303, 
Output 6: 0.0343015, 0.0716752, 0.0846061, 0.809417, 

Begin processing the 10th record. run: 4881 subRun: 0 event: 4220 at 04-Oct-2025 03:24:51 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, m_inputVertexListName, pVertexList) throw STATUS_CODE_NOT_INITIALIZED
    in function: GetHitRegion
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6935, 6794, 6722
minwire 0: 59
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.156726, 
Output 1: 0.0707247, 0.0995893, 0.0954812, 0.734205, 
Output 2: 0.0928419, 0.227512, 0.659955, 0.0196904, 
Output 3: 0.986453, 0.0106447, 0.000422176, 0.00248058, 
Output 4: 0.223502, 0.720575, 0.0522788, 0.00364413, 
Output 5: 0.310313, 0.575516, 0.0942475, 0.0199239, 
Output 6: 0.0363783, 0.089857, 0.106063, 0.767702, 

04-Oct-2025 03:35:08 UTC  Closed output file "mydetsimreco1reco2_20251004T014842Z.root"
04-Oct-2025 03:35:08 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7c/6f/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250125T004335Z_gen_000422_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              605.235       629.92        658.134       630.605       16.8931        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.001512     0.00766399     0.021781      0.0055075    0.00617547       10     
reco:pandora:StandardPandora                            604.249       628.743       656.996       628.909       16.8152        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00057534     0.0155213     0.0628118     0.0104804     0.0178156       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000223715    0.0115826     0.0884284    0.00214712     0.0258098       10     
reco:pandoracalo:Calorimetry                          0.000234815   0.00394043     0.0145709    0.00164423    0.00494379       10     
reco:pandorapid:Chi2ParticleID                        4.1418e-05    0.000275934   0.00085989    7.76915e-05   0.000292408      10     
reco:cvnmap:CVNMapper                                  0.0161204     0.0289902     0.0702805     0.0247026     0.0153334       10     
reco:cvneva:CVNEvaluator                               0.678114      0.826685       1.90356      0.694443      0.360066        10     
reco:energyrecnumu:EnergyReco                         0.00383879     0.0524541     0.0820535     0.0599589     0.0251731       10     
reco:energyrecnue:EnergyReco                          0.000756814   0.00289126    0.00534556    0.00300988    0.00191822       10     
reco:energyrecnc:EnergyReco                           0.00435736     0.0167705     0.0678166    0.00463461     0.0243158       10     
[art]:TriggerResults:TriggerResultInserter            1.0119e-05    1.98847e-05   5.8581e-05    1.6882e-05    1.33106e-05      10     
end_path:out1:RootOutput                               5.341e-06    1.07955e-05   2.6069e-05     8.636e-06    5.88884e-06      10     
end_path:out1:RootOutput(write)                        0.188499      0.209167      0.243863      0.204974      0.0193392       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21760.1 MB
  Peak resident set size usage (VmHWM): 3025.22 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0