Jobsub ID 224044.1@dunegpschedd02.fnal.gov

Jobsub ID	224044.1@dunegpschedd02.fnal.gov
Workflow ID	8431
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-04 01:34:49
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-10-04 03:22:25
From worker node	Hostname	dunegli-46731-0-cmswn4001.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-04 01:36:30
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T170840Z_gen_003473_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	1h (6343s)
	CPU time	1h (6230s = 98%)
	Max RSS bytes	3035357184 (2894 MiB)
Outputting started	2025-10-04 03:22:14
Output files
Finished	2025-10-04 03:22:25
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/97/a2/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T170840Z_gen_003473_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Output 6: 0.0361688, 0.0990092, 0.0922098, 0.772612, 

Begin processing the 8th record. run: 5048 subRun: 0 event: 34728 at 04-Oct-2025 02:51:31 UTC
Boundary wire vector sizes: 6697, 6606, 6591
minwire 0: 19
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1221, 1720
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.116769, 
Output 1: 0.0748138, 0.11151, 0.10027, 0.713406, 
Output 2: 0.0758158, 0.218813, 0.686456, 0.018915, 
Output 3: 0.976646, 0.0200797, 0.000854958, 0.00241938, 
Output 4: 0.474887, 0.482769, 0.0384161, 0.00392765, 
Output 5: 0.302029, 0.565826, 0.110991, 0.0211538, 
Output 6: 0.0325377, 0.0976405, 0.105942, 0.76388, 

Begin processing the 9th record. run: 5048 subRun: 0 event: 34729 at 04-Oct-2025 03:01:53 UTC
Boundary wire vector sizes: 6623, 6606, 6583
minwire 0: 19
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1482, 1981
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.126198, 
Output 1: 0.117493, 0.121641, 0.126851, 0.634015, 
Output 2: 0.0258722, 0.156736, 0.78754, 0.0298514, 
Output 3: 0.976768, 0.0143114, 0.00284944, 0.00607132, 
Output 4: 0.852611, 0.127493, 0.0165237, 0.00337154, 
Output 5: 0.209608, 0.247512, 0.381776, 0.161104, 
Output 6: 0.049571, 0.103621, 0.0951965, 0.751611, 

Begin processing the 10th record. run: 5048 subRun: 0 event: 34730 at 04-Oct-2025 03:12:02 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, m_inputVertexListName, pVertexList) throw STATUS_CODE_NOT_INITIALIZED
    in function: GetHitRegion
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 7089, 6978, 6820
minwire 0: 56
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 967, 1466
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.107138, 
Output 1: 0.0693573, 0.092297, 0.120088, 0.718257, 
Output 2: 0.0436767, 0.278186, 0.648505, 0.0296319, 
Output 3: 0.959478, 0.0266506, 0.00480469, 0.00906653, 
Output 4: 0.376063, 0.51695, 0.0983841, 0.00860312, 
Output 5: 0.766553, 0.185645, 0.0382519, 0.00955077, 
Output 6: 0.0199532, 0.0511096, 0.0680363, 0.860901, 

04-Oct-2025 03:22:13 UTC  Closed output file "mydetsimreco1reco2_20251004T013634Z.root"
04-Oct-2025 03:22:13 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/97/a2/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T170840Z_gen_003473_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              606.634       623.399       638.613       623.594       10.2376        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00113199    0.00580407     0.0146584    0.00442262    0.00404222       10     
reco:pandora:StandardPandora                            605.689       622.285       637.498       621.983       10.2007        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00146654     0.0126141     0.0498644    0.00483273     0.0153142       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000595118   0.00326752    0.00878142    0.00311177    0.00224541       10     
reco:pandoracalo:Calorimetry                          0.000174861   0.00317755     0.0137861    0.000308936   0.00527802       10     
reco:pandorapid:Chi2ParticleID                        3.9826e-05    0.000164772   0.00060675    7.6194e-05    0.000192881      10     
reco:cvnmap:CVNMapper                                  0.0156315     0.0223886     0.0272429     0.0235851    0.00391783       10     
reco:cvneva:CVNEvaluator                                0.67011      0.799734       1.82997      0.679678      0.343883        10     
reco:energyrecnumu:EnergyReco                         0.00247478     0.0313584     0.0904065     0.0218648     0.0297452       10     
reco:energyrecnue:EnergyReco                          0.000755743   0.00203131    0.00481537    0.000917474    0.0017252       10     
reco:energyrecnc:EnergyReco                           0.00448982     0.0316346     0.0784617     0.0201548     0.0289227       10     
[art]:TriggerResults:TriggerResultInserter             8.266e-06    1.4151e-05    4.0116e-05    1.0635e-05    9.18644e-06      10     
end_path:out1:RootOutput                               4.799e-06    7.3921e-06    1.8566e-05     6.412e-06    3.79709e-06      10     
end_path:out1:RootOutput(write)                        0.189325      0.201345      0.257529      0.192839      0.0203601       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21748.8 MB
  Peak resident set size usage (VmHWM): 3035.36 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0