justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 229869.0@dunegpschedd01.fnal.gov

Jobsub ID229869.0@dunegpschedd01.fnal.gov
Workflow ID8430
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-04 11:34:20
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-04 17:43:15
From worker nodeHostnamedunegli-7270795-0-fnpc17112.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-04 11:35:55
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T021312Z_gen_001407_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time6h (22026s)
CPU time6h (21825s = 99%)
Max RSS bytes3077394432 (2934 MiB)
Outputting started2025-10-04 17:43:01
Output files
Finished2025-10-04 17:43:15
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/1d/88/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T021312Z_gen_001407_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Output 2: 0.0729785, 0.290752, 0.606856, 0.0294134, 
Output 3: 0.965274, 0.0252689, 0.00291903, 0.00653817, 
Output 4: 0.405853, 0.539248, 0.0510459, 0.00385259, 
Output 5: 0.502549, 0.435328, 0.0541617, 0.00796171, 
Output 6: 0.0262548, 0.0719755, 0.084291, 0.817479, 

Begin processing the 8th record. run: 4909 subRun: 0 event: 14068 at 04-Oct-2025 15:56:18 UTC
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, m_inputVertexListName, pVertexList) throw STATUS_CODE_NOT_INITIALIZED
    in function: GetHitRegion
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6901, 6829, 6694
minwire 0: 37
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.125756, 
Output 1: 0.0673451, 0.0772813, 0.0993011, 0.756072, 
Output 2: 0.0554579, 0.259888, 0.659504, 0.0251499, 
Output 3: 0.978509, 0.0161784, 0.00141864, 0.00389341, 
Output 4: 0.302167, 0.616465, 0.076025, 0.0053427, 
Output 5: 0.597069, 0.319425, 0.067576, 0.01593, 
Output 6: 0.0223122, 0.0659644, 0.0839667, 0.827757, 

Begin processing the 9th record. run: 4909 subRun: 0 event: 14069 at 04-Oct-2025 16:34:37 UTC
Boundary wire vector sizes: 6677, 6674, 6630
minwire 0: 20
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.104831, 
Output 1: 0.266671, 0.0297503, 0.0777246, 0.625854, 
Output 2: 0.221566, 0.213832, 0.538735, 0.0258667, 
Output 3: 0.982683, 0.0076922, 0.000668408, 0.00895664, 
Output 4: 0.812786, 0.179498, 0.00622407, 0.001492, 
Output 5: 0.187196, 0.785207, 0.0252226, 0.00237414, 
Output 6: 0.0307358, 0.069552, 0.0738109, 0.825901, 

Begin processing the 10th record. run: 4909 subRun: 0 event: 14070 at 04-Oct-2025 17:11:42 UTC
Boundary wire vector sizes: 6936, 6922, 6834
minwire 0: 46
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.116494, 
Output 1: 0.0869063, 0.035435, 0.0827068, 0.794952, 
Output 2: 0.0394714, 0.204764, 0.736794, 0.0189706, 
Output 3: 0.969884, 0.0142486, 0.0031006, 0.0127674, 
Output 4: 0.565792, 0.372326, 0.0536912, 0.00819086, 
Output 5: 0.233706, 0.61144, 0.139032, 0.0158231, 
Output 6: 0.0172755, 0.0490642, 0.0682311, 0.865429, 

04-Oct-2025 17:43:00 UTC  Closed output file "mydetsimreco1reco2_20251004T113603Z.root"
04-Oct-2025 17:43:00 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/1d/88/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T021312Z_gen_001407_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1859.08       2183.35       2455.22       2255.98       201.755        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00192603     0.0123181     0.041055     0.00629507     0.0135759       10     
reco:pandora:StandardPandora                            1857.52       2180.7        2452.82       2253.68       200.946        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00179171     0.0292401     0.142095      0.0164332     0.0397961       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000434939   0.00480546     0.0248592    0.00217303    0.00687762       10     
reco:pandoracalo:Calorimetry                          0.000275159   0.00193033     0.0082674    0.000591203   0.00255157       10     
reco:pandorapid:Chi2ParticleID                        7.5818e-05    0.000422035   0.00197993    0.000142204   0.000616439      10     
reco:cvnmap:CVNMapper                                  0.0494844     0.068325      0.116544      0.0623817     0.0201023       10     
reco:cvneva:CVNEvaluator                               0.890486       1.73193       6.41142       1.32847       1.57587        10     
reco:energyrecnumu:EnergyReco                         0.00906204     0.111733      0.194084      0.134097      0.0559393       10     
reco:energyrecnue:EnergyReco                           0.001603     0.00656287     0.0156731    0.00334222     0.0052765       10     
reco:energyrecnc:EnergyReco                            0.0100388     0.0419477     0.180333      0.0143649     0.0587087       10     
[art]:TriggerResults:TriggerResultInserter            1.9394e-05    3.82938e-05   0.000148526   2.6549e-05    3.68743e-05      10     
end_path:out1:RootOutput                               7.679e-06    1.22289e-05   4.0846e-05     9.167e-06    9.58666e-06      10     
end_path:out1:RootOutput(write)                        0.411814      0.640612      0.897131      0.685562      0.156217        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14271.4 MB
  Peak resident set size usage (VmHWM): 3077.39 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-11-04 15:55:18 UTC       justIN version: 01.05.01