justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 224369.1@dunegpschedd02.fnal.gov

Jobsub ID224369.1@dunegpschedd02.fnal.gov
Workflow ID8430
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-04 13:08:25
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-04 17:24:56
From worker nodeHostnamedunegli-7276676-0-fnpc9008.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-04 13:09:01
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T173738Z_gen_001606_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time4h (15346s)
CPU time4h (15262s = 99%)
Max RSS bytes3066880000 (2924 MiB)
Outputting started2025-10-04 17:24:48
Output files
Finished2025-10-04 17:24:56
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/98/5f/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T173738Z_gen_001606_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Output 3: 0.95972, 0.0330963, 0.00234825, 0.00483579, 
Output 4: 0.292267, 0.659981, 0.0447378, 0.00301431, 
Output 5: 0.643895, 0.328486, 0.0230795, 0.00453993, 
Output 6: 0.0165411, 0.0616202, 0.0755271, 0.846311, 

Begin processing the 8th record. run: 4969 subRun: 0 event: 16058 at 04-Oct-2025 16:12:54 UTC
Boundary wire vector sizes: 6582, 6554, 6656
minwire 0: 30
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.117849, 
Output 1: 0.0513069, 0.0542125, 0.0948415, 0.799639, 
Output 2: 0.0728956, 0.278505, 0.614994, 0.0336054, 
Output 3: 0.975912, 0.0184609, 0.00157194, 0.00405472, 
Output 4: 0.199804, 0.731928, 0.0652639, 0.00300462, 
Output 5: 0.69478, 0.248186, 0.0450276, 0.0120068, 
Output 6: 0.017743, 0.0593738, 0.0784368, 0.844446, 

Begin processing the 9th record. run: 4969 subRun: 0 event: 16059 at 04-Oct-2025 16:36:14 UTC
Boundary wire vector sizes: 6719, 6593, 6583
minwire 0: 28
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.111639, 
Output 1: 0.0964028, 0.0665297, 0.149571, 0.687496, 
Output 2: 0.0685836, 0.146083, 0.755938, 0.0293956, 
Output 3: 0.97835, 0.011474, 0.00174133, 0.00843458, 
Output 4: 0.586635, 0.381156, 0.0279224, 0.00428635, 
Output 5: 0.482618, 0.375665, 0.103947, 0.0377692, 
Output 6: 0.0267292, 0.0669875, 0.0711845, 0.835099, 

Begin processing the 10th record. run: 4969 subRun: 0 event: 16060 at 04-Oct-2025 16:59:46 UTC
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, m_inputVertexListName, pVertexList) throw STATUS_CODE_NOT_INITIALIZED
    in function: GetHitRegion
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6995, 6941, 6777
minwire 0: 19
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.114461, 
Output 1: 0.0379277, 0.0385386, 0.113524, 0.810009, 
Output 2: 0.0638336, 0.147282, 0.758298, 0.0305858, 
Output 3: 0.975245, 0.0112067, 0.00207059, 0.0114776, 
Output 4: 0.326048, 0.624677, 0.0451901, 0.00408447, 
Output 5: 0.391069, 0.427566, 0.14017, 0.0411954, 
Output 6: 0.0213179, 0.0470484, 0.0585282, 0.873105, 

04-Oct-2025 17:24:47 UTC  Closed output file "mydetsimreco1reco2_20251004T130907Z.root"
04-Oct-2025 17:24:47 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/98/5f/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T173738Z_gen_001606_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1396.97       1520.26       1788.41       1475.76       123.55         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00166893     0.0106594     0.0443944     0.0059128     0.012207        10     
reco:pandora:StandardPandora                            1395.44       1518.32       1783.83       1473.92       122.893        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00265709     0.0224008     0.133005     0.00770972     0.0377404       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00116662    0.00556419     0.0200798    0.00428943    0.00493185       10     
reco:pandoracalo:Calorimetry                          0.000279278   0.00571484     0.0314382    0.000340567    0.0101181       10     
reco:pandorapid:Chi2ParticleID                        6.5164e-05    0.00029903    0.00213684    0.000103556   0.000612811      10     
reco:cvnmap:CVNMapper                                  0.0331205     0.044184      0.0732248     0.0392045     0.0114899       10     
reco:cvneva:CVNEvaluator                                1.00617       1.37748       3.74817       1.05092      0.811589        10     
reco:energyrecnumu:EnergyReco                         0.00496649     0.0790954     0.138646      0.0923118     0.0424314       10     
reco:energyrecnue:EnergyReco                          0.00101372     0.0027322    0.00900958    0.00126269    0.00287149       10     
reco:energyrecnc:EnergyReco                           0.00752562     0.0318737     0.115388     0.00812063     0.0381735       10     
[art]:TriggerResults:TriggerResultInserter            1.2368e-05    2.47243e-05   0.000108385   1.5432e-05    2.79283e-05      10     
end_path:out1:RootOutput                               3.418e-06    9.1081e-06    3.3972e-05    6.8805e-06    8.43489e-06      10     
end_path:out1:RootOutput(write)                        0.330087      0.352463      0.429816      0.344523      0.0263959       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12124.8 MB
  Peak resident set size usage (VmHWM): 3066.88 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-11-04 07:41:27 UTC       justIN version: 01.05.01