justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 224379.1@dunegpschedd02.fnal.gov

Jobsub ID224379.1@dunegpschedd02.fnal.gov
Workflow ID8431
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-04 13:46:27
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-04 19:33:56
From worker nodeHostnamedunegli-7270795-0-fnpc17112.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-04 13:47:21
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250125T012340Z_gen_000824_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (20783s)
CPU time5h (20639s = 99%)
Max RSS bytes3045634048 (2904 MiB)
Outputting started2025-10-04 19:33:45
Output files
Finished2025-10-04 19:33:56
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/9b/62/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250125T012340Z_gen_000824_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6459, 6342, 6409
minwire 0: 40
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 845, 1344
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.105345, 
Output 1: 0.145353, 0.0691172, 0.207256, 0.578274, 
Output 2: 0.0583383, 0.118734, 0.803176, 0.0197513, 
Output 3: 0.956264, 0.0368366, 0.0030056, 0.00389375, 
Output 4: 0.533796, 0.41988, 0.0401474, 0.0061766, 
Output 5: 0.303883, 0.472057, 0.174549, 0.0495105, 
Output 6: 0.102296, 0.184336, 0.118299, 0.595068, 

Begin processing the 9th record. run: 4881 subRun: 0 event: 8239 at 04-Oct-2025 18:27:12 UTC
Boundary wire vector sizes: 6714, 6662, 6736
minwire 0: 42
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1032, 1531
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.105384, 
Output 1: 0.0311395, 0.0453785, 0.100108, 0.823374, 
Output 2: 0.0843469, 0.321559, 0.550704, 0.0433897, 
Output 3: 0.956503, 0.0329733, 0.00400724, 0.00651649, 
Output 4: 0.18101, 0.725747, 0.0891125, 0.00413033, 
Output 5: 0.843764, 0.137161, 0.0154462, 0.00362867, 
Output 6: 0.0088732, 0.0335182, 0.0543892, 0.903219, 

Begin processing the 10th record. run: 4881 subRun: 0 event: 8240 at 04-Oct-2025 19:00:55 UTC
Boundary wire vector sizes: 6524, 6436, 6552
minwire 0: 33
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2333, 2832
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.147499, 
Output 1: 0.0695596, 0.05992, 0.0736636, 0.796857, 
Output 2: 0.103055, 0.288564, 0.582075, 0.0263053, 
Output 3: 0.986252, 0.00986665, 0.000601158, 0.00328004, 
Output 4: 0.246531, 0.681403, 0.0671471, 0.00491911, 
Output 5: 0.480309, 0.460203, 0.0501956, 0.00929241, 
Output 6: 0.0286506, 0.0801406, 0.100012, 0.791197, 

04-Oct-2025 19:33:43 UTC  Closed output file "mydetsimreco1reco2_20251004T134727Z.root"
04-Oct-2025 19:33:43 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/9b/62/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250125T012340Z_gen_000824_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1806.22       2061.6        2207.43       2047.48       117.665        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00430213    0.00982971     0.0207901     0.0085437    0.00545549       10     
reco:pandora:StandardPandora                            1804.72       2059.32       2205.85       2045.47       117.052        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00196969     0.034283      0.125751      0.0368734     0.0344219       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00187768    0.00591234     0.0195777    0.00536951    0.00495754       10     
reco:pandoracalo:Calorimetry                          0.000352946   0.00206751     0.0113531    0.00059321    0.00323834       10     
reco:pandorapid:Chi2ParticleID                        0.000117635   0.000385439   0.00123554    0.000178182   0.000407563      10     
reco:cvnmap:CVNMapper                                  0.0505419     0.0672425     0.0994985     0.0663878     0.0122832       10     
reco:cvneva:CVNEvaluator                               0.870598       1.50321       5.22135       1.01528       1.25391        10     
reco:energyrecnumu:EnergyReco                          0.0018581     0.0759326     0.177079      0.0955723     0.0573781       10     
reco:energyrecnue:EnergyReco                           0.001295     0.00602103     0.0143773    0.00195587    0.00550778       10     
reco:energyrecnc:EnergyReco                           0.00913637     0.0503861     0.124719      0.0141889     0.0484042       10     
[art]:TriggerResults:TriggerResultInserter            1.5462e-05    3.20639e-05   0.000105969   2.57555e-05   2.49676e-05      10     
end_path:out1:RootOutput                               9.844e-06    1.57029e-05   5.4036e-05    1.1769e-05    1.28169e-05      10     
end_path:out1:RootOutput(write)                        0.360198      0.527127      0.673969      0.595728      0.120339        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14237.2 MB
  Peak resident set size usage (VmHWM): 3045.63 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-11-04 07:43:33 UTC       justIN version: 01.05.01