justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 229900.1@dunegpschedd01.fnal.gov

Jobsub ID229900.1@dunegpschedd01.fnal.gov
Workflow ID8431
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-04 12:56:24
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-04 18:50:23
From worker nodeHostnamedunegli-7273006-0-fnpc17106.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-04 12:57:39
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250125T230928Z_gen_002867_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (21142s)
CPU time5h (20967s = 99%)
Max RSS bytes3077304320 (2934 MiB)
Outputting started2025-10-04 18:50:01
Output files
Finished2025-10-04 18:50:23
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/fe/84/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250125T230928Z_gen_002867_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6738, 6696, 6677
minwire 0: 45
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1800, 2299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.11367, 
Output 1: 0.0470585, 0.0398114, 0.114103, 0.799027, 
Output 2: 0.106772, 0.282054, 0.562686, 0.0484884, 
Output 3: 0.962379, 0.0243447, 0.00416524, 0.00911063, 
Output 4: 0.27452, 0.646587, 0.0735993, 0.00529442, 
Output 5: 0.751896, 0.221701, 0.0218451, 0.00455795, 
Output 6: 0.0166571, 0.0515598, 0.066964, 0.864819, 

Begin processing the 9th record. run: 4901 subRun: 0 event: 28669 at 04-Oct-2025 17:41:50 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6783, 6755, 6722
minwire 0: 32
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1272, 1771
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0849162, 
Output 1: 0.0582205, 0.0555202, 0.069043, 0.817216, 
Output 2: 0.0970979, 0.309275, 0.565621, 0.0280066, 
Output 3: 0.973689, 0.0200201, 0.00152886, 0.0047621, 
Output 4: 0.339994, 0.600345, 0.0556867, 0.00397394, 
Output 5: 0.673704, 0.298102, 0.0238914, 0.00430184, 
Output 6: 0.0254051, 0.0778125, 0.0894553, 0.807327, 

Begin processing the 10th record. run: 4901 subRun: 0 event: 28670 at 04-Oct-2025 18:17:48 UTC
Boundary wire vector sizes: 6498, 6445, 6583
minwire 0: 10
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0863967, 
Output 1: 0.0493432, 0.0636726, 0.0836803, 0.803304, 
Output 2: 0.0454012, 0.193639, 0.732453, 0.028507, 
Output 3: 0.963733, 0.0263863, 0.00350435, 0.00637638, 
Output 4: 0.787004, 0.178888, 0.0283692, 0.00573978, 
Output 5: 0.454496, 0.334401, 0.165614, 0.045489, 
Output 6: 0.0230503, 0.0549801, 0.0661789, 0.855791, 

04-Oct-2025 18:50:00 UTC  Closed output file "mydetsimreco1reco2_20251004T125746Z.root"
04-Oct-2025 18:50:00 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/fe/84/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250125T230928Z_gen_002867_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1924.99       2097.52       2221.93       2145.02       101.212        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00463741    0.00855499     0.0216526    0.00695633    0.00461787       10     
reco:pandora:StandardPandora                            1923.06       2094.88       2220.21        2143         101.243        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000558537    0.0319667      0.1413       0.0224986     0.0392916       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000554353    0.0102149     0.0486506    0.00240725     0.0145735       10     
reco:pandoracalo:Calorimetry                          0.000308437   0.00496651     0.039325     0.000535855    0.0115695       10     
reco:pandorapid:Chi2ParticleID                        0.000102182   0.000375035   0.00248069    0.000133481   0.000702793      10     
reco:cvnmap:CVNMapper                                  0.0484525     0.0655039     0.102981      0.0619352     0.0148406       10     
reco:cvneva:CVNEvaluator                               0.964211       1.73096       6.16253       1.3719        1.49015        10     
reco:energyrecnumu:EnergyReco                          0.0439803     0.110103      0.139193      0.121284      0.0288347       10     
reco:energyrecnue:EnergyReco                          0.00157174    0.00465487     0.014685      0.0023897    0.00494087       10     
reco:energyrecnc:EnergyReco                           0.00951364     0.0250272     0.134785      0.014292      0.036672        10     
[art]:TriggerResults:TriggerResultInserter            2.0546e-05    3.66175e-05   0.000105868   2.94685e-05   2.40238e-05      10     
end_path:out1:RootOutput                               6.741e-06    1.05234e-05    3.307e-05     8.464e-06    7.5782e-06       10     
end_path:out1:RootOutput(write)                        0.399109      0.647852       1.13499      0.689875      0.210696        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14268.8 MB
  Peak resident set size usage (VmHWM): 3077.3 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-11-04 07:42:20 UTC       justIN version: 01.05.01