justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 229914.0@dunegpschedd01.fnal.gov

Jobsub ID229914.0@dunegpschedd01.fnal.gov
Workflow ID8430
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-04 15:14:32
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-04 21:10:06
From worker nodeHostnamedunegli-7273006-0-fnpc17106.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-04 15:15:42
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T084542Z_gen_003611_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (21249s)
CPU time5h (21070s = 99%)
Max RSS bytes3067322368 (2925 MiB)
Outputting started2025-10-04 21:09:52
Output files
Finished2025-10-04 21:10:06
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/46/77/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T084542Z_gen_003611_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6618, 6623, 6582
minwire 0: 63
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 59, 558
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0668028, 
Output 1: 0.151077, 0.0261891, 0.0948144, 0.727919, 
Output 2: 0.143948, 0.15383, 0.687332, 0.0148905, 
Output 3: 0.971126, 0.023605, 0.00127149, 0.00399773, 
Output 4: 0.613427, 0.32759, 0.04473, 0.0142523, 
Output 5: 0.545207, 0.435153, 0.0169664, 0.00267377, 
Output 6: 0.06124, 0.135224, 0.102521, 0.701015, 

Begin processing the 9th record. run: 4933 subRun: 0 event: 36109 at 04-Oct-2025 19:57:19 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6573, 6515, 6609
minwire 0: 28
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.201629, 
Output 1: 0.108701, 0.144861, 0.16148, 0.584958, 
Output 2: 0.0296354, 0.161612, 0.774134, 0.0346186, 
Output 3: 0.981551, 0.00696707, 0.00210315, 0.00937899, 
Output 4: 0.604494, 0.34875, 0.0413345, 0.00542131, 
Output 5: 0.169798, 0.291563, 0.385365, 0.153275, 
Output 6: 0.065514, 0.105358, 0.0995492, 0.729579, 

Begin processing the 10th record. run: 4933 subRun: 0 event: 36110 at 04-Oct-2025 20:33:34 UTC
Boundary wire vector sizes: 6609, 6658, 6554
minwire 0: 4
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 883, 1382
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.159308, 
Output 1: 0.0455324, 0.0762662, 0.150954, 0.727247, 
Output 2: 0.0825546, 0.332312, 0.538107, 0.0470266, 
Output 3: 0.949268, 0.0364688, 0.00660126, 0.00766182, 
Output 4: 0.317927, 0.576091, 0.0984026, 0.00757877, 
Output 5: 0.623015, 0.330898, 0.0404688, 0.00561735, 
Output 6: 0.0131531, 0.0458663, 0.0682008, 0.87278, 

04-Oct-2025 21:09:50 UTC  Closed output file "mydetsimreco1reco2_20251004T151559Z.root"
04-Oct-2025 21:09:50 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/46/77/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T084542Z_gen_003611_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1778.67       2104.54       2218.3         2169         127.08         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0017912    0.00905432     0.021792     0.00719221    0.00547896       10     
reco:pandora:StandardPandora                            1776.22       2101.69       2215.97       2166.58       126.966        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000534202    0.0306772     0.152664     0.00953866     0.0436031       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000573153   0.00701696     0.0298014    0.00637485    0.00797933       10     
reco:pandoracalo:Calorimetry                          0.000281728    0.0083258     0.0368877    0.000995315    0.0137697       10     
reco:pandorapid:Chi2ParticleID                        0.000110768   0.000411999   0.00254395    0.00015657    0.000713309      10     
reco:cvnmap:CVNMapper                                  0.0496182     0.0693582     0.104751      0.0664183     0.0169967       10     
reco:cvneva:CVNEvaluator                                1.00412       1.85173       5.98038       1.4304        1.38424        10     
reco:energyrecnumu:EnergyReco                          0.020839      0.0725768     0.159188      0.0474873     0.0537691       10     
reco:energyrecnue:EnergyReco                          0.00142998    0.00574622     0.0148661     0.002263     0.00551174       10     
reco:energyrecnc:EnergyReco                            0.0135029     0.0913126      0.22485      0.100823      0.0711035       10     
[art]:TriggerResults:TriggerResultInserter            1.9984e-05    3.56421e-05   9.9388e-05    2.8302e-05    2.21038e-05      10     
end_path:out1:RootOutput                              1.0479e-05    1.57557e-05    3.401e-05    1.21195e-05   6.99268e-06      10     
end_path:out1:RootOutput(write)                        0.410155      0.701632      0.857779       0.69025      0.123738        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14260.5 MB
  Peak resident set size usage (VmHWM): 3067.32 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-11-04 15:56:49 UTC       justIN version: 01.05.01