justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 224370.1@dunegpschedd02.fnal.gov

Jobsub ID224370.1@dunegpschedd02.fnal.gov
Workflow ID8431
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-04 13:08:26
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-04 19:09:10
From worker nodeHostnamedunegli-7273006-0-fnpc17106.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-04 13:09:29
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T233533Z_gen_004182_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (21564s)
CPU time5h (21386s = 99%)
Max RSS bytes3061747712 (2919 MiB)
Outputting started2025-10-04 19:08:53
Output files
Finished2025-10-04 19:09:10
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/bc/71/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T233533Z_gen_004182_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===

Begin processing the 8th record. run: 3799 subRun: 0 event: 41818 at 04-Oct-2025 17:18:36 UTC
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6514, 6483, 6532
minwire 0: 38
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 801, 1300
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0850341, 
Output 1: 0.0882223, 0.215933, 0.0832724, 0.612572, 
Output 2: 0.146828, 0.433268, 0.385785, 0.0341182, 
Output 3: 0.965083, 0.0308654, 0.00103121, 0.00302062, 
Output 4: 0.584703, 0.393591, 0.0203403, 0.0013659, 
Output 5: 0.868176, 0.125346, 0.00517551, 0.00130188, 
Output 6: 0.00821639, 0.0350534, 0.0584805, 0.89825, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 41819 at 04-Oct-2025 17:55:03 UTC
Boundary wire vector sizes: 6757, 6702, 6677
minwire 0: 36
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.119255, 
Output 1: 0.0812142, 0.0636698, 0.120767, 0.734349, 
Output 2: 0.0448867, 0.21697, 0.717315, 0.0208286, 
Output 3: 0.970201, 0.0171361, 0.00255096, 0.0101122, 
Output 4: 0.291113, 0.632729, 0.0706035, 0.00555466, 
Output 5: 0.515023, 0.393916, 0.0759684, 0.0150929, 
Output 6: 0.0144045, 0.039876, 0.0598255, 0.885894, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 41820 at 04-Oct-2025 18:32:09 UTC
Boundary wire vector sizes: 6966, 6873, 6727
minwire 0: 26
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1276, 1775
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0465318, 
Output 1: 0.0183738, 0.0602406, 0.0987495, 0.822636, 
Output 2: 0.0561406, 0.155644, 0.754682, 0.0335326, 
Output 3: 0.924844, 0.0425398, 0.0141258, 0.0184901, 
Output 4: 0.844046, 0.12561, 0.0227685, 0.00757539, 
Output 5: 0.678085, 0.251052, 0.0565154, 0.0143472, 
Output 6: 0.0162486, 0.0522492, 0.0551115, 0.876391, 

04-Oct-2025 19:08:51 UTC  Closed output file "mydetsimreco1reco2_20251004T130936Z.root"
04-Oct-2025 19:08:51 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/bc/71/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T233533Z_gen_004182_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1958.8        2138.22       2321.24       2140.61       99.8058        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00195721    0.00975838     0.0219879    0.00809928    0.00527638       10     
reco:pandora:StandardPandora                            1957.3        2135.64       2318.82       2138.51       100.132        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00184175     0.0274423     0.105867      0.0127627     0.0302778       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00190755    0.00740349     0.0182668    0.00611096    0.00529773       10     
reco:pandoracalo:Calorimetry                          0.000503364   0.00864222     0.0362682    0.000783228    0.0127969       10     
reco:pandorapid:Chi2ParticleID                        0.000104302   0.000369764   0.00197542    0.000192355   0.00054059       10     
reco:cvnmap:CVNMapper                                  0.0503744     0.0713006     0.131915      0.0655781     0.0220052       10     
reco:cvneva:CVNEvaluator                               0.914237       1.62395       4.42453       1.42117      0.957503        10     
reco:energyrecnumu:EnergyReco                         0.00685494     0.073721      0.165019      0.0778812     0.0571671       10     
reco:energyrecnue:EnergyReco                          0.00143952    0.00564789     0.0168129    0.00244957    0.00545998       10     
reco:energyrecnc:EnergyReco                           0.00977136     0.0722831     0.169008      0.0569903     0.0610618       10     
[art]:TriggerResults:TriggerResultInserter            1.7871e-05    4.11963e-05   0.000135234   2.8432e-05    3.26434e-05      10     
end_path:out1:RootOutput                               7.38e-06     1.43545e-05   4.0892e-05    1.0308e-05    9.45967e-06      10     
end_path:out1:RootOutput(write)                        0.396547      0.681074      0.923528      0.684143      0.152242        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14269.8 MB
  Peak resident set size usage (VmHWM): 3061.75 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-11-04 12:21:35 UTC       justIN version: 01.05.01