justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 229862.1@dunegpschedd01.fnal.gov

Jobsub ID229862.1@dunegpschedd01.fnal.gov
Workflow ID8431
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-04 11:06:19
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-04 17:02:14
From worker nodeHostnamedunegli-7273006-0-fnpc17106.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-04 11:07:11
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T103542Z_gen_003899_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (21285s)
CPU time5h (21078s = 99%)
Max RSS bytes3051708416 (2910 MiB)
Outputting started2025-10-04 17:01:57
Output files
Finished2025-10-04 17:02:14
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/f4/7e/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T103542Z_gen_003899_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===

Begin processing the 8th record. run: 4980 subRun: 0 event: 38988 at 04-Oct-2025 15:08:32 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6552, 6555, 6542
minwire 0: 69
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 576, 1075
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.131989, 
Output 1: 0.0804097, 0.0848272, 0.171854, 0.66291, 
Output 2: 0.0596549, 0.263327, 0.649281, 0.0277373, 
Output 3: 0.93736, 0.0504819, 0.00653085, 0.00562749, 
Output 4: 0.343383, 0.544846, 0.102123, 0.00964792, 
Output 5: 0.516353, 0.411062, 0.0637241, 0.0088616, 
Output 6: 0.0299406, 0.0906012, 0.0989956, 0.780463, 

Begin processing the 9th record. run: 4980 subRun: 0 event: 38989 at 04-Oct-2025 15:46:48 UTC
Boundary wire vector sizes: 6716, 6709, 6581
minwire 0: 46
minwire 1: 1
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0647114, 
Output 1: 0.111167, 0.198747, 0.240582, 0.449504, 
Output 2: 0.0531962, 0.144787, 0.774641, 0.0273762, 
Output 3: 0.851532, 0.120372, 0.020018, 0.00807783, 
Output 4: 0.803135, 0.180446, 0.0137421, 0.00267701, 
Output 5: 0.224212, 0.525673, 0.205389, 0.0447266, 
Output 6: 0.115806, 0.371107, 0.157721, 0.355366, 

Begin processing the 10th record. run: 4980 subRun: 0 event: 38990 at 04-Oct-2025 16:23:49 UTC
Boundary wire vector sizes: 7011, 6926, 6833
minwire 0: 17
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.129457, 
Output 1: 0.0532227, 0.1262, 0.1359, 0.684677, 
Output 2: 0.0449667, 0.165064, 0.773347, 0.016622, 
Output 3: 0.980791, 0.0132247, 0.00139008, 0.00459429, 
Output 4: 0.34753, 0.57366, 0.0709189, 0.00789151, 
Output 5: 0.326287, 0.512486, 0.133005, 0.0282221, 
Output 6: 0.0350711, 0.0993878, 0.101849, 0.763692, 

04-Oct-2025 17:01:55 UTC  Closed output file "mydetsimreco1reco2_20251004T110719Z.root"
04-Oct-2025 17:01:55 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/f4/7e/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T103542Z_gen_003899_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1689.62       2108.27       2287.58       2209.71       217.154        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00120498     0.0120263     0.0333755    0.00892561     0.0100892       10     
reco:pandora:StandardPandora                            1687.29       2105.5        2285.07       2207.44       217.751        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000570218    0.0338834     0.145669      0.0168809     0.0414891       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000555776    0.012435      0.0744501     0.0058628     0.0213937       10     
reco:pandoracalo:Calorimetry                          0.000334066   0.00232939     0.0118383    0.000602675   0.00338936       10     
reco:pandorapid:Chi2ParticleID                        8.6269e-05    0.000407474   0.00160757    0.000180599   0.000509594      10     
reco:cvnmap:CVNMapper                                  0.0518697     0.0720268     0.0856143     0.0724943    0.00904966       10     
reco:cvneva:CVNEvaluator                               0.909576       1.78766       5.15634       1.45574       1.13652        10     
reco:energyrecnumu:EnergyReco                         0.00295021     0.0885814     0.161984       0.12428      0.0647551       10     
reco:energyrecnue:EnergyReco                           0.0014438    0.00591374     0.0148642     0.0025225     0.0054274       10     
reco:energyrecnc:EnergyReco                            0.0133334      0.0788       0.325549      0.0153356     0.096328        10     
[art]:TriggerResults:TriggerResultInserter            2.1321e-05    3.97536e-05   0.000135635   3.1219e-05    3.24484e-05      10     
end_path:out1:RootOutput                               4.268e-06    1.38022e-05   4.4648e-05    1.00295e-05   1.09127e-05      10     
end_path:out1:RootOutput(write)                        0.456605      0.671926      0.822576      0.688979      0.0864172       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14238.6 MB
  Peak resident set size usage (VmHWM): 3051.71 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-11-04 12:21:34 UTC       justIN version: 01.05.01