justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 224146.1@dunegpschedd02.fnal.gov

Jobsub ID224146.1@dunegpschedd02.fnal.gov
Workflow ID8430
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-04 04:33:59
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-10-04 06:28:49
From worker nodeHostnamedunegli-46731-0-cmswn4001.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-04 04:35:46
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T100959Z_gen_001180_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time1h (6188s)
CPU time1h (6115s = 98%)
Max RSS bytes3024334848 (2884 MiB)
Outputting started2025-10-04 06:18:54
Output files
Finished2025-10-04 06:28:49
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/33/66/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T100959Z_gen_001180_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
%MSG
Boundary wire vector sizes: 6698, 6636, 6683
minwire 0: 34
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2073, 2572
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0695091, 
Output 1: 0.0398465, 0.0445299, 0.0706757, 0.844948, 
Output 2: 0.0495444, 0.183161, 0.739241, 0.0280526, 
Output 3: 0.964853, 0.019976, 0.00354756, 0.011624, 
Output 4: 0.682721, 0.278456, 0.0332225, 0.00560084, 
Output 5: 0.570287, 0.299733, 0.0995821, 0.0303973, 
Output 6: 0.0153755, 0.045701, 0.0591406, 0.879783, 

Begin processing the 9th record. run: 5059 subRun: 0 event: 11799 at 04-Oct-2025 05:58:37 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6666, 6604, 6559
minwire 0: 55
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 265, 764
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.106561, 
Output 1: 0.061081, 0.171052, 0.166427, 0.60144, 
Output 2: 0.0356486, 0.23478, 0.702647, 0.0269246, 
Output 3: 0.944389, 0.0334717, 0.0074121, 0.0147272, 
Output 4: 0.471984, 0.452902, 0.0682408, 0.00687309, 
Output 5: 0.695066, 0.227425, 0.0609088, 0.0166, 
Output 6: 0.0123849, 0.0335579, 0.0514475, 0.90261, 

Begin processing the 10th record. run: 5059 subRun: 0 event: 11800 at 04-Oct-2025 06:08:42 UTC
Boundary wire vector sizes: 6914, 6965, 6810
minwire 0: 28
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.15076, 
Output 1: 0.100198, 0.107911, 0.178764, 0.613127, 
Output 2: 0.0618666, 0.19692, 0.712724, 0.0284899, 
Output 3: 0.962407, 0.0275341, 0.00357931, 0.0064799, 
Output 4: 0.385074, 0.547506, 0.0611682, 0.00625169, 
Output 5: 0.281326, 0.520837, 0.164163, 0.0336736, 
Output 6: 0.0717425, 0.143524, 0.121465, 0.663268, 

04-Oct-2025 06:18:53 UTC  Closed output file "mydetsimreco1reco2_20251004T043549Z.root"
04-Oct-2025 06:18:53 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/33/66/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T100959Z_gen_001180_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              597.37        610.862       633.695       608.113       10.6772        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00112472    0.00624181     0.0192136    0.00365414    0.00588011       10     
reco:pandora:StandardPandora                            596.375       609.756       632.696       607.088       10.5461        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000435325    0.0135716     0.0496491     0.014422      0.0136143       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000668677    0.0041564    0.00928977    0.00464817    0.00237744       10     
reco:pandoracalo:Calorimetry                          0.00018956    0.000502624    0.0022001    0.000351867   0.000569072      10     
reco:pandorapid:Chi2ParticleID                        4.4995e-05    0.000163357   0.000958427   7.82635e-05   0.000265379      10     
reco:cvnmap:CVNMapper                                  0.0147039     0.0185383     0.0291524     0.0171672    0.00427374       10     
reco:cvneva:CVNEvaluator                               0.665793      0.776142       1.72101      0.671452      0.314974        10     
reco:energyrecnumu:EnergyReco                         0.00272186     0.0377289     0.0925035     0.0413411     0.0318921       10     
reco:energyrecnue:EnergyReco                          0.000751274   0.00122584     0.0045271    0.000879531    0.0011047       10     
reco:energyrecnc:EnergyReco                           0.00445056     0.0504444     0.165512      0.0282784     0.0529867       10     
[art]:TriggerResults:TriggerResultInserter             8.737e-06    2.07672e-05   9.6763e-05    1.29595e-05   2.53984e-05      10     
end_path:out1:RootOutput                               5.25e-06     8.6654e-06    3.3764e-05     5.991e-06    8.37593e-06      10     
end_path:out1:RootOutput(write)                        0.194292       0.19683      0.208016      0.195123     0.00399059       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21790.9 MB
  Peak resident set size usage (VmHWM): 3024.33 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-11-03 18:40:30 UTC       justIN version: 01.05.01