justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 224378.1@dunegpschedd02.fnal.gov

Jobsub ID224378.1@dunegpschedd02.fnal.gov
Workflow ID8431
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-04 13:26:26
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-04 19:26:53
From worker nodeHostnamedunegli-7273006-0-fnpc17106.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-04 13:27:35
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T000433Z_gen_001588_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (21542s)
CPU time5h (21350s = 99%)
Max RSS bytes3059994624 (2918 MiB)
Outputting started2025-10-04 19:26:38
Output files
Finished2025-10-04 19:26:53
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/0b/b7/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T000433Z_gen_001588_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Output 5: 0.723392, 0.245234, 0.0259832, 0.00539116, 
Output 6: 0.0305933, 0.0958126, 0.0975253, 0.776069, 

Begin processing the 8th record. run: 5028 subRun: 0 event: 15878 at 04-Oct-2025 17:38:57 UTC
Boundary wire vector sizes: 6557, 6507, 6509
minwire 0: 62
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1821, 2320
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.10681, 
Output 1: 0.0630494, 0.0544413, 0.0880831, 0.794426, 
Output 2: 0.0413213, 0.213349, 0.719667, 0.0256631, 
Output 3: 0.972303, 0.0153759, 0.00263141, 0.00968989, 
Output 4: 0.404858, 0.527676, 0.0618701, 0.00559561, 
Output 5: 0.481167, 0.361174, 0.125772, 0.0318873, 
Output 6: 0.0214869, 0.0573555, 0.0740103, 0.847147, 

Begin processing the 9th record. run: 5028 subRun: 0 event: 15879 at 04-Oct-2025 18:14:35 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, m_inputVertexListName, pVertexList) throw STATUS_CODE_NOT_INITIALIZED
    in function: GetHitRegion
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6451, 6515, 6585
minwire 0: 43
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.136929, 
Output 1: 0.0530916, 0.106271, 0.268399, 0.572238, 
Output 2: 0.104912, 0.172753, 0.675401, 0.0469334, 
Output 3: 0.964073, 0.0188785, 0.00382246, 0.0132259, 
Output 4: 0.329311, 0.635642, 0.0325104, 0.00253618, 
Output 5: 0.596496, 0.335198, 0.0520646, 0.0162408, 
Output 6: 0.0171926, 0.0413542, 0.0510582, 0.890395, 

Begin processing the 10th record. run: 5028 subRun: 0 event: 15880 at 04-Oct-2025 18:49:15 UTC
Boundary wire vector sizes: 6797, 6792, 6799
minwire 0: 9
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2066, 2565
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.070525, 
Output 1: 0.0363324, 0.102398, 0.118696, 0.742574, 
Output 2: 0.0512935, 0.240347, 0.668599, 0.0397599, 
Output 3: 0.916566, 0.0500579, 0.0130204, 0.0203554, 
Output 4: 0.708803, 0.25677, 0.0302696, 0.00415723, 
Output 5: 0.685084, 0.235801, 0.0626656, 0.0164498, 
Output 6: 0.00920842, 0.0301744, 0.0457552, 0.914862, 

04-Oct-2025 19:26:36 UTC  Closed output file "mydetsimreco1reco2_20251004T132754Z.root"
04-Oct-2025 19:26:36 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/0b/b7/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T000433Z_gen_001588_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1858.84       2133.98       2331.55       2145.2        117.77         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00187764     0.0100896     0.0276257    0.00800971    0.00800314       10     
reco:pandora:StandardPandora                            1857.27       2131.5        2325.2        2143.25       116.839        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000542591     0.01947      0.122611     0.00430557     0.0365795       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000426768   0.00669705     0.0299938    0.00523816    0.00819396       10     
reco:pandoracalo:Calorimetry                          0.000321384   0.00903886     0.0330875    0.00393406     0.0116295       10     
reco:pandorapid:Chi2ParticleID                        8.2659e-05    0.000433473   0.00315315    0.00012196    0.000907626      10     
reco:cvnmap:CVNMapper                                  0.0478792     0.0650142      0.10558      0.063054      0.0166747       10     
reco:cvneva:CVNEvaluator                               0.938869       1.61813       4.96983       1.41143       1.14356        10     
reco:energyrecnumu:EnergyReco                         0.00980799     0.0977906     0.166079      0.112413      0.0506239       10     
reco:energyrecnue:EnergyReco                          0.00181915    0.00750061     0.0146186    0.00671084    0.00532293       10     
reco:energyrecnc:EnergyReco                           0.00949323     0.0515605     0.166886      0.0140093     0.0621247       10     
[art]:TriggerResults:TriggerResultInserter            1.5883e-05    3.90104e-05   0.000132723   2.77735e-05   3.23343e-05      10     
end_path:out1:RootOutput                               8.644e-06    1.46961e-05   3.6716e-05    1.11385e-05   8.30465e-06      10     
end_path:out1:RootOutput(write)                        0.387905      0.594492      0.836566      0.664814       0.16252        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14281.1 MB
  Peak resident set size usage (VmHWM): 3059.99 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-11-04 12:21:39 UTC       justIN version: 01.05.01