justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 295901.0@dunegpschedd02.fnal.gov

Jobsub ID295901.0@dunegpschedd02.fnal.gov
Workflow ID12434
Stage ID1
User namelwhite86@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2026-01-27 11:19:19
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2026-01-27 11:24:53
From worker nodeHostnamedunegli-8294173-0-fnpc9055.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-27 11:21:03
Input filesfardet-hd:nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2.root
JobscriptExit code0
Real time3m (205s)
CPU time3m (181s = 88%)
Max RSS bytes7484608512 (7137 MiB)
Outputting started2026-01-27 11:24:29
Output files
Finished2026-01-27 11:24:53
Saved logsjustin-logs:295901.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

grid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
Algorithm type 'LArDLTrackCharacterisation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
    in function: InitializeAlgorithms
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
    in function: ReadSettings
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Pandora/Pandora.cc line#: 149
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 100th record. run: 1423 subRun: 1 event: 45100 at 27-Jan-2026 11:24:24 UTC
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
Algorithm type 'LArDLTrackCharacterisation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
    in function: InitializeAlgorithms
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
    in function: ReadSettings
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Pandora/Pandora.cc line#: 149
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
27-Jan-2026 11:24:26 UTC  Closed output file "nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2_reco2.root"
27-Jan-2026 11:24:26 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/e4/7a/nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.905593       1.05747       1.6462       0.983578      0.182238        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00133883    0.00366236    0.00620596     0.0037539    0.00124693       100    
reco:pandora2:StandardPandora                  0.903061       1.05213       1.63959      0.980685      0.181376        100    
[art]:TriggerResults:TriggerResultInserter    1.7246e-05    2.42951e-05   5.7129e-05    2.30115e-05   4.70174e-06      100    
end_path:out1:RootOutput                       2.513e-06    3.28027e-06   1.5473e-05    3.0645e-06    1.31872e-06      100    
end_path:out1:RootOutput(write)               0.000273876   0.00147799     0.0159374    0.000876101   0.00200817       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8487.35 MB
  Peak resident set size usage (VmHWM): 7484.61 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
mv: cannot stat 'trackCharacterisationTraining.root': No such file or directory
total 876
-rw-r--r-- 1 dunegli fnalgrid    212 Jan 27 11:21 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid      0 Jan 27 11:21 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid 445780 Jan 27 11:24 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid    185 Jan 27 11:24 justin-processed-pfns.txt
drwxr-xr-x 4 dunegli fnalgrid     60 Jan 27 11:21 larpandoracontent
-rw-r--r-- 1 dunegli fnalgrid 359176 Jan 27 11:24 nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2_reco2.root
-rw-r--r-- 1 dunegli fnalgrid    519 Jan 27 11:24 reco2_hist.root
justIN time: 2026-02-04 19:27:59 UTC       justIN version: 01.06.00