Jobsub ID 295886.0@dunegpschedd02.fnal.gov

Jobsub ID	295886.0@dunegpschedd02.fnal.gov
Workflow ID	12431
Stage ID	1
User name	lwhite86@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	3600 (1 hours)
Submitted time	2026-01-27 09:51:14
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2026-01-27 09:54:00
From worker node	Hostname	dunegli-8293172-0-fnpc23035.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-01-27 09:52:06
Input files	fardet-hd:nue_dune10kt_1x2x6_1120_954_20230828T044443Z_gen_g4_detsim_hitreco__20240221T021502Z_reco2.root
Jobscript	Exit code	0
	Real time	1m (104s)
	CPU time	1m (87s = 83%)
	Max RSS bytes	7482216448 (7135 MiB)
Outputting started	2026-01-27 09:53:51
Output files
Finished	2026-01-27 09:54:00
Saved logs	justin-logs:295886.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

grid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
Algorithm type 'LArDLTrackCharacterisation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
    in function: InitializeAlgorithms
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
    in function: ReadSettings
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Pandora/Pandora.cc line#: 149
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 100th record. run: 1120 subRun: 1 event: 95500 at 27-Jan-2026 09:53:48 UTC
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
Algorithm type 'LArDLTrackCharacterisation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
    in function: InitializeAlgorithms
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
    in function: ReadSettings
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Pandora/Pandora.cc line#: 149
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
27-Jan-2026 09:53:49 UTC  Closed output file "nue_dune10kt_1x2x6_1120_954_20230828T044443Z_gen_g4_detsim_hitreco__20240221T021502Z_reco2_reco2.root"
27-Jan-2026 09:53:49 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/9c/d9/nue_dune10kt_1x2x6_1120_954_20230828T044443Z_gen_g4_detsim_hitreco__20240221T021502Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.477469       0.54086      0.926883      0.524321      0.0582567       100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00120033     0.0127859     0.0739936     0.0120258     0.0116563       100    
reco:pandora2:StandardPandora                  0.466815      0.527156      0.905045      0.510187      0.0574446       100    
[art]:TriggerResults:TriggerResultInserter    1.1061e-05    1.42481e-05   4.7972e-05    1.3371e-05    3.93405e-06      100    
end_path:out1:RootOutput                       3.507e-06    4.85448e-06   2.2964e-05     4.483e-06     2.598e-06       100    
end_path:out1:RootOutput(write)               0.000210804   0.000690115   0.00330926    0.000507156   0.000579424      100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8478.83 MB
  Peak resident set size usage (VmHWM): 7479.84 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
mv: cannot stat 'trackCharacterisationTraining.root': No such file or directory
total 868
-rw-r--r-- 1 dunegli fnalgrid    212 Jan 27 09:52 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid      0 Jan 27 09:52 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid 445780 Jan 27 09:53 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid    185 Jan 27 09:53 justin-processed-pfns.txt
drwxr-xr-x 4 dunegli fnalgrid     60 Jan 27 09:52 larpandoracontent
-rw-r--r-- 1 dunegli fnalgrid 350903 Jan 27 09:53 nue_dune10kt_1x2x6_1120_954_20230828T044443Z_gen_g4_detsim_hitreco__20240221T021502Z_reco2_reco2.root
-rw-r--r-- 1 dunegli fnalgrid    519 Jan 27 09:53 reco2_hist.root