Jobsub ID 295884.3@dunegpschedd02.fnal.gov

Jobsub ID	295884.3@dunegpschedd02.fnal.gov
Workflow ID	12429
Stage ID	1
User name	lwhite86@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	3600 (1 hours)
Submitted time	2026-01-27 09:23:19
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2026-01-27 09:28:04
From worker node	Hostname	dunegli-8294174-0-fnpc9051.fnal.gov
	cpuinfo	Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-01-27 09:23:44
Input files	fardet-hd:nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2.root
Jobscript	Exit code	0
	Real time	4m (247s)
	CPU time	3m (224s = 90%)
	Max RSS bytes	7484108800 (7137 MiB)
Outputting started	2026-01-27 09:27:52
Output files
Finished	2026-01-27 09:28:04
Saved logs	justin-logs:295884.3-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

grid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
Algorithm type 'LArDLTrackCharacterisation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
    in function: InitializeAlgorithms
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
    in function: ReadSettings
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Pandora/Pandora.cc line#: 149
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 100th record. run: 1423 subRun: 1 event: 45100 at 27-Jan-2026 09:27:45 UTC
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
Algorithm type 'LArDLTrackCharacterisation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
    in function: InitializeAlgorithms
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
    in function: ReadSettings
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Pandora/Pandora.cc line#: 149
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
27-Jan-2026 09:27:47 UTC  Closed output file "nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2_reco2.root"
27-Jan-2026 09:27:47 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/e4/7a/nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      0.89102       1.42804       2.01149       1.48638      0.234738        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.001907     0.00847703     0.0351308    0.00598692    0.00663077       100    
reco:pandora2:StandardPandora                  0.886309       1.41689       2.0043        1.47776      0.232756        100    
[art]:TriggerResults:TriggerResultInserter    2.3638e-05    3.69225e-05   0.000143884   3.5351e-05    1.23951e-05      100    
end_path:out1:RootOutput                       3.149e-06    5.04096e-06   2.7868e-05    4.5515e-06    2.64713e-06      100    
end_path:out1:RootOutput(write)               0.000292652   0.00238063     0.0172293    0.00132313    0.00312163       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8486.57 MB
  Peak resident set size usage (VmHWM): 7484.11 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
mv: cannot stat 'trackCharacterisationTraining.root': No such file or directory
total 816
-rw-r--r-- 1 dunegli fnalgrid    212 Jan 27 09:23 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid      0 Jan 27 09:24 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid 445780 Jan 27 09:27 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid    185 Jan 27 09:27 justin-processed-pfns.txt
drwxr-xr-x 5 dunegli fnalgrid   4096 Jan 27 09:23 larpandoracontent
-rw-r--r-- 1 dunegli fnalgrid 363286 Jan 27 09:27 nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2_reco2.root
-rw-r--r-- 1 dunegli fnalgrid    519 Jan 27 09:27 reco2_hist.root