Jobsub ID 295886.6@dunegpschedd02.fnal.gov

Jobsub ID	295886.6@dunegpschedd02.fnal.gov
Workflow ID	12431
Stage ID	1
User name	lwhite86@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	3600 (1 hours)
Submitted time	2026-01-27 09:51:14
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2026-01-27 09:54:36
From worker node	Hostname	dunegli-8293387-0-fnpc23010.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-01-27 09:52:19
Input files	fardet-hd:nue_dune10kt_1x2x6_1422_957_20230827T104924Z_gen_g4_detsim_hitreco__20240221T021622Z_reco2.root
Jobscript	Exit code	0
	Real time	1m (112s)
	CPU time	1m (93s = 83%)
	Max RSS bytes	7486853120 (7140 MiB)
Outputting started	2026-01-27 09:54:12
Output files
Finished	2026-01-27 09:54:36
Saved logs	justin-logs:295886.6-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

grid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
Algorithm type 'LArDLTrackCharacterisation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
    in function: InitializeAlgorithms
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
    in function: ReadSettings
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Pandora/Pandora.cc line#: 149
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 100th record. run: 1422 subRun: 1 event: 95800 at 27-Jan-2026 09:54:09 UTC
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
Algorithm type 'LArDLTrackCharacterisation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
    in function: InitializeAlgorithms
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
    in function: ReadSettings
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Pandora/Pandora.cc line#: 149
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
27-Jan-2026 09:54:10 UTC  Closed output file "nue_dune10kt_1x2x6_1422_957_20230827T104924Z_gen_g4_detsim_hitreco__20240221T021622Z_reco2_reco2.root"
27-Jan-2026 09:54:10 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/39/62/nue_dune10kt_1x2x6_1422_957_20230827T104924Z_gen_g4_detsim_hitreco__20240221T021622Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.501002      0.547948      0.939763      0.536407      0.0507971       100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00102986     0.0023797    0.00544443    0.00217986    0.000895982      100    
reco:pandora2:StandardPandora                   0.49882       0.54479      0.937745       0.53374      0.0506352       100    
[art]:TriggerResults:TriggerResultInserter     9.949e-06    1.34701e-05   4.0608e-05    1.25085e-05   4.19462e-06      100    
end_path:out1:RootOutput                       2.354e-06    3.1997e-06    1.5119e-05     3.096e-06    1.2394e-06       100    
end_path:out1:RootOutput(write)               0.000189612   0.000629916   0.00362006    0.000482943   0.000495937      100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8484.14 MB
  Peak resident set size usage (VmHWM): 7485.03 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
mv: cannot stat 'trackCharacterisationTraining.root': No such file or directory
total 872
-rw-r--r-- 1 dunegli fnalgrid    212 Jan 27 09:52 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid      0 Jan 27 09:52 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid 445780 Jan 27 09:54 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid    185 Jan 27 09:54 justin-processed-pfns.txt
drwxr-xr-x 4 dunegli fnalgrid     60 Jan 27 09:52 larpandoracontent
-rw-r--r-- 1 dunegli fnalgrid 355019 Jan 27 09:54 nue_dune10kt_1x2x6_1422_957_20230827T104924Z_gen_g4_detsim_hitreco__20240221T021622Z_reco2_reco2.root
-rw-r--r-- 1 dunegli fnalgrid    519 Jan 27 09:54 reco2_hist.root