Jobsub ID 295886.7@dunegpschedd02.fnal.gov

Jobsub ID	295886.7@dunegpschedd02.fnal.gov
Workflow ID	12431
Stage ID	1
User name	lwhite86@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	3600 (1 hours)
Submitted time	2026-01-27 09:51:14
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2026-01-27 09:54:07
From worker node	Hostname	dunegli-8293849-0-fnpc22014.fnal.gov
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-01-27 09:51:56
Input files	fardet-hd:nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2.root
Jobscript	Exit code	0
	Real time	2m (122s)
	CPU time	1m (110s = 90%)
	Max RSS bytes	7483080704 (7136 MiB)
Outputting started	2026-01-27 09:53:58
Output files
Finished	2026-01-27 09:54:07
Saved logs	justin-logs:295886.7-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

grid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
Algorithm type 'LArDLTrackCharacterisation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
    in function: InitializeAlgorithms
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
    in function: ReadSettings
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Pandora/Pandora.cc line#: 149
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 100th record. run: 1423 subRun: 1 event: 45100 at 27-Jan-2026 09:53:53 UTC
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/09b6e2d917fc3a1c0ed6beac6e164db639cf582d/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
Algorithm type 'LArDLTrackCharacterisation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
    in function: InitializeAlgorithms
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
    in function: ReadSettings
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Pandora/Pandora.cc line#: 149
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_03-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
27-Jan-2026 09:53:55 UTC  Closed output file "nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2_reco2.root"
27-Jan-2026 09:53:55 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/e4/7a/nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      0.55743      0.670067       1.06708      0.661112      0.0845453       100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00141061    0.00461262     0.0103287    0.00408637    0.00228996       100    
reco:pandora2:StandardPandora                  0.548744      0.660436       1.06324      0.651866      0.0797375       100    
[art]:TriggerResults:TriggerResultInserter    1.6782e-05    2.77859e-05   5.7048e-05    2.6686e-05    6.05062e-06      100    
end_path:out1:RootOutput                       4.528e-06    6.94035e-06   2.2252e-05    6.6075e-06    2.18079e-06      100    
end_path:out1:RootOutput(write)               0.000364439   0.00470453     0.0381891    0.00282244    0.00603081       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8487.33 MB
  Peak resident set size usage (VmHWM): 7483.08 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
mv: cannot stat 'trackCharacterisationTraining.root': No such file or directory
total 876
-rw-r--r-- 1 dunegli fnalgrid    212 Jan 27 09:51 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid      0 Jan 27 09:52 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid 445780 Jan 27 09:53 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid    185 Jan 27 09:53 justin-processed-pfns.txt
drwxr-xr-x 4 dunegli fnalgrid     60 Jan 27 09:51 larpandoracontent
-rw-r--r-- 1 dunegli fnalgrid 359176 Jan 27 09:53 nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2_reco2.root
-rw-r--r-- 1 dunegli fnalgrid    519 Jan 27 09:53 reco2_hist.root