Jobsub ID 247638.131@dunegpschedd02.fnal.gov

Jobsub ID	247638.131@dunegpschedd02.fnal.gov
Workflow ID	10294
Stage ID	1
User name	imawby@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	7200 (2 hours)
Submitted time	2025-11-17 13:32:01
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce01
Last heartbeat	2025-11-17 14:23:45
From worker node	Hostname	dune001-355871.0-lcg2695.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-11-17 13:33:39
Input files	fardet-hd:nu_dune10kt_1x2x6_1108_745_20230826T224718Z_gen_g4_detsim_hitreco__20240229T182510Z_reco2.root
Outputting started	2025-11-17 13:48:09
Output files
Finished	2025-11-17 15:17:36
List job events (HTCondor job logs unavailable)
Jobscript log (last 10,000 characters)

encegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/79690b7a63ad96c929040888fed05585986a3d93/localProducts_larsoft_v10_10_06_e26_prof

local product directory is /cvmfs/fifeuser4.opensciencegrid.org/sw/dune/79690b7a63ad96c929040888fed05585986a3d93/localProducts_larsoft_v10_10_06_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ad/a9/nu_dune10kt_1x2x6_1108_745_20230826T224718Z_gen_g4_detsim_hitreco__20240229T182510Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 96th record. run: 1108 subRun: 1 event: 74596 at 17-Nov-2025 13:48:03 UTC
Unable to find file  PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt in any path specified by environment variable FW_SEARCH_PATH, delimiter :
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 97th record. run: 1108 subRun: 1 event: 74597 at 17-Nov-2025 13:48:04 UTC
Unable to find file  PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt in any path specified by environment variable FW_SEARCH_PATH, delimiter :
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 98th record. run: 1108 subRun: 1 event: 74598 at 17-Nov-2025 13:48:04 UTC
Unable to find file  PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt in any path specified by environment variable FW_SEARCH_PATH, delimiter :
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 99th record. run: 1108 subRun: 1 event: 74599 at 17-Nov-2025 13:48:05 UTC
Unable to find file  PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt in any path specified by environment variable FW_SEARCH_PATH, delimiter :
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 100th record. run: 1108 subRun: 1 event: 74600 at 17-Nov-2025 13:48:06 UTC
Unable to find file  PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt in any path specified by environment variable FW_SEARCH_PATH, delimiter :
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
17-Nov-2025 13:48:07 UTC  Closed output file "nu_dune10kt_1x2x6_1108_745_20230826T224718Z_gen_g4_detsim_hitreco__20240229T182510Z_reco2_reco2.root"
17-Nov-2025 13:48:07 UTC  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ad/a9/nu_dune10kt_1x2x6_1108_745_20230826T224718Z_gen_g4_detsim_hitreco__20240229T182510Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.245862      0.309881      0.565806      0.298725      0.0530888       100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0420424     0.0691657     0.0987418     0.0680367    0.00990795       100    
reco:pandora:StandardPandora                   0.183068      0.235674      0.482691      0.222692      0.0497146       100    
[art]:TriggerResults:TriggerResultInserter    1.0696e-05    2.9529e-05    0.000331349   2.0581e-05    3.77185e-05      100    
end_path:out1:RootOutput                       2.614e-06    3.8014e-06    1.8117e-05     3.36e-06     1.61879e-06      100    
end_path:out1:RootOutput(write)               0.00112216    0.00481255     0.0188604    0.00411585    0.00316026       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2093.13 MB
  Peak resident set size usage (VmHWM): 997.044 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
mv: cannot stat 'MLPHierarchyValidationFile.root': No such file or directory
mv: cannot stat 'Validation.root': No such file or directory