Jobsub ID 253289.178@dunegpschedd01.fnal.gov

Jobsub ID	253289.178@dunegpschedd01.fnal.gov
Workflow ID	10294
Stage ID	1
User name	imawby@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	7200 (2 hours)
Submitted time	2025-11-17 13:30:10
Site	ES_CIEMAT
Entry	DUNE_T2_ES_CIEMAT_condorce2
Last heartbeat	2025-11-17 14:22:25
From worker node	Hostname	gaew0194.ciemat.es
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	232200 (64 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-11-17 13:32:18
Input files	fardet-hd:nu_dune10kt_1x2x6_1102_303_20230826T064604Z_gen_g4_detsim_hitreco__20240229T165556Z_reco2.root
Outputting started	2025-11-17 13:49:32
Output files
Finished	2025-11-17 15:17:36
List job events (HTCondor job logs unavailable)
Jobscript log (last 10,000 characters)

encegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/79690b7a63ad96c929040888fed05585986a3d93/localProducts_larsoft_v10_10_06_e26_prof

local product directory is /cvmfs/fifeuser4.opensciencegrid.org/sw/dune/79690b7a63ad96c929040888fed05585986a3d93/localProducts_larsoft_v10_10_06_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/58/63/nu_dune10kt_1x2x6_1102_303_20230826T064604Z_gen_g4_detsim_hitreco__20240229T165556Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 96th record. run: 1102 subRun: 1 event: 30396 at 17-Nov-2025 14:49:23 CET
Unable to find file  PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt in any path specified by environment variable FW_SEARCH_PATH, delimiter :
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 97th record. run: 1102 subRun: 1 event: 30397 at 17-Nov-2025 14:49:26 CET
Unable to find file  PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt in any path specified by environment variable FW_SEARCH_PATH, delimiter :
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 98th record. run: 1102 subRun: 1 event: 30398 at 17-Nov-2025 14:49:27 CET
Unable to find file  PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt in any path specified by environment variable FW_SEARCH_PATH, delimiter :
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 99th record. run: 1102 subRun: 1 event: 30399 at 17-Nov-2025 14:49:28 CET
Unable to find file  PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt in any path specified by environment variable FW_SEARCH_PATH, delimiter :
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 100th record. run: 1102 subRun: 1 event: 30400 at 17-Nov-2025 14:49:29 CET
Unable to find file  PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt in any path specified by environment variable FW_SEARCH_PATH, delimiter :
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
    in function: CreateWorkerInstance
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_atmosHierarchy/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
17-Nov-2025 14:49:31 CET  Closed output file "nu_dune10kt_1x2x6_1102_303_20230826T064604Z_gen_g4_detsim_hitreco__20240229T165556Z_reco2_reco2.root"
17-Nov-2025 14:49:31 CET  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/58/63/nu_dune10kt_1x2x6_1102_303_20230826T064604Z_gen_g4_detsim_hitreco__20240229T165556Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.324058      0.437133      0.850289      0.410255      0.0904466       100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0484823     0.0892607     0.131587      0.0873972     0.0120199       100    
reco:pandora:StandardPandora                   0.246984      0.339829       0.72101      0.311912      0.0857607       100    
[art]:TriggerResults:TriggerResultInserter    1.6134e-05    2.77191e-05   0.000117858   2.4402e-05    1.36242e-05      100    
end_path:out1:RootOutput                       2.902e-06    4.21977e-06   1.9173e-05     4.001e-06    1.55786e-06      100    
end_path:out1:RootOutput(write)               0.00144703    0.00779131     0.0366326    0.00666762    0.00561927       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2080.86 MB
  Peak resident set size usage (VmHWM): 1024.94 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
mv: cannot stat 'MLPHierarchyValidationFile.root': No such file or directory
mv: cannot stat 'Validation.root': No such file or directory