Jobsub ID 296473.5@dunegpschedd02.fnal.gov
| Jobsub ID | 296473.5@dunegpschedd02.fnal.gov |
| Workflow ID | 12511 |
| Stage ID | 1 |
| User name | imawby@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 1048576000 (1000 MiB) |
| Wall seconds limit | 7200 (2 hours) |
| Submitted time | 2026-01-29 10:23:50 |
| Site | ES_CIEMAT |
| Entry | DUNE_T2_ES_CIEMAT_condorce1 |
| Last heartbeat | 2026-01-29 10:39:40 |
| From worker node | Hostname | gaew0218.ciemat.es |
| cpuinfo | Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 1048576000 (1000 MiB) |
| Wall seconds limit | 232200 (64 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | finished |
| Started | 2026-01-29 10:25:43 |
| Input files | fardet-hd:nu_dune10kt_1x2x6_1109_646_20230827T003339Z_gen_g4_detsim_hitreco__20240227T211845Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 13m (819s) |
| CPU time | 3m (216s = 26%) |
| Max RSS bytes | 10354659328 (9874 MiB) |
| Outputting started | 2026-01-29 10:39:23 |
| Output files | |
| Finished | 2026-01-29 10:39:40 |
| Saved logs | justin-logs:296473.5-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
DE_FAILURE
in function: Run
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Api/PandoraContentApiImpl.cc line#: 263
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 99th record. run: 1109 subRun: 1 event: 64699 at 29-Jan-2026 11:39:13 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_U_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_V_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_W_v04_13_00.pt'
Algorithm type 'LArClusterValidation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
in function: InitializeAlgorithms
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
in function: ReadSettings
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Pandora/Pandora.cc line#: 154
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
in function: CreateWorkerInstance
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
in function: Run
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Api/PandoraContentApiImpl.cc line#: 263
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 100th record. run: 1109 subRun: 1 event: 64700 at 29-Jan-2026 11:39:16 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_U_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_V_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_W_v04_13_00.pt'
Algorithm type 'LArClusterValidation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
in function: InitializeAlgorithms
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
in function: ReadSettings
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Pandora/Pandora.cc line#: 154
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
in function: CreateWorkerInstance
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
in function: Run
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Api/PandoraContentApiImpl.cc line#: 263
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
29-Jan-2026 11:39:19 CET Closed output file "nu_dune10kt_1x2x6_1109_646_20230827T003339Z_gen_g4_detsim_hitreco__20240227T211845Z_reco2_reco2.root"
29-Jan-2026 11:39:19 CET Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/1d/ac/nu_dune10kt_1x2x6_1109_646_20230827T003339Z_gen_g4_detsim_hitreco__20240227T211845Z_reco2.root"
================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 1.44536 2.08163 7.93384 2.08891 0.665496 100
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0951893 0.195279 0.372444 0.191395 0.0401321 100
reco:pandora:StandardPandora 1.29824 1.87939 7.81136 1.89444 0.668337 100
[art]:TriggerResults:TriggerResultInserter 1.8673e-05 2.68893e-05 6.3964e-05 2.59235e-05 6.74942e-06 100
end_path:out1:RootOutput 3.087e-06 4.8395e-06 2.0543e-05 4.129e-06 2.50914e-06 100
end_path:out1:RootOutput(write) 0.00126778 0.0067019 0.0376619 0.00479999 0.00619935 100
================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 11370.4 MB
Peak resident set size usage (VmHWM): 10369.5 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
mv: cannot stat 'ClusterValidation_WithAlg.root': No such file or directory 