Jobsub ID 296473.4@dunegpschedd02.fnal.gov
| Jobsub ID | 296473.4@dunegpschedd02.fnal.gov |
| Workflow ID | 12511 |
| Stage ID | 1 |
| User name | imawby@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 1048576000 (1000 MiB) |
| Wall seconds limit | 7200 (2 hours) |
| Submitted time | 2026-01-29 10:23:50 |
| Site | ES_CIEMAT |
| Entry | DUNE_T2_ES_CIEMAT_condorce2 |
| Last heartbeat | 2026-01-29 10:38:20 |
| From worker node | Hostname | gaew0218.ciemat.es |
| cpuinfo | Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 1048576000 (1000 MiB) |
| Wall seconds limit | 232200 (64 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | finished |
| Started | 2026-01-29 10:25:39 |
| Input files | fardet-hd:nu_dune10kt_1x2x6_1109_749_20230827T003442Z_gen_g4_detsim_hitreco__20240227T212542Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 12m (743s) |
| CPU time | 3m (221s = 29%) |
| Max RSS bytes | 10345656320 (9866 MiB) |
| Outputting started | 2026-01-29 10:38:03 |
| Output files | |
| Finished | 2026-01-29 10:38:20 |
| Saved logs | justin-logs:296473.4-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
DE_FAILURE
in function: Run
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Api/PandoraContentApiImpl.cc line#: 263
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 99th record. run: 1109 subRun: 1 event: 74999 at 29-Jan-2026 11:37:51 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_U_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_V_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_W_v04_13_00.pt'
Algorithm type 'LArClusterValidation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
in function: InitializeAlgorithms
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
in function: ReadSettings
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Pandora/Pandora.cc line#: 154
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
in function: CreateWorkerInstance
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
in function: Run
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Api/PandoraContentApiImpl.cc line#: 263
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 100th record. run: 1109 subRun: 1 event: 75000 at 29-Jan-2026 11:37:54 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_U_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_V_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_W_v04_13_00.pt'
Algorithm type 'LArClusterValidation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
in function: InitializeAlgorithms
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
in function: ReadSettings
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Pandora/Pandora.cc line#: 154
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
in function: CreateWorkerInstance
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
in function: Run
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Api/PandoraContentApiImpl.cc line#: 263
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
29-Jan-2026 11:37:58 CET Closed output file "nu_dune10kt_1x2x6_1109_749_20230827T003442Z_gen_g4_detsim_hitreco__20240227T212542Z_reco2_reco2.root"
29-Jan-2026 11:37:58 CET Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/70/18/nu_dune10kt_1x2x6_1109_749_20230827T003442Z_gen_g4_detsim_hitreco__20240227T212542Z_reco2.root"
================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 1.59652 2.22757 8.27478 2.18551 0.638084 100
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0662807 0.186414 0.651747 0.172328 0.0651689 100
reco:pandora:StandardPandora 1.42589 2.03402 8.15351 1.99813 0.642962 100
[art]:TriggerResults:TriggerResultInserter 1.5956e-05 2.94481e-05 5.0925e-05 2.91335e-05 5.97026e-06 100
end_path:out1:RootOutput 2.986e-06 5.69861e-06 2.0753e-05 4.743e-06 2.65659e-06 100
end_path:out1:RootOutput(write) 0.0015871 0.00684144 0.0337347 0.00579933 0.0050108 100
================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 11364.9 MB
Peak resident set size usage (VmHWM): 10362.6 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
mv: cannot stat 'ClusterValidation_WithAlg.root': No such file or directory 