Jobsub ID 296473.7@dunegpschedd02.fnal.gov
| Jobsub ID | 296473.7@dunegpschedd02.fnal.gov |
| Workflow ID | 12511 |
| Stage ID | 1 |
| User name | imawby@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 1048576000 (1000 MiB) |
| Wall seconds limit | 7200 (2 hours) |
| Submitted time | 2026-01-29 10:23:50 |
| Site | ES_CIEMAT |
| Entry | DUNE_T2_ES_CIEMAT_condorce1 |
| Last heartbeat | 2026-01-29 10:36:54 |
| From worker node | Hostname | gaew0218.ciemat.es |
| cpuinfo | Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 1048576000 (1000 MiB) |
| Wall seconds limit | 232200 (64 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | finished |
| Started | 2026-01-29 10:25:11 |
| Input files | fardet-hd:nu_dune10kt_1x2x6_1408_689_20230826T094008Z_gen_g4_detsim_hitreco__20240229T181317Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 11m (682s) |
| CPU time | 3m (219s = 32%) |
| Max RSS bytes | 10425126912 (9942 MiB) |
| Outputting started | 2026-01-29 10:36:35 |
| Output files | |
| Finished | 2026-01-29 10:36:54 |
| Saved logs | justin-logs:296473.7-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
LURE
in function: Run
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Api/PandoraContentApiImpl.cc line#: 263
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 99th record. run: 1408 subRun: 1 event: 68999 at 29-Jan-2026 11:36:20 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_U_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_V_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_W_v04_13_00.pt'
Algorithm type 'LArClusterValidation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
in function: InitializeAlgorithms
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
in function: ReadSettings
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Pandora/Pandora.cc line#: 154
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
in function: CreateWorkerInstance
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
in function: Run
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Api/PandoraContentApiImpl.cc line#: 263
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
Begin processing the 100th record. run: 1408 subRun: 1 event: 69000 at 29-Jan-2026 11:36:23 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_U_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_V_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/07a864404679176cbcba8ffec28910705f208954/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_W_v04_13_00.pt'
Algorithm type 'LArClusterValidation' not registered with pandora algorithm manager.
CreateAlgorithm(pXmlElement, algorithmName) return STATUS_CODE_NOT_FOUND
in function: InitializeAlgorithms
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Managers/AlgorithmManager.cc line#: 83
m_pPandoraImpl->InitializeAlgorithms(&xmlHandle) throw STATUS_CODE_NOT_FOUND
in function: ReadSettings
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Pandora/Pandora.cc line#: 154
Failure in reading pandora settings, STATUS_CODE_NOT_FOUND
PandoraApi::ReadSettings(*pPandora, settingsFile) throw STATUS_CODE_FAILURE
in function: CreateWorkerInstance
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 1051
MasterAlgorithm: Exception during initialization of worker instances STATUS_CODE_FAILURE
this->InitializeWorkerInstances() return STATUS_CODE_FAILURE
in function: Run
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 158
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Api/PandoraContentApiImpl.cc line#: 263
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_FAILURE
29-Jan-2026 11:36:30 CET Closed output file "nu_dune10kt_1x2x6_1408_689_20230826T094008Z_gen_g4_detsim_hitreco__20240229T181317Z_reco2_reco2.root"
29-Jan-2026 11:36:30 CET Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/69/f9/nu_dune10kt_1x2x6_1408_689_20230826T094008Z_gen_g4_detsim_hitreco__20240229T181317Z_reco2.root"
================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 1.22693 1.85659 8.9576 1.77507 0.733645 100
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0308212 0.0639539 0.0711606 0.0641483 0.00419486 100
reco:pandora:StandardPandora 1.15853 1.78479 8.91468 1.70082 0.735591 100
[art]:TriggerResults:TriggerResultInserter 1.4239e-05 2.73879e-05 5.4857e-05 2.66635e-05 6.17477e-06 100
end_path:out1:RootOutput 2.689e-06 4.58288e-06 1.7231e-05 4.078e-06 2.01955e-06 100
end_path:out1:RootOutput(write) 0.00158439 0.00758709 0.0420781 0.00597919 0.00611611 100
================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 11439.1 MB
Peak resident set size usage (VmHWM): 10437 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
mv: cannot stat 'ClusterValidation_WithAlg.root': No such file or directory 