Jobsub ID 258940.26@dunegpschedd01.fnal.gov
| Jobsub ID | 258940.26@dunegpschedd01.fnal.gov |
| Workflow ID | 10817 |
| Stage ID | 1 |
| User name | imawby@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 1048576000 (1000 MiB) |
| Wall seconds limit | 7200 (2 hours) |
| Submitted time | 2025-11-26 18:48:50 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce01 |
| Last heartbeat | 2025-11-26 18:52:47 |
| From worker node | Hostname | dune001-479155.0-lcg2655.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 1048576000 (1000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-11-26 18:50:16 |
| Input files | fardet-hd:nu_dune10kt_1x2x6_1408_315_20230826T093333Z_gen_g4_detsim_hitreco__20240229T170759Z_reco2.root
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-11-26 18:52:47 |
| Saved logs | justin-logs:258940.26-dunegpschedd01.fnal.gov.logs.tgz |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/localProducts_larsoft_v10_08_00_e26_prof
PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/localProducts_larsoft_v10_08_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/localProducts_larsoft_v10_08_00_e26_prof
local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/localProducts_larsoft_v10_08_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/fb/70/nu_dune10kt_1x2x6_1408_315_20230826T093333Z_gen_g4_detsim_hitreco__20240229T170759Z_reco2.root
../justin-jobscript: line 64: 1308 Segmentation fault (core dumped) lar -c $FCL_FILE $events_option "$pfn" > ${fname}_reco_${now}.log 2>&1
lar exit code 139
=== Start last 100 lines of lar log file ===
\_ MCPDG 2212, Energy 1.00423, Dist. 3.83336, nMCHits 6 (3, 3, 0)
\_ MCPDG 2212, Energy 0.974124, Dist. 1.27683, nMCHits 3 (2, 1, 0)
\_ MCPDG -211, Energy 0.235243, Dist. 25.5273, nMCHits 57 (24, 20, 13)
\_ MCPDG 22, Energy 1.77283e-05, Dist. 0, nMCHits 3 (0, 2, 1)
\_ MCPDG 2212, Energy 1.18182, Dist. 35.7743, nMCHits 3 (1, 2, 0)
\_ MCPDG 22, Energy 0.0443321, Dist. 0, nMCHits 22 (8, 11, 3)
\_ MCPDG 22, Energy 0.746378, Dist. 0, nMCHits 989 (314, 349, 326)
\_ MCPDG 211, Energy 0.725172, Dist. 8.4977, nMCHits 5 (3, 2, 0)
\_ MCPDG 22, Energy 0.000163579, Dist. 0, nMCHits 1 (1, 0, 0)
\_ MCPDG 22, Energy 0.0057105, Dist. 0, nMCHits 10 (4, 4, 2)
\_ MCPDG 2212, Energy 1.11253, Dist. 14.7918, nMCHits 27 (9, 16, 2)
\_ MCPDG 2212, Energy 1.02684, Dist. 6.4143, nMCHits 15 (7, 8, 0)
\_ MCPDG 2212, Energy 0.981027, Dist. 1.74165, nMCHits 7 (3, 3, 1)
\_ MCPDG 22, Energy 0.00382185, Dist. 0, nMCHits 9 (4, 4, 1)
\_ MCPDG -211, Energy 0.722643, Dist. 95.8623, nMCHits 369 (96, 137, 136)
\_ MCPDG 22, Energy 0.00656567, Dist. 0, nMCHits 13 (4, 4, 5)
\_ MCPDG 22, Energy 0.187759, Dist. 0, nMCHits 243 (63, 101, 79)
\_ MCPDG 22, Energy 0.262144, Dist. 0, nMCHits 103 (52, 33, 18)
\_ MCPDG 22, Energy 0.0794965, Dist. 0, nMCHits 131 (39, 46, 46)
\_ MCPDG -211, Energy 0.323643, Dist. 2.32208, nMCHits 4 (3, 1, 0)
\_ MCPDG 2212, Energy 1.04679, Dist. 8.97951, nMCHits 5 (3, 2, 0)
\_ MCPDG 22, Energy 0.00411188, Dist. 0.533888, nMCHits 4 (1, 3, 0)
--Primary 3, MCPDG 211, Energy 3.80591, Dist. 123.742, nMCHits 1706 (536, 596, 574)
MCPDG 211, Energy 3.80591, Dist. 123.742, nMCHits 40 (13, 17, 10)
\_ MCPDG 2212, Energy 3.38466, Dist. 58.3247, nMCHits 4 (2, 2, 0)
\_ MCPDG 2212, Energy 2.80523, Dist. 37.0228, nMCHits 40 (9, 19, 12)
\_ MCPDG 2212, Energy 1.17078, Dist. 15.7012, nMCHits 73 (20, 26, 27)
\_ MCPDG 22, Energy 0.00489883, Dist. 0, nMCHits 5 (0, 1, 4)
\_ MCPDG 1000180370, Energy 34.4261, Dist. 0.000222171, nMCHits 3 (0, 2, 1)
\_ MCPDG 2212, Energy 0.974113, Dist. 1.28034, nMCHits 5 (1, 2, 2)
\_ MCPDG 22, Energy 1.10243, Dist. 0, nMCHits 1420 (435, 497, 488)
\_ MCPDG 2212, Energy 1.10278, Dist. 18.4825, nMCHits 63 (29, 14, 20)
\_ MCPDG 22, Energy 0.000108803, Dist. 1.97722, nMCHits 1 (0, 0, 1)
\_ MCPDG 1000010020, Energy 1.95634, Dist. 3.12987, nMCHits 8 (4, 2, 2)
\_ MCPDG 2212, Energy 1.15538, Dist. 16.3105, nMCHits 29 (18, 7, 4)
\_ MCPDG 2212, Energy 0.997869, Dist. 3.19488, nMCHits 15 (5, 7, 3)
--Primary 4, MCPDG 13, Energy 3.53297, Dist. 270.645, nMCHits 1118 (416, 303, 399)
MCPDG 13, Energy 3.53297, Dist. 270.645, nMCHits 1118 (416, 303, 399)
--Primary 5, MCPDG 211, Energy 0.983863, Dist. 34.8231, nMCHits 328 (114, 116, 98)
MCPDG 211, Energy 0.983863, Dist. 34.8231, nMCHits 19 (11, 4, 4)
\_ MCPDG 211, Energy 0.381467, Dist. 20.5859, nMCHits 53 (25, 18, 10)
\_ MCPDG 1000010020, Energy 1.90169, Dist. 0.431874, nMCHits 3 (2, 1, 0)
\_ MCPDG 2212, Energy 1.02229, Dist. 5.95567, nMCHits 15 (4, 8, 3)
\_ MCPDG 22, Energy 0.110096, Dist. 0, nMCHits 170 (52, 58, 60)
\_ MCPDG 22, Energy 0.0573841, Dist. 0, nMCHits 67 (20, 26, 21)
\_ MCPDG -211, Energy 0.166564, Dist. 3.26775, nMCHits 1 (0, 1, 0)
--Primary 6, MCPDG 211, Energy 0.610347, Dist. 60.736, nMCHits 312 (98, 103, 111)
MCPDG 211, Energy 0.610347, Dist. 60.736, nMCHits 195 (59, 65, 71)
\_ MCPDG 2212, Energy 1.11801, Dist. 21.5876, nMCHits 95 (28, 29, 38)
\_ MCPDG 2212, Energy 1.04751, Dist. 9.21629, nMCHits 22 (11, 9, 2)
--Primary 7, MCPDG 22, Energy 0.204759, Dist. 0, nMCHits 227 (83, 92, 52)
MCPDG 22, Energy 0.204759, Dist. 0, nMCHits 227 (83, 92, 52)
--Primary 8, MCPDG -211, Energy 0.24801, Dist. 1.53589, nMCHits 124 (66, 27, 31)
MCPDG -211, Energy 0.24801, Dist. 1.53589, nMCHits 6 (1, 3, 2)
\_ MCPDG 13, Energy 0.23092, Dist. 40.8637, nMCHits 115 (64, 23, 28)
\_ MCPDG 11, Energy 0.000515634, Dist. 3.73762e-05, nMCHits 3 (1, 1, 1)
--Primary 9, MCPDG 22, Energy 0.0881551, Dist. 78.701, nMCHits 121 (45, 43, 33)
MCPDG 22, Energy 0.0881551, Dist. 78.701, nMCHits 121 (45, 43, 33)
--Primary 10, MCPDG 22, Energy 0.0612314, Dist. 0, nMCHits 102 (33, 37, 32)
MCPDG 22, Energy 0.0612314, Dist. 0, nMCHits 102 (33, 37, 32)
--Primary 11, MCPDG 22, Energy 0.0590302, Dist. 44.9362, nMCHits 65 (26, 24, 15)
MCPDG 22, Energy 0.0590302, Dist. 44.9362, nMCHits 65 (26, 24, 15)
--Primary 12, MCPDG 2212, Energy 1.0647, Dist. 11.7855, nMCHits 36 (0, 21, 15)
MCPDG 2212, Energy 1.0647, Dist. 11.7855, nMCHits 36 (0, 21, 15)
------------------------------------------------------------------------------------------------
PandoraMonitoring, only able to use default TApplication (limited functionality).
PandoraMonitoring::InitializeEve(): DISPLAY environment not set
PandoraMonitoring::InitializeEve(): Caught TEveException: TEveManager::Create ROOT is running in batch mode.
PandoraMonitoring::InitializeEve(): Attempt to release ROOT from batch mode.
Failure in algorithm Alg0003, LArVisualParticleMonitoring, unknown exception
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_U_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_V_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_W_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/WindowModel_U.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/SplitPosModel_U.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
PandoraMonitoring, only able to use default TApplication (limited functionality).
PandoraMonitoring::InitializeEve(): DISPLAY environment not set
=== End last 100 lines of lar log file ===
mv: cannot stat 'ClusterValidation_WithAlg.root': No such file or directory
mv: cannot stat 'ClusterValidation_WithoutAlg.root': No such file or directory 