Jobsub ID 258940.60@dunegpschedd01.fnal.gov
| Jobsub ID | 258940.60@dunegpschedd01.fnal.gov |
| Workflow ID | 10817 |
| Stage ID | 1 |
| User name | imawby@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 1048576000 (1000 MiB) |
| Wall seconds limit | 7200 (2 hours) |
| Submitted time | 2025-11-26 18:48:50 |
| Site | UK_QMUL |
| Entry | DUNE_UK_London_QMUL_arcce03 |
| Last heartbeat | 2025-11-26 18:55:35 |
| From worker node | Hostname | cn032.htc.esc.qmul |
| cpuinfo | Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 1048576000 (1000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-11-26 18:50:33 |
| Input files | fardet-hd:nu_dune10kt_1x2x6_1100_400_20230826T034308Z_gen_g4_detsim_hitreco__20240229T174341Z_reco2.root
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-11-26 18:55:35 |
| Saved logs | justin-logs:258940.60-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
279f869/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/localProducts_larsoft_v10_08_00_e26_prof
PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/localProducts_larsoft_v10_08_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/localProducts_larsoft_v10_08_00_e26_prof
local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/localProducts_larsoft_v10_08_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/fc/b3/nu_dune10kt_1x2x6_1100_400_20230826T034308Z_gen_g4_detsim_hitreco__20240229T174341Z_reco2.root
../justin-jobscript: line 64: 1311 Segmentation fault (core dumped) lar -c $FCL_FILE $events_option "$pfn" > ${fname}_reco_${now}.log 2>&1
lar exit code 139
=== Start last 100 lines of lar log file ===
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
26-Nov-2025 18:51:52 GMT Initiating request to open input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/fc/b3/nu_dune10kt_1x2x6_1100_400_20230826T034308Z_gen_g4_detsim_hitreco__20240229T174341Z_reco2.root"
26-Nov-2025 18:52:43 GMT Opened input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/fc/b3/nu_dune10kt_1x2x6_1100_400_20230826T034308Z_gen_g4_detsim_hitreco__20240229T174341Z_reco2.root"
Begin processing the 1st record. run: 1100 subRun: 1 event: 40001 at 26-Nov-2025 18:52:44 GMT
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
BeamNeutrinos:
--Primary 0, MCPDG 211, Energy 8.18607, Dist. 67.8892, nMCHits 6123 (1842, 2337, 1944)
MCPDG 211, Energy 8.18607, Dist. 67.8892, nMCHits 343 (91, 119, 133)
\_ MCPDG 22, Energy 0.828647, Dist. 0, nMCHits 982 (278, 401, 303)
\_ MCPDG 22, Energy 0.503878, Dist. 0, nMCHits 563 (153, 233, 177)
\_ MCPDG 2212, Energy 1.14498, Dist. 27.5202, nMCHits 70 (39, 16, 15)
\_ MCPDG 2212, Energy 1.12505, Dist. 14.9307, nMCHits 23 (7, 8, 8)
\_ MCPDG 22, Energy 0.00561431, Dist. 0, nMCHits 6 (2, 3, 1)
\_ MCPDG 2212, Energy 1.03077, Dist. 6.85717, nMCHits 23 (13, 4, 6)
\_ MCPDG 2212, Energy 1.06428, Dist. 11.5819, nMCHits 43 (16, 16, 11)
\_ MCPDG 211, Energy 0.877605, Dist. 95.7838, nMCHits 337 (53, 156, 128)
\_ MCPDG 211, Energy 0.401251, Dist. 66.1146, nMCHits 166 (31, 97, 38)
\_ MCPDG 2212, Energy 1.08302, Dist. 15.0632, nMCHits 40 (23, 9, 8)
\_ MCPDG 2212, Energy 1.02912, Dist. 6.53087, nMCHits 18 (9, 7, 2)
\_ MCPDG 22, Energy 0.116725, Dist. 0, nMCHits 163 (47, 71, 45)
\_ MCPDG 22, Energy 0.103539, Dist. 0, nMCHits 172 (51, 64, 57)
\_ MCPDG 211, Energy 0.769506, Dist. 179.743, nMCHits 690 (127, 304, 259)
\_ MCPDG 2212, Energy 1.06017, Dist. 11.1149, nMCHits 56 (23, 12, 21)
\_ MCPDG 1000180400, Energy 37.2156, Dist. 1.52588e-05, nMCHits 4 (0, 2, 2)
\_ MCPDG 211, Energy 0.222219, Dist. 15.9817, nMCHits 53 (26, 7, 20)
\_ MCPDG 2212, Energy 0.999294, Dist. 3.2972, nMCHits 11 (4, 4, 3)
\_ MCPDG 22, Energy 0.545, Dist. 0, nMCHits 654 (255, 199, 200)
\_ MCPDG 22, Energy 0.219171, Dist. 0, nMCHits 197 (90, 63, 44)
\_ MCPDG -211, Energy 0.706119, Dist. 39.4066, nMCHits 115 (19, 59, 37)
\_ MCPDG 2212, Energy 1.02593, Dist. 6.1602, nMCHits 22 (3, 13, 6)
\_ MCPDG -211, Energy 0.508703, Dist. 28.9942, nMCHits 125 (29, 48, 48)
\_ MCPDG 22, Energy 0.361063, Dist. 0, nMCHits 492 (147, 176, 169)
\_ MCPDG 22, Energy 0.326305, Dist. 0, nMCHits 350 (130, 141, 79)
\_ MCPDG 211, Energy 0.552659, Dist. 41.5155, nMCHits 111 (35, 38, 38)
\_ MCPDG 211, Energy 0.41855, Dist. 9.15286, nMCHits 18 (10, 7, 1)
\_ MCPDG 2212, Energy 1.0711, Dist. 13.0024, nMCHits 38 (16, 9, 13)
\_ MCPDG 2212, Energy 1.02205, Dist. 5.72111, nMCHits 13 (5, 6, 2)
\_ MCPDG 211, Energy 0.375784, Dist. 89.5388, nMCHits 164 (88, 24, 52)
\_ MCPDG -13, Energy 0.109778, Dist. 0.153239, nMCHits 1 (0, 0, 1)
\_ MCPDG -11, Energy 0.0484206, Dist. 9.82635, nMCHits 28 (15, 4, 9)
\_ MCPDG -211, Energy 0.201829, Dist. 12.8561, nMCHits 28 (6, 16, 6)
\_ MCPDG 1000020030, Energy 2.81846, Dist. 0.0165214, nMCHits 4 (1, 1, 2)
--Primary 1, MCPDG 13, Energy 1.21234, Dist. 489.441, nMCHits 1990 (869, 386, 735)
MCPDG 13, Energy 1.21234, Dist. 489.441, nMCHits 1940 (856, 367, 717)
\_ MCPDG 11, Energy 0.0373451, Dist. 8.87245, nMCHits 50 (13, 19, 18)
--Primary 2, MCPDG 2212, Energy 2.50883, Dist. 119.436, nMCHits 943 (338, 269, 336)
MCPDG 2212, Energy 2.50883, Dist. 119.436, nMCHits 518 (145, 176, 197)
\_ MCPDG 2212, Energy 1.30725, Dist. 71.0016, nMCHits 208 (136, 11, 61)
\_ MCPDG 2212, Energy 1.13195, Dist. 25.0574, nMCHits 112 (29, 43, 40)
\_ MCPDG 2212, Energy 1.12713, Dist. 23.8686, nMCHits 91 (23, 33, 35)
\_ MCPDG 2212, Energy 0.980411, Dist. 1.75859, nMCHits 5 (1, 3, 1)
\_ MCPDG 2212, Energy 0.978107, Dist. 1.54528, nMCHits 9 (4, 3, 2)
--Primary 3, MCPDG 22, Energy 0.736663, Dist. 0, nMCHits 939 (267, 347, 325)
MCPDG 22, Energy 0.736663, Dist. 0, nMCHits 939 (267, 347, 325)
--Primary 4, MCPDG 22, Energy 0.409376, Dist. 0, nMCHits 559 (214, 166, 179)
MCPDG 22, Energy 0.409376, Dist. 0, nMCHits 559 (214, 166, 179)
--Primary 5, MCPDG 22, Energy 0.291691, Dist. 0, nMCHits 394 (123, 142, 129)
MCPDG 22, Energy 0.291691, Dist. 0, nMCHits 394 (123, 142, 129)
--Primary 6, MCPDG 22, Energy 0.164593, Dist. 0, nMCHits 240 (63, 97, 80)
MCPDG 22, Energy 0.164593, Dist. 0, nMCHits 240 (63, 97, 80)
--Primary 7, MCPDG 2212, Energy 1.05832, Dist. 10.7129, nMCHits 30 (12, 14, 4)
MCPDG 2212, Energy 1.05832, Dist. 10.7129, nMCHits 30 (12, 14, 4)
--Primary 8, MCPDG 2212, Energy 1.0258, Dist. 6.21931, nMCHits 25 (7, 9, 9)
MCPDG 2212, Energy 1.0258, Dist. 6.21931, nMCHits 25 (7, 9, 9)
------------------------------------------------------------------------------------------------
PandoraMonitoring, only able to use default TApplication (limited functionality).
PandoraMonitoring::InitializeEve(): DISPLAY environment not set
PandoraMonitoring::InitializeEve(): Caught TEveException: TEveManager::Create ROOT is running in batch mode.
PandoraMonitoring::InitializeEve(): Attempt to release ROOT from batch mode.
Failure in algorithm Alg0003, LArVisualParticleMonitoring, unknown exception
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_U_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_V_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_W_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/WindowModel_U.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/SplitPosModel_U.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/2401d7e66a71c813bf259d2986daa79e4279f869/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
PandoraMonitoring, only able to use default TApplication (limited functionality).
PandoraMonitoring::InitializeEve(): DISPLAY environment not set
=== End last 100 lines of lar log file ===
mv: cannot stat 'ClusterValidation_WithAlg.root': No such file or directory
mv: cannot stat 'ClusterValidation_WithoutAlg.root': No such file or directory 