justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258933.142@dunegpschedd01.fnal.gov

Jobsub ID	258933.142@dunegpschedd01.fnal.gov
Workflow ID	10818
Stage ID	1
User name	imawby@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	7200 (2 hours)
Submitted time	2025-11-26 18:40:48
Site	UK_QMUL
Entry	DUNE_UK_London_QMUL_arcce03
Last heartbeat	2025-11-26 18:47:01
From worker node	Hostname	cn501.htc.esc.qmul
	cpuinfo	Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-11-26 18:41:52
Input files	fardet-hd:nu_dune10kt_1x2x6_1100_400_20230826T034308Z_gen_g4_detsim_hitreco__20240229T174341Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-11-26 18:47:01
Saved logs	justin-logs:258933.142-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

e6b7af9/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/localProducts_larsoft_v10_08_00_e26_prof

PRODUCTS=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/localProducts_larsoft_v10_08_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/localProducts_larsoft_v10_08_00_e26_prof

local product directory is /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/localProducts_larsoft_v10_08_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/fc/b3/nu_dune10kt_1x2x6_1100_400_20230826T034308Z_gen_g4_detsim_hitreco__20240229T174341Z_reco2.root
../justin-jobscript: line 64:  1314 Segmentation fault      (core dumped) lar -c $FCL_FILE $events_option "$pfn" > ${fname}_reco_${now}.log 2>&1
lar exit code 139
=== Start last 100 lines of lar log file ===
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
26-Nov-2025 18:43:09 GMT  Initiating request to open input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/fc/b3/nu_dune10kt_1x2x6_1100_400_20230826T034308Z_gen_g4_detsim_hitreco__20240229T174341Z_reco2.root"
26-Nov-2025 18:44:00 GMT  Opened input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/fc/b3/nu_dune10kt_1x2x6_1100_400_20230826T034308Z_gen_g4_detsim_hitreco__20240229T174341Z_reco2.root"
Begin processing the 1st record. run: 1100 subRun: 1 event: 40001 at 26-Nov-2025 18:44:05 GMT
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------

BeamNeutrinos: 

--Primary 0, MCPDG 211, Energy 8.18607, Dist. 67.8892, nMCHits 6123 (1842, 2337, 1944)
MCPDG 211, Energy 8.18607, Dist. 67.8892, nMCHits 343 (91, 119, 133)
   \_ MCPDG 22, Energy 0.828647, Dist. 0, nMCHits 982 (278, 401, 303)
   \_ MCPDG 22, Energy 0.503878, Dist. 0, nMCHits 563 (153, 233, 177)
\_ MCPDG 2212, Energy 1.14498, Dist. 27.5202, nMCHits 70 (39, 16, 15)
\_ MCPDG 2212, Energy 1.12505, Dist. 14.9307, nMCHits 23 (7, 8, 8)
   \_ MCPDG 22, Energy 0.00561431, Dist. 0, nMCHits 6 (2, 3, 1)
   \_ MCPDG 2212, Energy 1.03077, Dist. 6.85717, nMCHits 23 (13, 4, 6)
\_ MCPDG 2212, Energy 1.06428, Dist. 11.5819, nMCHits 43 (16, 16, 11)
\_ MCPDG 211, Energy 0.877605, Dist. 95.7838, nMCHits 337 (53, 156, 128)
   \_ MCPDG 211, Energy 0.401251, Dist. 66.1146, nMCHits 166 (31, 97, 38)
      \_ MCPDG 2212, Energy 1.08302, Dist. 15.0632, nMCHits 40 (23, 9, 8)
      \_ MCPDG 2212, Energy 1.02912, Dist. 6.53087, nMCHits 18 (9, 7, 2)
      \_ MCPDG 22, Energy 0.116725, Dist. 0, nMCHits 163 (47, 71, 45)
      \_ MCPDG 22, Energy 0.103539, Dist. 0, nMCHits 172 (51, 64, 57)
\_ MCPDG 211, Energy 0.769506, Dist. 179.743, nMCHits 690 (127, 304, 259)
   \_ MCPDG 2212, Energy 1.06017, Dist. 11.1149, nMCHits 56 (23, 12, 21)
      \_ MCPDG 1000180400, Energy 37.2156, Dist. 1.52588e-05, nMCHits 4 (0, 2, 2)
   \_ MCPDG 211, Energy 0.222219, Dist. 15.9817, nMCHits 53 (26, 7, 20)
      \_ MCPDG 2212, Energy 0.999294, Dist. 3.2972, nMCHits 11 (4, 4, 3)
   \_ MCPDG 22, Energy 0.545, Dist. 0, nMCHits 654 (255, 199, 200)
   \_ MCPDG 22, Energy 0.219171, Dist. 0, nMCHits 197 (90, 63, 44)
\_ MCPDG -211, Energy 0.706119, Dist. 39.4066, nMCHits 115 (19, 59, 37)
   \_ MCPDG 2212, Energy 1.02593, Dist. 6.1602, nMCHits 22 (3, 13, 6)
   \_ MCPDG -211, Energy 0.508703, Dist. 28.9942, nMCHits 125 (29, 48, 48)
   \_ MCPDG 22, Energy 0.361063, Dist. 0, nMCHits 492 (147, 176, 169)
   \_ MCPDG 22, Energy 0.326305, Dist. 0, nMCHits 350 (130, 141, 79)
\_ MCPDG 211, Energy 0.552659, Dist. 41.5155, nMCHits 111 (35, 38, 38)
   \_ MCPDG 211, Energy 0.41855, Dist. 9.15286, nMCHits 18 (10, 7, 1)
      \_ MCPDG 2212, Energy 1.0711, Dist. 13.0024, nMCHits 38 (16, 9, 13)
      \_ MCPDG 2212, Energy 1.02205, Dist. 5.72111, nMCHits 13 (5, 6, 2)
\_ MCPDG 211, Energy 0.375784, Dist. 89.5388, nMCHits 164 (88, 24, 52)
   \_ MCPDG -13, Energy 0.109778, Dist. 0.153239, nMCHits 1 (0, 0, 1)
      \_ MCPDG -11, Energy 0.0484206, Dist. 9.82635, nMCHits 28 (15, 4, 9)
\_ MCPDG -211, Energy 0.201829, Dist. 12.8561, nMCHits 28 (6, 16, 6)
   \_ MCPDG 1000020030, Energy 2.81846, Dist. 0.0165214, nMCHits 4 (1, 1, 2)

--Primary 1, MCPDG 13, Energy 1.21234, Dist. 489.441, nMCHits 1990 (869, 386, 735)
MCPDG 13, Energy 1.21234, Dist. 489.441, nMCHits 1940 (856, 367, 717)
\_ MCPDG 11, Energy 0.0373451, Dist. 8.87245, nMCHits 50 (13, 19, 18)

--Primary 2, MCPDG 2212, Energy 2.50883, Dist. 119.436, nMCHits 943 (338, 269, 336)
MCPDG 2212, Energy 2.50883, Dist. 119.436, nMCHits 518 (145, 176, 197)
\_ MCPDG 2212, Energy 1.30725, Dist. 71.0016, nMCHits 208 (136, 11, 61)
\_ MCPDG 2212, Energy 1.13195, Dist. 25.0574, nMCHits 112 (29, 43, 40)
\_ MCPDG 2212, Energy 1.12713, Dist. 23.8686, nMCHits 91 (23, 33, 35)
\_ MCPDG 2212, Energy 0.980411, Dist. 1.75859, nMCHits 5 (1, 3, 1)
\_ MCPDG 2212, Energy 0.978107, Dist. 1.54528, nMCHits 9 (4, 3, 2)

--Primary 3, MCPDG 22, Energy 0.736663, Dist. 0, nMCHits 939 (267, 347, 325)
MCPDG 22, Energy 0.736663, Dist. 0, nMCHits 939 (267, 347, 325)

--Primary 4, MCPDG 22, Energy 0.409376, Dist. 0, nMCHits 559 (214, 166, 179)
MCPDG 22, Energy 0.409376, Dist. 0, nMCHits 559 (214, 166, 179)

--Primary 5, MCPDG 22, Energy 0.291691, Dist. 0, nMCHits 394 (123, 142, 129)
MCPDG 22, Energy 0.291691, Dist. 0, nMCHits 394 (123, 142, 129)

--Primary 6, MCPDG 22, Energy 0.164593, Dist. 0, nMCHits 240 (63, 97, 80)
MCPDG 22, Energy 0.164593, Dist. 0, nMCHits 240 (63, 97, 80)

--Primary 7, MCPDG 2212, Energy 1.05832, Dist. 10.7129, nMCHits 30 (12, 14, 4)
MCPDG 2212, Energy 1.05832, Dist. 10.7129, nMCHits 30 (12, 14, 4)

--Primary 8, MCPDG 2212, Energy 1.0258, Dist. 6.21931, nMCHits 25 (7, 9, 9)
MCPDG 2212, Energy 1.0258, Dist. 6.21931, nMCHits 25 (7, 9, 9)
------------------------------------------------------------------------------------------------
PandoraMonitoring, only able to use default TApplication (limited functionality).
PandoraMonitoring::InitializeEve(): DISPLAY environment not set
PandoraMonitoring::InitializeEve(): Caught TEveException: TEveManager::Create ROOT is running in batch mode.
PandoraMonitoring::InitializeEve(): Attempt to release ROOT from batch mode.
Failure in algorithm Alg0003, LArVisualParticleMonitoring, unknown exception
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_U_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_V_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_W_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/WindowModel_U.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/SplitPosModel_U.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/d86f056b4b72fb9cbc30fcda95eeb4c9fe6b7af9/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
PandoraMonitoring, only able to use default TApplication (limited functionality).
PandoraMonitoring::InitializeEve(): DISPLAY environment not set
=== End last 100 lines of lar log file ===
mv: cannot stat 'ClusterValidation_WithAlg.root': No such file or directory
mv: cannot stat 'ClusterValidation_WithoutAlg.root': No such file or directory

justIN time: 2025-12-20 00:47:28 UTC       justIN version: 01.05.03