Jobsub ID 14068.6@dunegpschedd02.fnal.gov

Jobsub ID	14068.6@dunegpschedd02.fnal.gov
Workflow ID	253
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-02 23:05:47
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_klomp
Last heartbeat	2025-08-02 23:09:45
From worker node	Hostname	wn-pep-004.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-02 23:06:40
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_314_20231203T221658Z_gen_g4_detsim_hitreco__20240509T204820Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-02 23:09:45
Saved logs	justin-logs:14068.6-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

-like PFParticle with ID 0
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 97th record. run: 50577091 subRun: 1 event: 31497 at 03-Aug-2025 01:09:27 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 98th record. run: 50577091 subRun: 1 event: 31498 at 03-Aug-2025 01:09:27 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 99th record. run: 50577091 subRun: 1 event: 31499 at 03-Aug-2025 01:09:28 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 3
Begin processing the 100th record. run: 50577091 subRun: 1 event: 31500 at 03-Aug-2025 01:09:28 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 3
03-Aug-2025 01:09:29 CEST  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/95/01/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_314_20231203T221658Z_gen_g4_detsim_hitreco__20240509T204820Z_reco2.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                              0.2659       0.485056       1.06044      0.485043      0.0927093       100    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.00873996     0.0130782     0.0179266    0.00939204    0.00424885       100    
end_path:analysistree:AnalysisTree     0.256318      0.471867       1.05063      0.469052      0.0934548       100    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7256.94 MB
  Peak resident set size usage (VmHWM): 909.271 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
\_ MCPDG 1000010020, Energy 1.89492, Dist. 0.250931, nMCHits 2 (1, 1, 0)
\_ MCPDG -211, Energy 1.87498, Dist. 68.0889, nMCHits 83 (46, 32, 5)
   \_ MCPDG 2212, Energy 0.944734, Dist. 0.0642749, nMCHits 1 (0, 0, 1)
   \_ MCPDG -211, Energy 0.602508, Dist. 38.9865, nMCHits 83 (25, 35, 23)
      \_ MCPDG -211, Energy 0.504351, Dist. 26.7318, nMCHits 95 (30, 35, 30)
         \_ MCPDG -211, Energy 0.39738, Dist. 29.8985, nMCHits 139 (59, 28, 52)
            \_ MCPDG 2212, Energy 0.960032, Dist. 0.523191, nMCHits 4 (2, 1, 1)
            \_ MCPDG -211, Energy 0.241661, Dist. 11.3718, nMCHits 29 (16, 5, 8)
               \_ MCPDG 2212, Energy 1.0233, Dist. 5.91998, nMCHits 26 (5, 11, 10)
\_ MCPDG -211, Energy 1.17334, Dist. 72.5135, nMCHits 141 (61, 56, 24)
   \_ MCPDG 2212, Energy 1.01646, Dist. 5.20367, nMCHits 18 (2, 10, 6)
   \_ MCPDG 2212, Energy 0.944206, Dist. 0.0544444, nMCHits 5 (2, 2, 1)
\_ MCPDG 211, Energy 0.256707, Dist. 33.3129, nMCHits 62 (25, 34, 3)
   \_ MCPDG -13, Energy 0.109778, Dist. 0.143613, nMCHits 2 (0, 1, 1)
      \_ MCPDG -11, Energy 0.0249833, Dist. 8.02995, nMCHits 42 (14, 16, 12)
   \_ MCPDG 2212, Energy 1.01711, Dist. 5.14007, nMCHits 15 (11, 4, 0)

--Primary 2, MCPDG -211, Energy 0.980426, Dist. 59.5775, nMCHits 578 (119, 254, 205)
MCPDG -211, Energy 0.980426, Dist. 59.5775, nMCHits 133 (12, 69, 52)
\_ MCPDG 2212, Energy 1.0804, Dist. 14.3825, nMCHits 20 (5, 10, 5)
\_ MCPDG 2212, Energy 1.07312, Dist. 13.2583, nMCHits 42 (14, 13, 15)
\_ MCPDG 2212, Energy 1.0047, Dist. 3.65842, nMCHits 12 (7, 3, 2)
\_ MCPDG -211, Energy 0.385483, Dist. 93.7883, nMCHits 371 (81, 159, 131)

--Primary 3, MCPDG 3222, Energy 5.01532, Dist. 9.32406, nMCHits 378 (212, 137, 29)
MCPDG 3222, Energy 5.01532, Dist. 9.32406, nMCHits 10 (9, 0, 1)
\_ MCPDG 211, Energy 0.80321, Dist. 153.629, nMCHits 317 (181, 122, 14)
   \_ MCPDG 2212, Energy 1.21776, Dist. 10.4775, nMCHits 38 (16, 8, 14)
      \_ MCPDG 2212, Energy 0.942831, Dist. 0.0358752, nMCHits 1 (0, 1, 0)
   \_ MCPDG 2212, Energy 1.03877, Dist. 6.62179, nMCHits 12 (6, 6, 0)

--Primary 4, MCPDG 22, Energy 1.26942, Dist. 35.9448, nMCHits 328 (115, 140, 73)
MCPDG 22, Energy 1.26942, Dist. 35.9448, nMCHits 328 (115, 140, 73)

--Primary 5, MCPDG 2212, Energy 1.47773, Dist. 64.1416, nMCHits 277 (78, 77, 122)
MCPDG 2212, Energy 1.47773, Dist. 64.1416, nMCHits 181 (51, 44, 86)
\_ MCPDG 2212, Energy 1.10715, Dist. 19.3399, nMCHits 91 (26, 31, 34)
\_ MCPDG 2212, Energy 0.975726, Dist. 1.38938, nMCHits 1 (0, 1, 0)
\_ MCPDG 2212, Energy 0.963881, Dist. 0.71826, nMCHits 4 (1, 1, 2)

--Primary 6, MCPDG 321, Energy 0.549439, Dist. 4.69337, nMCHits 150 (51, 58, 41)
   \_ MCPDG 22, Energy 0.0888663, Dist. 1.70297, nMCHits 110 (35, 47, 28)
\_ MCPDG 211, Energy 0.175796, Dist. 5.25333, nMCHits 12 (6, 5, 1)
   \_ MCPDG -13, Energy 0.109778, Dist. 0.146717, nMCHits 1 (0, 0, 1)
      \_ MCPDG -11, Energy 0.0260083, Dist. 10.3357, nMCHits 27 (10, 6, 11)

--Primary 7, MCPDG -211, Energy 0.940091, Dist. 74.7248, nMCHits 139 (62, 38, 39)
MCPDG -211, Energy 0.940091, Dist. 74.7248, nMCHits 57 (40, 6, 11)
   \_ MCPDG 2212, Energy 1.03957, Dist. 7.92578, nMCHits 12 (4, 5, 3)
\_ MCPDG 211, Energy 0.231815, Dist. 21.2086, nMCHits 48 (13, 18, 17)
   \_ MCPDG 2212, Energy 1.02194, Dist. 5.89126, nMCHits 22 (5, 9, 8)

--Primary 8, MCPDG -211, Energy 0.254129, Dist. 29.9714, nMCHits 75 (29, 37, 9)
MCPDG -211, Energy 0.254129, Dist. 29.9714, nMCHits 60 (26, 30, 4)
\_ MCPDG 2212, Energy 1.01216, Dist. 4.56605, nMCHits 15 (3, 7, 5)
------------------------------------------------------------------------------------------------
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_U_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_V_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_W_v04_03_00.pt'
Operating in training mode.
The eid is 0
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 95284 bytes.
The eid is 0
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 169932 bytes.
The eid is 0
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 182084 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 95284 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 169932 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 182084 bytes.
Boundary wire vector sizes: 6928, 7434, 3936
minwire 0: 44
minwire 1: 1802
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
03-Aug-2025 01:07:43 CEST  Opened output file with pattern "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_314_20231203T221658Z_gen_g4_detsim_hitreco__20240509T204820Z_reco2_graph_2025-08-02T_230645Z.root"
03-Aug-2025 01:07:45 CEST  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/95/01/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_314_20231203T221658Z_gen_g4_detsim_hitreco__20240509T204820Z_reco2.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8588.54 MB
  Peak resident set size usage (VmHWM): 1673.37 MB
====================================================================================================
=== End last 100 lines of lar log file ===
=== Generated output files ===
14068.6_dunegpschedd02.fnal.gov.logs.tgz
RootOutput-c5ef-cc08-5dac-931a.root
TFileService-856a-08d1-2476-fe14.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_314_20231203T221658Z_gen_g4_detsim_hitreco__20240509T204820Z_reco2_graph_2025-08-02T_230645Z.log
debugprod.log
jobscript.log
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_314_20231203T221658Z_gen_g4_detsim_hitreco__20240509T204820Z_reco2_graph_2025-08-02T_230645Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_314_20231203T221658Z_gen_g4_detsim_hitreco__20240509T204820Z_reco2_graph_2025-08-02T_230645Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data