Jobsub ID 20676.0@dunegpschedd01.fnal.gov

Jobsub ID	20676.0@dunegpschedd01.fnal.gov
Workflow ID	270
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-03 17:32:48
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2025-08-03 17:36:23
From worker node	Hostname	wn-pep-013.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-03 17:34:22
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_348_20231122T121940Z_gen_g4_detsim_hitreco__20240507T181845Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-03 17:36:23
Saved logs	justin-logs:20676.0-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

PFParticle with ID 1
Begin processing the 189th record. run: 6209058 subRun: 1 event: 69789 at 03-Aug-2025 19:36:06 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 4
Begin processing the 190th record. run: 6209058 subRun: 1 event: 69790 at 03-Aug-2025 19:36:06 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 191st record. run: 6209058 subRun: 1 event: 69791 at 03-Aug-2025 19:36:06 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 2
Begin processing the 192nd record. run: 6209058 subRun: 1 event: 69792 at 03-Aug-2025 19:36:06 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 6
Begin processing the 193rd record. run: 6209058 subRun: 1 event: 69793 at 03-Aug-2025 19:36:06 CEST
Analysing.

Warning: there was no vertex found for PFParticle with ID 3
Warning: there was no track found for track-like PFParticle with ID 3
Warning: there was no track found for track-like PFParticle with ID 4
Begin processing the 194th record. run: 6209058 subRun: 1 event: 69794 at 03-Aug-2025 19:36:06 CEST
Analysing.

Begin processing the 195th record. run: 6209058 subRun: 1 event: 69795 at 03-Aug-2025 19:36:06 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 6
Begin processing the 196th record. run: 6209058 subRun: 1 event: 69796 at 03-Aug-2025 19:36:06 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 8
Begin processing the 197th record. run: 6209058 subRun: 1 event: 69797 at 03-Aug-2025 19:36:06 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 198th record. run: 6209058 subRun: 1 event: 69798 at 03-Aug-2025 19:36:06 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 0
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 199th record. run: 6209058 subRun: 1 event: 69799 at 03-Aug-2025 19:36:06 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 4
Begin processing the 200th record. run: 6209058 subRun: 1 event: 69800 at 03-Aug-2025 19:36:07 CEST
Analysing.

Warning: there was no track found for track-like PFParticle with ID 1
03-Aug-2025 19:36:07 CEST  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/a0/38/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_348_20231122T121940Z_gen_g4_detsim_hitreco__20240507T181845Z_reco2.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                            0.00798706     0.0358285     0.614213      0.0188124     0.0620692       200    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.000421316   0.000731582    0.0081954    0.000663135   0.000559058      200    
end_path:analysistree:AnalysisTree    0.00742208     0.0349788     0.613628      0.0178425     0.0620653       200    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7336.7 MB
  Peak resident set size usage (VmHWM): 987.042 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
C:0 T:16 1231 XUs and 1380 XVs -> 638 XUVs
C:0 T:20 55 XUs and 63 XVs -> 30 XUVs
9242 XUVs total
3026 collection wire objects
9242 potential space points
Neighbour search...
1841430 tests to find 719888 neighbours
Iterating with no regularization...
Begin: 2.21086e+08
0 1.34443e+08
1 1.27533e+08
2 1.26627e+08
3 1.26333e+08
4 1.26184e+08
5 1.26094e+08
Now with regularization...
Begin: 8.02988e+07
0 7.97045e+07
1 7.92982e+07
2 7.89847e+07
3 7.87219e+07
4 7.85105e+07
5 7.83435e+07
6 7.82106e+07
7 7.81045e+07
8 7.80201e+07
9 7.79492e+07
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------

BeamNeutrinos: 

--Primary 0, MCPDG 13, Energy 6.61507, Dist. 914.325, nMCHits 4860 (1610, 1438, 1812)
MCPDG 13, Energy 6.61507, Dist. 914.325, nMCHits 4860 (1610, 1438, 1812)

--Primary 1, MCPDG 211, Energy 1.14796, Dist. 190.286, nMCHits 701 (262, 179, 260)
MCPDG 211, Energy 1.14796, Dist. 190.286, nMCHits 593 (254, 117, 222)
\_ MCPDG 2212, Energy 1.13752, Dist. 25.9893, nMCHits 82 (3, 50, 29)
\_ MCPDG 2212, Energy 1.02077, Dist. 5.58318, nMCHits 26 (5, 12, 9)

--Primary 2, MCPDG -211, Energy 0.530979, Dist. 112.311, nMCHits 308 (127, 112, 69)
MCPDG -211, Energy 0.530979, Dist. 112.311, nMCHits 308 (127, 112, 69)

--Primary 3, MCPDG 211, Energy 0.22017, Dist. 19.8663, nMCHits 92 (32, 36, 24)
MCPDG 211, Energy 0.22017, Dist. 19.8663, nMCHits 47 (13, 25, 9)
   \_ MCPDG -11, Energy 0.0382895, Dist. 8.32047, nMCHits 45 (19, 11, 15)

--Primary 4, MCPDG 2212, Energy 1.17503, Dist. 13.575, nMCHits 39 (20, 8, 11)
MCPDG 2212, Energy 1.17503, Dist. 13.575, nMCHits 36 (19, 7, 10)
\_ MCPDG 2212, Energy 0.970824, Dist. 1.08511, nMCHits 3 (1, 1, 1)

--Primary 5, MCPDG 2212, Energy 1.07976, Dist. 14.349, nMCHits 39 (17, 18, 4)
MCPDG 2212, Energy 1.07976, Dist. 14.349, nMCHits 39 (17, 18, 4)

--Primary 6, MCPDG 2212, Energy 1.01294, Dist. 4.68579, nMCHits 20 (6, 7, 7)
MCPDG 2212, Energy 1.01294, Dist. 4.68579, nMCHits 20 (6, 7, 7)
------------------------------------------------------------------------------------------------
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_U_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_V_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_W_v04_03_00.pt'
Operating in training mode.
The eid is 0
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 78996 bytes.
The eid is 0
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 76316 bytes.
The eid is 0
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 69980 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 78996 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 76316 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 69980 bytes.
Boundary wire vector sizes: 2893, 3179, 3277
minwire 0: 418
minwire 1: 830
minwire 2: 258
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
03-Aug-2025 19:35:11 CEST  Opened output file with pattern "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_348_20231122T121940Z_gen_g4_detsim_hitreco__20240507T181845Z_reco2_graph_2025-08-03T_173426Z.root"
03-Aug-2025 19:35:12 CEST  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/a0/38/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_348_20231122T121940Z_gen_g4_detsim_hitreco__20240507T181845Z_reco2.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.84 MB
  Peak resident set size usage (VmHWM): 1685.38 MB
====================================================================================================
=== End last 100 lines of lar log file ===
=== Generated output files ===
20676.0_dunegpschedd01.fnal.gov.logs.tgz
RootOutput-d6a5-6aa9-6670-b23f.root
TFileService-0c68-8e8c-563c-0b92.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_348_20231122T121940Z_gen_g4_detsim_hitreco__20240507T181845Z_reco2_graph_2025-08-03T_173426Z.log
debugprod.log
jobscript.log
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_348_20231122T121940Z_gen_g4_detsim_hitreco__20240507T181845Z_reco2_graph_2025-08-03T_173426Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_348_20231122T121940Z_gen_g4_detsim_hitreco__20240507T181845Z_reco2_graph_2025-08-03T_173426Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data