Jobsub ID 20544.40@dunegpschedd01.fnal.gov

Jobsub ID	20544.40@dunegpschedd01.fnal.gov
Workflow ID	251
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-02 23:49:51
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc02
Last heartbeat	2025-08-03 04:16:56
From worker node	Hostname	wn-la-21.gina.surf.nl
	cpuinfo	AMD EPYC 9754 128-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-02 23:50:53
Input files	monte-carlo-000251-008549
Jobscript	Exit code	0
	Real time	4h (15789s)
	CPU time	4h (15753s = 99%)
	Max RSS bytes	1922478080 (1833 MiB)
Outputting started	2025-08-03 04:14:03
Output files	fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T235102Z_gen_008549_supernova_g4_halfActiveVol_detsim_reco.root
Finished	2025-08-03 04:16:56
Saved logs	justin-logs:20544.40-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

10 XUs and 12 XVs -> 7 XUVs
C:0 T:19 17 XUs and 18 XVs -> 10 XUVs
C:0 T:21 23 XUs and 24 XVs -> 16 XUVs
C:0 T:23 13 XUs and 16 XVs -> 12 XUVs
108 XUVs total
79 collection wire objects
108 potential space points
Neighbour search...
222 tests to find 114 neighbours
Iterating with no regularization...
Begin: 303059
0 266158
1 266093
Now with regularization...
Begin: 188069
0 188063
Begin processing the 44th record. run: 251 subRun: 0 event: 427444 at 03-Aug-2025 06:13:52 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 22 XUs and 20 XVs -> 12 XUVs
C:0 T:3 31 XUs and 35 XVs -> 20 XUVs
C:0 T:5 4 XUs and 4 XVs -> 2 XUVs
C:0 T:7 2 XUs and 3 XVs -> 1 XUVs
C:0 T:9 14 XUs and 13 XVs -> 9 XUVs
C:0 T:11 7 XUs and 4 XVs -> 2 XUVs
C:0 T:13 11 XUs and 7 XVs -> 6 XUVs
C:0 T:15 5 XUs and 8 XVs -> 3 XUVs
C:0 T:17 2 XUs and 2 XVs -> 1 XUVs
C:0 T:19 6 XUs and 4 XVs -> 4 XUVs
C:0 T:21 17 XUs and 13 XVs -> 7 XUVs
C:0 T:23 26 XUs and 23 XVs -> 12 XUVs
79 XUVs total
57 collection wire objects
79 potential space points
Neighbour search...
157 tests to find 78 neighbours
Iterating with no regularization...
Begin: 244335
0 210747
1 210546
Now with regularization...
Begin: 147726
0 147721
Begin processing the 45th record. run: 251 subRun: 0 event: 427445 at 03-Aug-2025 06:13:53 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 15 XUs and 19 XVs -> 12 XUVs
C:0 T:3 14 XUs and 14 XVs -> 10 XUVs
C:0 T:5 9 XUs and 10 XVs -> 6 XUVs
C:0 T:7 6 XUs and 7 XVs -> 4 XUVs
C:0 T:9 5 XUs and 4 XVs -> 3 XUVs
C:0 T:11 7 XUs and 10 XVs -> 6 XUVs
C:0 T:13 23 XUs and 21 XVs -> 11 XUVs
C:0 T:15 10 XUs and 13 XVs -> 8 XUVs
C:0 T:17 12 XUs and 10 XVs -> 6 XUVs
C:0 T:19 5 XUs and 4 XVs -> 4 XUVs
C:0 T:21 46 XUs and 44 XVs -> 28 XUVs
C:0 T:23 17 XUs and 18 XVs -> 9 XUVs
107 XUVs total
81 collection wire objects
107 potential space points
Neighbour search...
229 tests to find 120 neighbours
Iterating with no regularization...
Begin: 332017
0 305952
1 305817
Now with regularization...
Begin: 228786
0 228781
Begin processing the 46th record. run: 251 subRun: 0 event: 427446 at 03-Aug-2025 06:13:53 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 22 XUs and 20 XVs -> 12 XUVs
C:0 T:3 40 XUs and 41 XVs -> 22 XUVs
C:0 T:5 7 XUs and 6 XVs -> 5 XUVs
C:0 T:7 4 XUs and 2 XVs -> 2 XUVs
C:0 T:9 16 XUs and 16 XVs -> 11 XUVs
C:0 T:11 4 XUs and 2 XVs -> 2 XUVs
C:0 T:13 16 XUs and 20 XVs -> 12 XUVs
C:0 T:15 24 XUs and 18 XVs -> 15 XUVs
C:0 T:17 12 XUs and 11 XVs -> 6 XUVs
C:0 T:19 14 XUs and 12 XVs -> 9 XUVs
C:0 T:21 22 XUs and 19 XVs -> 8 XUVs
C:0 T:23 20 XUs and 20 XVs -> 13 XUVs
117 XUVs total
84 collection wire objects
117 potential space points
Neighbour search...
261 tests to find 140 neighbours
Iterating with no regularization...
Begin: 350194
0 285850
1 283956
2 283632
3 283622
Now with regularization...
Begin: 194014
0 194004
Begin processing the 47th record. run: 251 subRun: 0 event: 427447 at 03-Aug-2025 06:13:54 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 12 XUs and 12 XVs -> 7 XUVs
C:0 T:3 13 XUs and 19 XVs -> 9 XUVs
C:0 T:5 18 XUs and 18 XVs -> 10 XUVs
C:0 T:7 9 XUs and 10 XVs -> 5 XUVs
C:0 T:9 7 XUs and 10 XVs -> 5 XUVs
C:0 T:11 16 XUs and 15 XVs -> 7 XUVs
C:0 T:13 17 XUs and 14 XVs -> 11 XUVs
C:0 T:15 7 XUs and 10 XVs -> 7 XUVs
C:0 T:17 16 XUs and 15 XVs -> 8 XUVs
C:0 T:19 8 XUs and 6 XVs -> 4 XUVs
C:0 T:21 17 XUs and 21 XVs -> 11 XUVs
C:0 T:23 20 XUs and 19 XVs -> 12 XUVs
96 XUVs total
72 collection wire objects
96 potential space points
Neighbour search...
204 tests to find 104 neighbours
Iterating with no regularization...
Begin: 304410
0 283994
1 283654
2 283644
Now with regularization...
Begin: 202933
0 202929
Begin processing the 48th record. run: 251 subRun: 0 event: 427448 at 03-Aug-2025 06:13:54 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 24 XUs and 34 XVs -> 18 XUVs
C:0 T:3 29 XUs and 28 XVs -> 15 XUVs
C:0 T:5 20 XUs and 19 XVs -> 13 XUVs
C:0 T:7 5 XUs and 3 XVs -> 3 XUVs
C:0 T:9 31 XUs and 27 XVs -> 19 XUVs
C:0 T:11 6 XUs and 6 XVs -> 3 XUVs
C:0 T:13 13 XUs and 14 XVs -> 5 XUVs
C:0 T:15 8 XUs and 9 XVs -> 5 XUVs
C:0 T:17 26 XUs and 26 XVs -> 19 XUVs
C:0 T:19 4 XUs and 6 XVs -> 4 XUVs
C:0 T:21 29 XUs and 34 XVs -> 17 XUVs
C:0 T:23 30 XUs and 26 XVs -> 18 XUVs
139 XUVs total
93 collection wire objects
139 potential space points
Neighbour search...
309 tests to find 170 neighbours
Iterating with no regularization...
Begin: 347426
0 282616
1 282565
Now with regularization...
Begin: 182276
0 182266
Begin processing the 49th record. run: 251 subRun: 0 event: 427449 at 03-Aug-2025 06:13:54 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 21 XUs and 26 XVs -> 13 XUVs
C:0 T:3 18 XUs and 16 XVs -> 10 XUVs
C:0 T:5 12 XUs and 15 XVs -> 8 XUVs
C:0 T:7 12 XUs and 9 XVs -> 8 XUVs
C:0 T:9 10 XUs and 10 XVs -> 7 XUVs
C:0 T:11 12 XUs and 12 XVs -> 7 XUVs
C:0 T:13 26 XUs and 28 XVs -> 20 XUVs
C:0 T:15 5 XUs and 6 XVs -> 3 XUVs
C:0 T:17 16 XUs and 12 XVs -> 11 XUVs
C:0 T:19 7 XUs and 6 XVs -> 4 XUVs
C:0 T:21 16 XUs and 12 XVs -> 8 XUVs
C:0 T:23 30 XUs and 31 XVs -> 19 XUVs
118 XUVs total
84 collection wire objects
118 potential space points
Neighbour search...
290 tests to find 172 neighbours
Iterating with no regularization...
Begin: 469950
0 402546
1 401859
2 401842
Now with regularization...
Begin: 312634
0 312625
Begin processing the 50th record. run: 251 subRun: 0 event: 427450 at 03-Aug-2025 06:13:55 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 9 XUs and 8 XVs -> 6 XUVs
C:0 T:3 32 XUs and 30 XVs -> 16 XUVs
C:0 T:5 10 XUs and 7 XVs -> 4 XUVs
C:0 T:7 8 XUs and 8 XVs -> 4 XUVs
C:0 T:9 17 XUs and 17 XVs -> 11 XUVs
C:0 T:11 7 XUs and 8 XVs -> 3 XUVs
C:0 T:13 8 XUs and 8 XVs -> 5 XUVs
C:0 T:15 8 XUs and 7 XVs -> 4 XUVs
C:0 T:17 15 XUs and 16 XVs -> 8 XUVs
C:0 T:19 6 XUs and 7 XVs -> 4 XUVs
C:0 T:21 10 XUs and 10 XVs -> 4 XUVs
C:0 T:23 16 XUs and 17 XVs -> 12 XUVs
81 XUVs total
66 collection wire objects
81 potential space points
Neighbour search...
147 tests to find 64 neighbours
Iterating with no regularization...
Begin: 371601
0 336066
1 335987
Now with regularization...
Begin: 262514
0 262507
03-Aug-2025 06:13:58 CEST  Closed output file "prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T235102Z_gen_008549_supernova_g4_halfActiveVol_detsim_reco.root"
03-Aug-2025 06:13:58 CEST  Closed input file "prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T235102Z_gen_008549_supernova_g4_halfActiveVol_detsim.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0350576     0.0393094     0.0466108     0.0389602    0.00197185       50     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00031444    0.000511696   0.00103207    0.000507328   0.000114954      50     
reco:gaushit:GausHitFinder                     0.0267545     0.0292334     0.0309764     0.0290358    0.000914304      50     
reco:spsolve:SpacePointSolver                 0.00212718    0.00284313    0.00516465    0.00276569    0.000458078      50     
reco:hitfd:DisambigFromSpacePoints            0.000667357   0.000871256   0.00222216    0.000842348   0.000209199      50     
reco:rns:RandomNumberSaver                    2.0852e-05    2.5987e-05    0.000143174   2.2424e-05    1.73167e-05      50     
[art]:TriggerResults:TriggerResultInserter     9.734e-06    1.13792e-05   3.5443e-05    1.0826e-05    3.51261e-06      50     
end_path:out1:RootOutput                       4.336e-06    5.68608e-06   3.1647e-05     4.987e-06    4.00884e-06      50     
end_path:out1:RootOutput(write)               0.00462004    0.00555613    0.00702698    0.00547552    0.000471517      50     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1773.73 MB
  Peak resident set size usage (VmHWM): 889.577 MB
====================================================================================================
Art has completed and will exit with status 0.
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T235102Z_gen_008549.root 110004945
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T235102Z_gen_008549_supernova_g4_halfActiveVol.root 722840908
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T235102Z_gen_008549_supernova_g4_halfActiveVol_detsim.root 1840859839
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T235102Z_gen_008549_supernova_g4_halfActiveVol_detsim_hist.root 12750
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T235102Z_gen_008549_supernova_g4_halfActiveVol_detsim_reco.root 1844304374
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T235102Z_gen_008549_supernova_g4_halfActiveVol_detsim_reco.root.ext.json 1606
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T235102Z_gen_008549_supernova_g4_halfActiveVol_detsim_reco.root.json 1835
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T235102Z_gen_008549_supernova_g4_halfActiveVol_detsim_reco.root.temp.json 1835