Jobsub ID 20400.87@dunegpschedd01.fnal.gov

Jobsub ID	20400.87@dunegpschedd01.fnal.gov
Workflow ID	249
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-02 09:07:09
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc01
Last heartbeat	2025-08-02 16:24:34
From worker node	Hostname	wn-lb-15.gina.surf.nl
	cpuinfo	AMD EPYC 9754 128-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-02 10:53:27
Input files	monte-carlo-000249-000810
Jobscript	Exit code	0
	Real time	5h (19639s)
	CPU time	5h (19608s = 99%)
	Max RSS bytes	1992925184 (1900 MiB)
Outputting started	2025-08-02 16:20:47
Output files	fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T105335Z_gen_000810_supernova_g4_detsim_reco.root
Finished	2025-08-02 16:24:34
Saved logs	justin-logs:20400.87-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

8141
Now with regularization...
Begin: 229186
0 229179
Begin processing the 46th record. run: 249 subRun: 0 event: 40496 at 02-Aug-2025 18:20:36 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 22 XUs and 18 XVs -> 12 XUVs
C:0 T:1 16 XUs and 16 XVs -> 12 XUVs
C:0 T:2 13 XUs and 15 XVs -> 8 XUVs
C:0 T:3 22 XUs and 19 XVs -> 10 XUVs
C:0 T:4 6 XUs and 8 XVs -> 6 XUVs
C:0 T:5 11 XUs and 12 XVs -> 6 XUVs
C:0 T:6 3 XUs and 2 XVs -> 1 XUVs
C:0 T:7 10 XUs and 11 XVs -> 6 XUVs
C:0 T:8 5 XUs and 6 XVs -> 4 XUVs
C:0 T:9 8 XUs and 4 XVs -> 4 XUVs
C:0 T:10 5 XUs and 3 XVs -> 3 XUVs
C:0 T:11 10 XUs and 8 XVs -> 7 XUVs
C:0 T:12 3 XUs and 7 XVs -> 2 XUVs
C:0 T:13 5 XUs and 6 XVs -> 5 XUVs
C:0 T:14 5 XUs and 5 XVs -> 3 XUVs
C:0 T:15 0 XUs and 0 XVs -> 0 XUVs
C:0 T:16 7 XUs and 4 XVs -> 2 XUVs
C:0 T:17 5 XUs and 5 XVs -> 4 XUVs
C:0 T:18 9 XUs and 8 XVs -> 6 XUVs
C:0 T:19 4 XUs and 6 XVs -> 1 XUVs
C:0 T:20 22 XUs and 20 XVs -> 11 XUVs
C:0 T:21 10 XUs and 11 XVs -> 8 XUVs
C:0 T:22 17 XUs and 16 XVs -> 10 XUVs
C:0 T:23 22 XUs and 25 XVs -> 13 XUVs
144 XUVs total
103 collection wire objects
144 potential space points
Neighbour search...
308 tests to find 164 neighbours
Iterating with no regularization...
Begin: 277701
0 231779
1 231547
2 231544
Now with regularization...
Begin: 143793
0 143788
Begin processing the 47th record. run: 249 subRun: 0 event: 40497 at 02-Aug-2025 18:20:36 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 18 XUs and 22 XVs -> 10 XUVs
C:0 T:1 17 XUs and 19 XVs -> 12 XUVs
C:0 T:2 18 XUs and 14 XVs -> 9 XUVs
C:0 T:3 17 XUs and 14 XVs -> 11 XUVs
C:0 T:4 6 XUs and 4 XVs -> 2 XUVs
C:0 T:5 2 XUs and 4 XVs -> 2 XUVs
C:0 T:6 10 XUs and 9 XVs -> 6 XUVs
C:0 T:7 10 XUs and 10 XVs -> 8 XUVs
C:0 T:8 3 XUs and 3 XVs -> 1 XUVs
C:0 T:9 4 XUs and 1 XVs -> 0 XUVs
C:0 T:10 6 XUs and 4 XVs -> 4 XUVs
C:0 T:11 1 XUs and 3 XVs -> 0 XUVs
C:0 T:12 8 XUs and 7 XVs -> 5 XUVs
C:0 T:13 8 XUs and 8 XVs -> 3 XUVs
C:0 T:14 3 XUs and 6 XVs -> 3 XUVs
C:0 T:15 1 XUs and 2 XVs -> 0 XUVs
C:0 T:16 6 XUs and 5 XVs -> 4 XUVs
C:0 T:17 5 XUs and 5 XVs -> 5 XUVs
C:0 T:18 1 XUs and 2 XVs -> 1 XUVs
C:0 T:19 4 XUs and 3 XVs -> 2 XUVs
C:0 T:20 14 XUs and 15 XVs -> 8 XUVs
C:0 T:21 12 XUs and 11 XVs -> 5 XUVs
C:0 T:22 30 XUs and 27 XVs -> 18 XUVs
C:0 T:23 32 XUs and 30 XVs -> 16 XUVs
135 XUVs total
99 collection wire objects
135 potential space points
Neighbour search...
281 tests to find 146 neighbours
Iterating with no regularization...
Begin: 349453
0 316241
1 315918
2 315908
Now with regularization...
Begin: 232852
0 232845
Begin processing the 48th record. run: 249 subRun: 0 event: 40498 at 02-Aug-2025 18:20:37 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 15 XUs and 13 XVs -> 8 XUVs
C:0 T:1 28 XUs and 25 XVs -> 16 XUVs
C:0 T:2 16 XUs and 13 XVs -> 7 XUVs
C:0 T:3 29 XUs and 31 XVs -> 16 XUVs
C:0 T:4 19 XUs and 21 XVs -> 9 XUVs
C:0 T:5 12 XUs and 11 XVs -> 5 XUVs
C:0 T:6 6 XUs and 4 XVs -> 4 XUVs
C:0 T:7 5 XUs and 7 XVs -> 3 XUVs
C:0 T:8 1 XUs and 0 XVs -> 0 XUVs
C:0 T:9 1 XUs and 3 XVs -> 1 XUVs
C:0 T:10 3 XUs and 2 XVs -> 2 XUVs
C:0 T:11 1 XUs and 0 XVs -> 0 XUVs
C:0 T:12 6 XUs and 3 XVs -> 2 XUVs
C:0 T:13 5 XUs and 3 XVs -> 2 XUVs
C:0 T:14 15 XUs and 12 XVs -> 8 XUVs
C:0 T:15 3 XUs and 4 XVs -> 3 XUVs
C:0 T:16 1 XUs and 1 XVs -> 1 XUVs
C:0 T:17 3 XUs and 4 XVs -> 2 XUVs
C:0 T:18 8 XUs and 8 XVs -> 7 XUVs
C:0 T:19 8 XUs and 11 XVs -> 5 XUVs
C:0 T:20 22 XUs and 24 XVs -> 15 XUVs
C:0 T:21 4 XUs and 4 XVs -> 4 XUVs
C:0 T:22 12 XUs and 12 XVs -> 8 XUVs
C:0 T:23 16 XUs and 10 XVs -> 7 XUVs
135 XUVs total
106 collection wire objects
135 potential space points
Neighbour search...
283 tests to find 146 neighbours
Iterating with no regularization...
Begin: 370844
0 320544
1 320439
Now with regularization...
Begin: 233108
0 233094
Begin processing the 49th record. run: 249 subRun: 0 event: 40499 at 02-Aug-2025 18:20:37 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12 XUs and 17 XVs -> 9 XUVs
C:0 T:1 27 XUs and 26 XVs -> 16 XUVs
C:0 T:2 15 XUs and 15 XVs -> 11 XUVs
C:0 T:3 26 XUs and 29 XVs -> 19 XUVs
C:0 T:4 12 XUs and 10 XVs -> 7 XUVs
C:0 T:5 6 XUs and 6 XVs -> 2 XUVs
C:0 T:6 13 XUs and 11 XVs -> 7 XUVs
C:0 T:7 1 XUs and 2 XVs -> 1 XUVs
C:0 T:8 3 XUs and 5 XVs -> 2 XUVs
C:0 T:9 0 XUs and 0 XVs -> 0 XUVs
C:0 T:10 5 XUs and 2 XVs -> 2 XUVs
C:0 T:11 7 XUs and 6 XVs -> 3 XUVs
C:0 T:12 1 XUs and 2 XVs -> 0 XUVs
C:0 T:13 2 XUs and 2 XVs -> 1 XUVs
C:0 T:14 3 XUs and 2 XVs -> 0 XUVs
C:0 T:15 11 XUs and 9 XVs -> 6 XUVs
C:0 T:16 1 XUs and 0 XVs -> 0 XUVs
C:0 T:17 4 XUs and 2 XVs -> 1 XUVs
C:0 T:18 3 XUs and 2 XVs -> 2 XUVs
C:0 T:19 6 XUs and 7 XVs -> 2 XUVs
C:0 T:20 18 XUs and 16 XVs -> 11 XUVs
C:0 T:21 9 XUs and 8 XVs -> 1 XUVs
C:0 T:22 15 XUs and 14 XVs -> 8 XUVs
C:0 T:23 13 XUs and 13 XVs -> 7 XUVs
118 XUVs total
91 collection wire objects
118 potential space points
Neighbour search...
248 tests to find 128 neighbours
Iterating with no regularization...
Begin: 308786
0 269690
1 269324
2 269315
Now with regularization...
Begin: 193592
0 193586
Begin processing the 50th record. run: 249 subRun: 0 event: 40500 at 02-Aug-2025 18:20:38 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 41 XUs and 46 XVs -> 27 XUVs
C:0 T:1 14 XUs and 15 XVs -> 10 XUVs
C:0 T:2 14 XUs and 12 XVs -> 8 XUVs
C:0 T:3 9 XUs and 6 XVs -> 3 XUVs
C:0 T:4 4 XUs and 4 XVs -> 2 XUVs
C:0 T:5 4 XUs and 2 XVs -> 2 XUVs
C:0 T:6 12 XUs and 12 XVs -> 8 XUVs
C:0 T:7 6 XUs and 5 XVs -> 4 XUVs
C:0 T:8 10 XUs and 8 XVs -> 5 XUVs
C:0 T:9 11 XUs and 12 XVs -> 9 XUVs
C:0 T:10 8 XUs and 7 XVs -> 6 XUVs
C:0 T:11 18 XUs and 18 XVs -> 9 XUVs
C:0 T:12 5 XUs and 4 XVs -> 4 XUVs
C:0 T:13 6 XUs and 5 XVs -> 2 XUVs
C:0 T:14 8 XUs and 9 XVs -> 4 XUVs
C:0 T:15 1 XUs and 4 XVs -> 1 XUVs
C:0 T:16 15 XUs and 17 XVs -> 11 XUVs
C:0 T:17 7 XUs and 10 XVs -> 3 XUVs
C:0 T:18 3 XUs and 3 XVs -> 2 XUVs
C:0 T:19 13 XUs and 10 XVs -> 4 XUVs
C:0 T:20 11 XUs and 10 XVs -> 5 XUVs
C:0 T:21 15 XUs and 11 XVs -> 7 XUVs
C:0 T:22 22 XUs and 19 XVs -> 11 XUVs
C:0 T:23 7 XUs and 9 XVs -> 5 XUVs
152 XUVs total
114 collection wire objects
152 potential space points
Neighbour search...
340 tests to find 188 neighbours
Iterating with no regularization...
Begin: 375305
0 326798
1 326061
2 326045
Now with regularization...
Begin: 223780
0 223772
02-Aug-2025 18:20:40 CEST  Closed output file "prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T105335Z_gen_000810_supernova_g4_detsim_reco.root"
02-Aug-2025 18:20:40 CEST  Closed input file "prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T105335Z_gen_000810_supernova_g4_detsim.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0378852     0.0409936     0.0536002     0.0406948    0.00260248       50     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000297975   0.000448191   0.000779978   0.000448701   8.43226e-05      50     
reco:gaushit:GausHitFinder                     0.0284517     0.0303182     0.0395553     0.0300031    0.00160612       50     
reco:spsolve:SpacePointSolver                 0.00242517    0.00296291    0.00525499    0.00294324    0.000446587      50     
reco:hitfd:DisambigFromSpacePoints            0.000768021   0.000905308   0.00214823    0.000866848   0.000199813      50     
reco:rns:RandomNumberSaver                    1.8587e-05    2.30603e-05   0.000126239   2.0606e-05    1.4819e-05       50     
[art]:TriggerResults:TriggerResultInserter     7.842e-06    9.20554e-06   2.9505e-05     8.548e-06    3.16049e-06      50     
end_path:out1:RootOutput                       4.047e-06    4.8663e-06    1.4291e-05     4.632e-06    1.39943e-06      50     
end_path:out1:RootOutput(write)               0.00535489    0.00609975    0.00891012    0.00604982    0.000575968      50     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1773.74 MB
  Peak resident set size usage (VmHWM): 889.688 MB
====================================================================================================
Art has completed and will exit with status 0.
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T105335Z_gen_000810.root 109412668
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T105335Z_gen_000810_supernova_g4.root 1028682345
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T105335Z_gen_000810_supernova_g4_detsim.root 2951778049
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T105335Z_gen_000810_supernova_g4_detsim_hist.root 12750
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T105335Z_gen_000810_supernova_g4_detsim_reco.root 2956729936
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T105335Z_gen_000810_supernova_g4_detsim_reco.root.ext.json 1574
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T105335Z_gen_000810_supernova_g4_detsim_reco.root.json 1817
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T105335Z_gen_000810_supernova_g4_detsim_reco.root.temp.json 1817