justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14031.121@dunegpschedd02.fnal.gov

Jobsub ID14031.121@dunegpschedd02.fnal.gov
Workflow ID251
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-02 19:13:39
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk_multicore
Last heartbeat2025-08-03 00:10:09
From worker nodeHostnamewn-sate-024.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-02 19:54:12
Input filesmonte-carlo-000251-009106
JobscriptExit code0
Real time4h (15232s)
CPU time4h (15146s = 99%)
Max RSS bytes1923076096 (1833 MiB)
Outputting started2025-08-03 00:08:05
Output filesfardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T195416Z_gen_009106_supernova_g4_halfActiveVol_detsim_reco.root
Finished2025-08-03 00:10:09
Saved logsjustin-logs:14031.121-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

T:15 17 XUs and 19 XVs -> 12 XUVs
C:0 T:17 18 XUs and 18 XVs -> 13 XUVs
C:0 T:19 1 XUs and 2 XVs -> 1 XUVs
C:0 T:21 26 XUs and 25 XVs -> 17 XUVs
C:0 T:23 2 XUs and 4 XVs -> 2 XUVs
107 XUVs total
73 collection wire objects
107 potential space points
Neighbour search...
241 tests to find 134 neighbours
Iterating with no regularization...
Begin: 277210
0 227997
1 227683
2 227677
Now with regularization...
Begin: 144905
0 144897
Begin processing the 44th record. run: 251 subRun: 0 event: 455294 at 03-Aug-2025 02:07:57 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 20 XUs and 23 XVs -> 11 XUVs
C:0 T:3 13 XUs and 12 XVs -> 8 XUVs
C:0 T:5 0 XUs and 1 XVs -> 0 XUVs
C:0 T:7 8 XUs and 7 XVs -> 4 XUVs
C:0 T:9 8 XUs and 8 XVs -> 4 XUVs
C:0 T:11 8 XUs and 7 XVs -> 5 XUVs
C:0 T:13 14 XUs and 18 XVs -> 12 XUVs
C:0 T:15 10 XUs and 9 XVs -> 5 XUVs
C:0 T:17 7 XUs and 6 XVs -> 3 XUVs
C:0 T:19 21 XUs and 20 XVs -> 9 XUVs
C:0 T:21 25 XUs and 30 XVs -> 17 XUVs
C:0 T:23 25 XUs and 27 XVs -> 17 XUVs
95 XUVs total
73 collection wire objects
95 potential space points
Neighbour search...
183 tests to find 86 neighbours
Iterating with no regularization...
Begin: 297137
0 274993
1 274963
Now with regularization...
Begin: 207835
0 207831
Begin processing the 45th record. run: 251 subRun: 0 event: 455295 at 03-Aug-2025 02:07:58 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 10 XUs and 13 XVs -> 7 XUVs
C:0 T:3 20 XUs and 19 XVs -> 10 XUVs
C:0 T:5 10 XUs and 11 XVs -> 7 XUVs
C:0 T:7 12 XUs and 12 XVs -> 9 XUVs
C:0 T:9 21 XUs and 21 XVs -> 17 XUVs
C:0 T:11 7 XUs and 8 XVs -> 4 XUVs
C:0 T:13 18 XUs and 15 XVs -> 11 XUVs
C:0 T:15 10 XUs and 9 XVs -> 5 XUVs
C:0 T:17 2 XUs and 2 XVs -> 1 XUVs
C:0 T:19 11 XUs and 11 XVs -> 7 XUVs
C:0 T:21 15 XUs and 15 XVs -> 11 XUVs
C:0 T:23 30 XUs and 26 XVs -> 15 XUVs
104 XUVs total
77 collection wire objects
104 potential space points
Neighbour search...
202 tests to find 98 neighbours
Iterating with no regularization...
Begin: 305468
0 275939
1 275922
Now with regularization...
Begin: 210346
0 210345
Begin processing the 46th record. run: 251 subRun: 0 event: 455296 at 03-Aug-2025 02:07:58 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 23 XUs and 21 XVs -> 12 XUVs
C:0 T:3 16 XUs and 18 XVs -> 7 XUVs
C:0 T:5 20 XUs and 13 XVs -> 11 XUVs
C:0 T:7 7 XUs and 8 XVs -> 5 XUVs
C:0 T:9 5 XUs and 4 XVs -> 3 XUVs
C:0 T:11 9 XUs and 8 XVs -> 5 XUVs
C:0 T:13 7 XUs and 10 XVs -> 5 XUVs
C:0 T:15 3 XUs and 5 XVs -> 3 XUVs
C:0 T:17 4 XUs and 3 XVs -> 1 XUVs
C:0 T:19 5 XUs and 5 XVs -> 3 XUVs
C:0 T:21 22 XUs and 23 XVs -> 14 XUVs
C:0 T:23 21 XUs and 20 XVs -> 12 XUVs
81 XUVs total
63 collection wire objects
81 potential space points
Neighbour search...
169 tests to find 88 neighbours
Iterating with no regularization...
Begin: 255739
0 239729
1 239710
Now with regularization...
Begin: 184270
0 184268
Begin processing the 47th record. run: 251 subRun: 0 event: 455297 at 03-Aug-2025 02:07:59 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 6 XUs and 11 XVs -> 6 XUVs
C:0 T:3 16 XUs and 16 XVs -> 9 XUVs
C:0 T:5 6 XUs and 8 XVs -> 4 XUVs
C:0 T:7 6 XUs and 5 XVs -> 3 XUVs
C:0 T:9 13 XUs and 17 XVs -> 5 XUVs
C:0 T:11 16 XUs and 16 XVs -> 12 XUVs
C:0 T:13 8 XUs and 10 XVs -> 6 XUVs
C:0 T:15 12 XUs and 11 XVs -> 8 XUVs
C:0 T:17 11 XUs and 13 XVs -> 7 XUVs
C:0 T:19 16 XUs and 18 XVs -> 9 XUVs
C:0 T:21 13 XUs and 14 XVs -> 8 XUVs
C:0 T:23 21 XUs and 22 XVs -> 11 XUVs
88 XUVs total
66 collection wire objects
88 potential space points
Neighbour search...
186 tests to find 96 neighbours
Iterating with no regularization...
Begin: 253435
0 211063
1 210895
Now with regularization...
Begin: 141426
0 141397
Begin processing the 48th record. run: 251 subRun: 0 event: 455298 at 03-Aug-2025 02:07:59 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 19 XUs and 22 XVs -> 13 XUVs
C:0 T:3 21 XUs and 23 XVs -> 12 XUVs
C:0 T:5 7 XUs and 6 XVs -> 4 XUVs
C:0 T:7 19 XUs and 20 XVs -> 12 XUVs
C:0 T:9 12 XUs and 10 XVs -> 6 XUVs
C:0 T:11 9 XUs and 8 XVs -> 7 XUVs
C:0 T:13 17 XUs and 19 XVs -> 12 XUVs
C:0 T:15 7 XUs and 7 XVs -> 5 XUVs
C:0 T:17 10 XUs and 8 XVs -> 7 XUVs
C:0 T:19 11 XUs and 11 XVs -> 10 XUVs
C:0 T:21 9 XUs and 12 XVs -> 6 XUVs
C:0 T:23 23 XUs and 23 XVs -> 13 XUVs
107 XUVs total
76 collection wire objects
107 potential space points
Neighbour search...
249 tests to find 136 neighbours
Iterating with no regularization...
Begin: 289786
0 265934
1 265785
Now with regularization...
Begin: 187127
0 187117
Begin processing the 49th record. run: 251 subRun: 0 event: 455299 at 03-Aug-2025 02:08:00 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 11 XUs and 10 XVs -> 7 XUVs
C:0 T:3 25 XUs and 23 XVs -> 15 XUVs
C:0 T:5 10 XUs and 7 XVs -> 6 XUVs
C:0 T:7 7 XUs and 8 XVs -> 5 XUVs
C:0 T:9 1 XUs and 1 XVs -> 0 XUVs
C:0 T:11 10 XUs and 11 XVs -> 8 XUVs
C:0 T:13 3 XUs and 3 XVs -> 2 XUVs
C:0 T:15 7 XUs and 7 XVs -> 5 XUVs
C:0 T:17 12 XUs and 11 XVs -> 7 XUVs
C:0 T:19 8 XUs and 11 XVs -> 6 XUVs
C:0 T:21 8 XUs and 14 XVs -> 6 XUVs
C:0 T:23 16 XUs and 12 XVs -> 8 XUVs
75 XUVs total
57 collection wire objects
75 potential space points
Neighbour search...
137 tests to find 62 neighbours
Iterating with no regularization...
Begin: 177088
0 158584
1 158583
Now with regularization...
Begin: 111032
0 111030
Begin processing the 50th record. run: 251 subRun: 0 event: 455300 at 03-Aug-2025 02:08:00 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 22 XUs and 21 XVs -> 12 XUVs
C:0 T:3 24 XUs and 28 XVs -> 14 XUVs
C:0 T:5 13 XUs and 12 XVs -> 9 XUVs
C:0 T:7 6 XUs and 8 XVs -> 3 XUVs
C:0 T:9 12 XUs and 13 XVs -> 8 XUVs
C:0 T:11 11 XUs and 10 XVs -> 7 XUVs
C:0 T:13 10 XUs and 9 XVs -> 5 XUVs
C:0 T:15 8 XUs and 9 XVs -> 5 XUVs
C:0 T:17 27 XUs and 25 XVs -> 17 XUVs
C:0 T:19 11 XUs and 12 XVs -> 6 XUVs
C:0 T:21 18 XUs and 23 XVs -> 15 XUVs
C:0 T:23 15 XUs and 22 XVs -> 10 XUVs
111 XUVs total
80 collection wire objects
111 potential space points
Neighbour search...
243 tests to find 132 neighbours
Iterating with no regularization...
Begin: 358925
0 328034
1 327917
Now with regularization...
Begin: 244369
0 244365
03-Aug-2025 02:08:02 CEST  Closed output file "prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T195416Z_gen_009106_supernova_g4_halfActiveVol_detsim_reco.root"
03-Aug-2025 02:08:02 CEST  Closed input file "prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T195416Z_gen_009106_supernova_g4_halfActiveVol_detsim.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.041117      0.0455371     0.0625677     0.0449538    0.00308395       50     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000319151   0.000485987   0.00109902    0.000463081   0.000129683      50     
reco:gaushit:GausHitFinder                     0.0314136     0.0342995     0.0393224     0.0340218    0.00135566       50     
reco:spsolve:SpacePointSolver                 0.00238702    0.00308425    0.00777653    0.00294689    0.000772469      50     
reco:hitfd:DisambigFromSpacePoints            0.000770751   0.00110557     0.0039489     0.0010227    0.000427426      50     
reco:rns:RandomNumberSaver                    1.7563e-05    3.20283e-05   0.000164981   2.8539e-05    2.11161e-05      50     
[art]:TriggerResults:TriggerResultInserter     9.618e-06    1.48683e-05   3.8292e-05    1.43415e-05   4.31346e-06      50     
end_path:out1:RootOutput                       2.775e-06    5.6506e-06    1.8414e-05     5.245e-06    2.4931e-06       50     
end_path:out1:RootOutput(write)                0.0052543     0.0061554    0.00969404    0.00605555    0.000672632      50     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1773.72 MB
  Peak resident set size usage (VmHWM): 890.647 MB
====================================================================================================
Art has completed and will exit with status 0.
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T195416Z_gen_009106.root 110062179
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T195416Z_gen_009106_supernova_g4_halfActiveVol.root 721498418
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T195416Z_gen_009106_supernova_g4_halfActiveVol_detsim.root 1838678413
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T195416Z_gen_009106_supernova_g4_halfActiveVol_detsim_hist.root 12750
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T195416Z_gen_009106_supernova_g4_halfActiveVol_detsim_reco.root 1842065433
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T195416Z_gen_009106_supernova_g4_halfActiveVol_detsim_reco.root.ext.json 1606
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T195416Z_gen_009106_supernova_g4_halfActiveVol_detsim_reco.root.json 1835
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T195416Z_gen_009106_supernova_g4_halfActiveVol_detsim_reco.root.temp.json 1835
justIN time: 2025-08-04 14:18:07 UTC       justIN version: 01.04.00