justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 272912.0@dunegpschedd02.fnal.gov

Jobsub ID272912.0@dunegpschedd02.fnal.gov
Workflow ID11944
Stage ID3
User nameusher@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit28800 (8 hours)
Submitted time2026-01-15 03:13:21
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2026-01-15 03:28:43
From worker nodeHostnameheplnc159.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-15 03:17:52
Input filesusertests:prod_monte-carlo-011944-000002_gen_g4.root
JobscriptExit code0
Real time9m (581s)
CPU time9m (557s = 95%)
Max RSS bytes1732788224 (1652 MiB)
Outputting started2026-01-15 03:27:33
Output files
Finished2026-01-15 03:28:43
Saved logsjustin-logs:272912.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

aver: saving 2560 traces tagged "gauss"
FrameSaver: q=104919 n=10686 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=110037 n=9204 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 10th record. run: 200000 subRun: 0 event: 10 at 15-Jan-2026 03:23:30 GMT
SimDepoSetSource got 11601 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 11601 depos spanning: [0.166593, 0.663442]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=358316 n=35751 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=372351 n=32384 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 11th record. run: 200000 subRun: 0 event: 11 at 15-Jan-2026 03:23:55 GMT
SimDepoSetSource got 19020 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 19020 depos spanning: [0.091535, 0.112515]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=511894 n=76334 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=530838 n=75209 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 12th record. run: 200000 subRun: 0 event: 12 at 15-Jan-2026 03:24:13 GMT
SimDepoSetSource got 2446 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 2446 depos spanning: [0.451339, 0.456495]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=50579.7 n=9121 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=53490.2 n=8073 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 13th record. run: 200000 subRun: 0 event: 13 at 15-Jan-2026 03:24:26 GMT
SimDepoSetSource got 6506 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 6506 depos spanning: [0.560762, 0.568584]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=174386 n=30089 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=179429 n=29452 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 14th record. run: 200000 subRun: 0 event: 14 at 15-Jan-2026 03:24:37 GMT
SimDepoSetSource got 25023 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 25023 depos spanning: [0.578609, 0.605186]us
Retagger: tagging trace set: wiener with 10240 traces, 10240 summary
Retagger: tagging trace set: gauss with 10240 traces, 0 summary
wclsFrameSaver: saving 10240 traces tagged "gauss"
FrameSaver: q=694889 n=98434 tag=gauss
wclsFrameSaver: saving 10240 traces tagged "wiener"
FrameSaver: q=726761 n=91483 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 15th record. run: 200000 subRun: 0 event: 15 at 15-Jan-2026 03:25:28 GMT
SimDepoSetSource got 8399 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 8399 depos spanning: [0.340534, 0.350252]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=229825 n=36981 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=237482 n=36583 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 16th record. run: 200000 subRun: 0 event: 16 at 15-Jan-2026 03:25:41 GMT
SimDepoSetSource got 3272 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 3272 depos spanning: [0.750622, 0.754847]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=87877 n=15966 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=91923.2 n=15120 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 17th record. run: 200000 subRun: 0 event: 17 at 15-Jan-2026 03:26:02 GMT
SimDepoSetSource got 7475 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 7475 depos spanning: [0.349248, 0.357747]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=210806 n=27814 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=230734 n=21371 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 18th record. run: 200000 subRun: 0 event: 18 at 15-Jan-2026 03:26:16 GMT
SimDepoSetSource got 23993 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 23993 depos spanning: [0.493, 0.515153]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=653465 n=68783 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=678907 n=63222 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 19th record. run: 200000 subRun: 0 event: 19 at 15-Jan-2026 03:26:44 GMT
SimDepoSetSource got 2148 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 2148 depos spanning: [0.660702, 0.670214]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=56480.9 n=9810 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=60712.7 n=8793 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 20th record. run: 200000 subRun: 0 event: 20 at 15-Jan-2026 03:27:07 GMT
SimDepoSetSource got 10305 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 10305 depos spanning: [0.539188, 0.549233]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=287401 n=44237 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=298143 n=43810 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
15-Jan-2026 03:27:32 GMT  Closed output file "prod_monte-carlo-011944-000002_gen_g4_detsim.root"
15-Jan-2026 03:27:32 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/d3/db/prod_monte-carlo-011944-000002_gen_g4.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      11.569        26.3392       65.6255       23.5342       15.2698        20     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000614146   0.00162593     0.0054459    0.00123771    0.00104074       20     
simulate:tpcrawdecoder:WireCellToolkit          11.3401       26.0105       64.9409       23.2874       15.1579        20     
simulate:opdigi:OpDetDigitizerDUNE             0.0901547     0.185346      0.462813       0.14524      0.0977087       20     
simulate:rns:RandomNumberSaver                1.8695e-05    4.91504e-05   0.000336193   3.5226e-05    6.62588e-05      20     
[art]:TriggerResults:TriggerResultInserter    1.0029e-05    2.17228e-05   5.7819e-05    1.89205e-05   1.15718e-05      20     
end_path:out1:RootOutput                       2.705e-06    5.7858e-06     2.655e-05    4.6535e-06    4.94565e-06      20     
end_path:out1:RootOutput(write)                0.0785163     0.141183      0.242059       0.13262      0.0505424       20     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3471.39 MB
  Peak resident set size usage (VmHWM): 1732.79 MB
====================================================================================================
Art has completed and will exit with status 0.
-rw-r--r-- 1 pltdune04 pltdune 86M Jan 15 03:27 prod_monte-carlo-011944-000002_gen_g4_detsim.root
Wrote processed DID: usertests:prod_monte-carlo-011944-000002_gen_g4.root
=== DETSIM job completed successfully ===
justIN time: 2026-02-06 01:33:03 UTC       justIN version: 01.06.00