justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297770.0@dunegpschedd02.fnal.gov

Jobsub ID297770.0@dunegpschedd02.fnal.gov
Workflow ID12686
Stage ID3
User nameusher@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2026-02-03 15:58:04
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel
Last heartbeat2026-02-03 16:30:27
From worker nodeHostnamewn-choc-015.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 15:58:24
Input filesusertests:prod_monte-carlo-012686-000012_gen_g4.root
JobscriptExit code0
Real time30m (1859s)
CPU time30m (1828s = 98%)
Max RSS bytes1677049856 (1599 MiB)
Outputting started2026-02-03 16:29:24
Output files
Finished2026-02-03 16:30:27
Saved logsjustin-logs:297770.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

lsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=101919 n=19480 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=105574 n=19271 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 40th record. run: 210012 subRun: 0 event: 40 at 03-Feb-2026 17:24:01 CET
SimDepoSetSource got 3190 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 3190 depos spanning: [0.921075, 0.924537]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=97629.1 n=8532 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=107511 n=7001 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 41st record. run: 210012 subRun: 0 event: 41 at 03-Feb-2026 17:24:17 CET
SimDepoSetSource got 4086 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 4086 depos spanning: [0.497074, 0.510206]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=118812 n=16620 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=123164 n=16085 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 42nd record. run: 210012 subRun: 0 event: 42 at 03-Feb-2026 17:24:48 CET
SimDepoSetSource got 3716 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 3716 depos spanning: [0.975543, 1.02966]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=108549 n=11912 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=118356 n=9568 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 43rd record. run: 210012 subRun: 0 event: 43 at 03-Feb-2026 17:25:05 CET
SimDepoSetSource got 25100 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 25100 depos spanning: [0.516265, 3.88654]us
Retagger: tagging trace set: wiener with 10240 traces, 10240 summary
Retagger: tagging trace set: gauss with 10240 traces, 0 summary
wclsFrameSaver: saving 10240 traces tagged "gauss"
FrameSaver: q=760344 n=111523 tag=gauss
wclsFrameSaver: saving 10240 traces tagged "wiener"
FrameSaver: q=826874 n=88746 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 44th record. run: 210012 subRun: 0 event: 44 at 03-Feb-2026 17:26:16 CET
SimDepoSetSource got 2354 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 2354 depos spanning: [0.535691, 0.542221]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=61905 n=9586 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=65693.2 n=8118 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 45th record. run: 210012 subRun: 0 event: 45 at 03-Feb-2026 17:26:32 CET
SimDepoSetSource got 10644 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 10644 depos spanning: [0.376088, 0.388392]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=298710 n=44892 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=305682 n=43911 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 46th record. run: 210012 subRun: 0 event: 46 at 03-Feb-2026 17:27:07 CET
SimDepoSetSource got 3704 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 3704 depos spanning: [0.938967, 0.943175]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=91489 n=14759 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=96227.6 n=13352 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 47th record. run: 210012 subRun: 0 event: 47 at 03-Feb-2026 17:27:23 CET
SimDepoSetSource got 5525 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 5525 depos spanning: [0.149248, 1.73437]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=173547 n=16289 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=189654 n=13459 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 48th record. run: 210012 subRun: 0 event: 48 at 03-Feb-2026 17:27:54 CET
SimDepoSetSource got 10563 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 10563 depos spanning: [0.056713, 0.067608]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=290690 n=43602 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=304495 n=37573 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 49th record. run: 210012 subRun: 0 event: 49 at 03-Feb-2026 17:28:30 CET
SimDepoSetSource got 6314 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 6314 depos spanning: [0.844693, 0.851797]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=174957 n=29848 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=184621 n=24993 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 50th record. run: 210012 subRun: 0 event: 50 at 03-Feb-2026 17:29:04 CET
SimDepoSetSource got 6354 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 6354 depos spanning: [0.0343465, 0.0416928]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=173708 n=29910 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=178171 n=29195 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
03-Feb-2026 17:29:23 CET  Closed output file "prod_monte-carlo-012686-000012_gen_g4_detsim.root"
03-Feb-2026 17:29:23 CET  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/66/d6/prod_monte-carlo-012686-000012_gen_g4.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      14.7576       35.8994       83.6247       33.7679       18.4492        50     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000335553   0.00278759     0.0123475    0.00221196     0.0030185       50     
simulate:tpcrawdecoder:WireCellToolkit          14.6582       35.571        83.1047       33.4441       18.3617        50     
simulate:opdigi:OpDetDigitizerDUNE             0.0506329     0.195788      0.349814      0.179236      0.0761581       50     
simulate:rns:RandomNumberSaver                2.3044e-05    3.89673e-05   0.000278193   3.5299e-05    3.44283e-05      50     
[art]:TriggerResults:TriggerResultInserter    1.3885e-05    1.69752e-05   4.8632e-05    1.59515e-05   5.18917e-06      50     
end_path:out1:RootOutput                       3.276e-06    7.77468e-06   7.3401e-05    6.5195e-06    9.70922e-06      50     
end_path:out1:RootOutput(write)                0.0460138     0.129234      0.250275      0.123501      0.054885        50     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3373.84 MB
  Peak resident set size usage (VmHWM): 1677.05 MB
====================================================================================================
Art has completed and will exit with status 0.
-rw-r--r-- 1 dune090 dune 214M Feb  3 17:29 prod_monte-carlo-012686-000012_gen_g4_detsim.root
Wrote processed DID: usertests:prod_monte-carlo-012686-000012_gen_g4.root
=== DETSIM job completed successfully ===
justIN time: 2026-02-04 04:26:50 UTC       justIN version: 01.06.00