justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303346.24@dunegpschedd01.fnal.gov

Jobsub ID303346.24@dunegpschedd01.fnal.gov
Workflow ID12673
Stage ID3
User namedrielsma@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2026-02-03 10:54:44
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce16-multicore
Last heartbeat2026-02-03 11:30:33
From worker nodeHostnametd820.pic.es
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 10:55:57
Input filesusertests:prod_monte-carlo-012673-000009_gen_g4.root
JobscriptExit code0
Real time33m (1997s)
CPU time28m (1728s = 86%)
Max RSS bytes1812578304 (1728 MiB)
Outputting started2026-02-03 11:29:15
Output files
Finished2026-02-03 11:30:33
Saved logsjustin-logs:303346.24-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

rameSaver: q=745824 n=78236 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=770710 n=72346 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 40th record. run: 210009 subRun: 0 event: 40 at 03-Feb-2026 12:20:14 CET
SimDepoSetSource got 21964 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 21964 depos spanning: [0.549327, 0.577563]us
Retagger: tagging trace set: wiener with 10240 traces, 10240 summary
Retagger: tagging trace set: gauss with 10240 traces, 0 summary
wclsFrameSaver: saving 10240 traces tagged "gauss"
FrameSaver: q=688463 n=91385 tag=gauss
wclsFrameSaver: saving 10240 traces tagged "wiener"
FrameSaver: q=760633 n=73326 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 41st record. run: 210009 subRun: 0 event: 41 at 03-Feb-2026 12:21:17 CET
SimDepoSetSource got 8405 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 8405 depos spanning: [0.629351, 0.639279]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=235853 n=43224 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=241428 n=41805 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 42nd record. run: 210009 subRun: 0 event: 42 at 03-Feb-2026 12:21:41 CET
SimDepoSetSource got 21945 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 21945 depos spanning: [0.315263, 0.338301]us
Retagger: tagging trace set: wiener with 7680 traces, 7680 summary
Retagger: tagging trace set: gauss with 7680 traces, 0 summary
wclsFrameSaver: saving 7680 traces tagged "gauss"
FrameSaver: q=630400 n=88666 tag=gauss
wclsFrameSaver: saving 7680 traces tagged "wiener"
FrameSaver: q=695614 n=69010 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 43rd record. run: 210009 subRun: 0 event: 43 at 03-Feb-2026 12:22:39 CET
SimDepoSetSource got 6296 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 6296 depos spanning: [0.907423, 0.914741]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=186975 n=24478 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=196083 n=22110 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 44th record. run: 210009 subRun: 0 event: 44 at 03-Feb-2026 12:22:54 CET
SimDepoSetSource got 10755 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 10755 depos spanning: [0.237273, 0.24911]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=324293 n=44676 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=338773 n=41390 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 45th record. run: 210009 subRun: 0 event: 45 at 03-Feb-2026 12:23:28 CET
SimDepoSetSource got 38166 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 38166 depos spanning: [0.53987, 0.580966]us
Retagger: tagging trace set: wiener with 15360 traces, 15360 summary
Retagger: tagging trace set: gauss with 15360 traces, 0 summary
wclsFrameSaver: saving 15360 traces tagged "gauss"
FrameSaver: q=1.13426e+06 n=176840 tag=gauss
wclsFrameSaver: saving 15360 traces tagged "wiener"
FrameSaver: q=1.18715e+06 n=154825 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 46th record. run: 210009 subRun: 0 event: 46 at 03-Feb-2026 12:25:05 CET
SimDepoSetSource got 5814 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 5814 depos spanning: [0.673558, 0.683967]us
Retagger: tagging trace set: wiener with 2560 traces, 2560 summary
Retagger: tagging trace set: gauss with 2560 traces, 0 summary
wclsFrameSaver: saving 2560 traces tagged "gauss"
FrameSaver: q=153360 n=17878 tag=gauss
wclsFrameSaver: saving 2560 traces tagged "wiener"
FrameSaver: q=158376 n=15446 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 47th record. run: 210009 subRun: 0 event: 47 at 03-Feb-2026 12:25:21 CET
SimDepoSetSource got 21535 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 21535 depos spanning: [0.244766, 2.11832]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=587129 n=81784 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=604736 n=78732 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 48th record. run: 210009 subRun: 0 event: 48 at 03-Feb-2026 12:25:52 CET
SimDepoSetSource got 38123 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 38123 depos spanning: [0.971595, 1.01013]us
Retagger: tagging trace set: wiener with 15360 traces, 15360 summary
Retagger: tagging trace set: gauss with 15360 traces, 0 summary
wclsFrameSaver: saving 15360 traces tagged "gauss"
FrameSaver: q=1.03533e+06 n=144614 tag=gauss
wclsFrameSaver: saving 15360 traces tagged "wiener"
FrameSaver: q=1.12961e+06 n=113554 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 49th record. run: 210009 subRun: 0 event: 49 at 03-Feb-2026 12:27:26 CET
SimDepoSetSource got 10470 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 10470 depos spanning: [0.353022, 0.368228]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=302880 n=51988 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=310872 n=48522 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 50th record. run: 210009 subRun: 0 event: 50 at 03-Feb-2026 12:27:57 CET
SimDepoSetSource got 32981 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 32981 depos spanning: [0.350487, 0.383718]us
Retagger: tagging trace set: wiener with 12800 traces, 12800 summary
Retagger: tagging trace set: gauss with 12800 traces, 0 summary
wclsFrameSaver: saving 12800 traces tagged "gauss"
FrameSaver: q=1.00319e+06 n=125795 tag=gauss
wclsFrameSaver: saving 12800 traces tagged "wiener"
FrameSaver: q=1.14296e+06 n=102536 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
03-Feb-2026 12:29:14 CET  Closed output file "prod_monte-carlo-012673-000009_gen_g4_detsim.root"
03-Feb-2026 12:29:14 CET  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/db/5e/prod_monte-carlo-012673-000009_gen_g4.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      12.7895       35.6649       95.9725       29.8736       21.0895        50     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000415679    0.0509617      0.8237      0.00216818     0.164406        50     
simulate:tpcrawdecoder:WireCellToolkit          12.4109       35.2417       95.0743       29.5238       20.966         50     
simulate:opdigi:OpDetDigitizerDUNE             0.0632251     0.206653      0.483936      0.190721      0.0925258       50     
simulate:rns:RandomNumberSaver                 2.087e-05    3.28263e-05   0.000234699   2.7775e-05    2.94685e-05      50     
[art]:TriggerResults:TriggerResultInserter     9.891e-06    1.57908e-05    6.406e-05     1.365e-05    8.53842e-06      50     
end_path:out1:RootOutput                       2.73e-06     4.33774e-06    1.902e-05     3.84e-06     2.37369e-06      50     
end_path:out1:RootOutput(write)                0.0526105     0.164939      0.330779       0.15733       0.07042        50     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3557.61 MB
  Peak resident set size usage (VmHWM): 1812.58 MB
====================================================================================================
Art has completed and will exit with status 0.
-rw-r--r-- 1 dune001 dune 241M Feb  3 12:29 prod_monte-carlo-012673-000009_gen_g4_detsim.root
Wrote processed DID: usertests:prod_monte-carlo-012673-000009_gen_g4.root
=== DETSIM job completed successfully ===
justIN time: 2026-02-04 04:47:20 UTC       justIN version: 01.06.00