justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 281318.35@dunegpschedd01.fnal.gov

Jobsub ID281318.35@dunegpschedd01.fnal.gov
Workflow ID12080
Stage ID3
User nameusher@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit28800 (8 hours)
Submitted time2026-01-18 19:24:11
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2026-01-21 16:07:48
From worker nodeHostnameheplnc157.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statestalled
Started2026-01-18 21:13:35
Input filesusertests:mpvmpr_monte-carlo-012080-001197_gen_g4.root
Outputting started2026-01-18 23:04:12
Output files
Finished2026-01-21 16:54:39
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 40th record. run: 211197 subRun: 0 event: 40 at 18-Jan-2026 22:42:37 GMT
SimDepoSetSource got 63955 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 63955 depos spanning: [0.23584, 9.21237]us
Retagger: tagging trace set: wiener with 17920 traces, 17920 summary
Retagger: tagging trace set: gauss with 17920 traces, 0 summary
wclsFrameSaver: saving 17920 traces tagged "gauss"
FrameSaver: q=1.6823e+06 n=129338 tag=gauss
wclsFrameSaver: saving 17920 traces tagged "wiener"
FrameSaver: q=1.90839e+06 n=119961 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 41st record. run: 211197 subRun: 0 event: 41 at 18-Jan-2026 22:44:13 GMT
SimDepoSetSource got 111007 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 111007 depos spanning: [4.09477, 13.5599]us
Retagger: tagging trace set: wiener with 25600 traces, 25600 summary
Retagger: tagging trace set: gauss with 25600 traces, 0 summary
wclsFrameSaver: saving 25600 traces tagged "gauss"
FrameSaver: q=3.00201e+06 n=332206 tag=gauss
wclsFrameSaver: saving 25600 traces tagged "wiener"
FrameSaver: q=3.22857e+06 n=303398 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 42nd record. run: 211197 subRun: 0 event: 42 at 18-Jan-2026 22:46:53 GMT
SimDepoSetSource got 61341 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 61341 depos spanning: [0.085457, 8.89873]us
Retagger: tagging trace set: wiener with 7680 traces, 7680 summary
Retagger: tagging trace set: gauss with 7680 traces, 0 summary
wclsFrameSaver: saving 7680 traces tagged "gauss"
FrameSaver: q=1.38562e+06 n=143137 tag=gauss
wclsFrameSaver: saving 7680 traces tagged "wiener"
FrameSaver: q=1.49981e+06 n=132288 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 43rd record. run: 211197 subRun: 0 event: 43 at 18-Jan-2026 22:47:56 GMT
SimDepoSetSource got 98888 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 98888 depos spanning: [0.384526, 7.11818]us
Retagger: tagging trace set: wiener with 15360 traces, 15360 summary
Retagger: tagging trace set: gauss with 15360 traces, 0 summary
wclsFrameSaver: saving 15360 traces tagged "gauss"
FrameSaver: q=2.55608e+06 n=268140 tag=gauss
wclsFrameSaver: saving 15360 traces tagged "wiener"
FrameSaver: q=2.76021e+06 n=246410 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 44th record. run: 211197 subRun: 0 event: 44 at 18-Jan-2026 22:49:47 GMT
SimDepoSetSource got 66555 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 66555 depos spanning: [5.85897, 9.24181]us
Retagger: tagging trace set: wiener with 20480 traces, 20480 summary
Retagger: tagging trace set: gauss with 20480 traces, 0 summary
wclsFrameSaver: saving 20480 traces tagged "gauss"
FrameSaver: q=1.94572e+06 n=249861 tag=gauss
wclsFrameSaver: saving 20480 traces tagged "wiener"
FrameSaver: q=2.07749e+06 n=215099 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 45th record. run: 211197 subRun: 0 event: 45 at 18-Jan-2026 22:51:51 GMT
SimDepoSetSource got 217214 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 217214 depos spanning: [0.620363, 8.81545]us
Retagger: tagging trace set: wiener with 15360 traces, 15360 summary
Retagger: tagging trace set: gauss with 15360 traces, 0 summary
wclsFrameSaver: saving 15360 traces tagged "gauss"
FrameSaver: q=4.93277e+06 n=428923 tag=gauss
wclsFrameSaver: saving 15360 traces tagged "wiener"
FrameSaver: q=5.42195e+06 n=388557 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 46th record. run: 211197 subRun: 0 event: 46 at 18-Jan-2026 22:54:10 GMT
SimDepoSetSource got 134878 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 134878 depos spanning: [2.59521, 5.9993]us
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=2.7119e+06 n=221365 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=2.94738e+06 n=202126 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 47th record. run: 211197 subRun: 0 event: 47 at 18-Jan-2026 22:55:21 GMT
SimDepoSetSource got 217055 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 217055 depos spanning: [2.10725, 9.62571]us
Retagger: tagging trace set: wiener with 20480 traces, 20480 summary
Retagger: tagging trace set: gauss with 20480 traces, 0 summary
wclsFrameSaver: saving 20480 traces tagged "gauss"
FrameSaver: q=5.33595e+06 n=439313 tag=gauss
wclsFrameSaver: saving 20480 traces tagged "wiener"
FrameSaver: q=5.77579e+06 n=392579 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 48th record. run: 211197 subRun: 0 event: 48 at 18-Jan-2026 22:58:16 GMT
SimDepoSetSource got 84315 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 84315 depos spanning: [1.75708, 9.58266]us
Retagger: tagging trace set: wiener with 20480 traces, 20480 summary
Retagger: tagging trace set: gauss with 20480 traces, 0 summary
wclsFrameSaver: saving 20480 traces tagged "gauss"
FrameSaver: q=2.3555e+06 n=247034 tag=gauss
wclsFrameSaver: saving 20480 traces tagged "wiener"
FrameSaver: q=2.52831e+06 n=208838 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 49th record. run: 211197 subRun: 0 event: 49 at 18-Jan-2026 23:00:29 GMT
SimDepoSetSource got 113502 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 113502 depos spanning: [3.11011, 4.56487]us
Retagger: tagging trace set: wiener with 17920 traces, 17920 summary
Retagger: tagging trace set: gauss with 17920 traces, 0 summary
wclsFrameSaver: saving 17920 traces tagged "gauss"
FrameSaver: q=1.97494e+06 n=193909 tag=gauss
wclsFrameSaver: saving 17920 traces tagged "wiener"
FrameSaver: q=2.18224e+06 n=167822 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
Begin processing the 50th record. run: 211197 subRun: 0 event: 50 at 18-Jan-2026 23:02:42 GMT
SimDepoSetSource got 101043 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 101043 depos spanning: [0.466152, 1.64535]us
Retagger: tagging trace set: wiener with 10240 traces, 10240 summary
Retagger: tagging trace set: gauss with 10240 traces, 0 summary
wclsFrameSaver: saving 10240 traces tagged "gauss"
FrameSaver: q=2.18876e+06 n=143355 tag=gauss
wclsFrameSaver: saving 10240 traces tagged "wiener"
FrameSaver: q=2.31058e+06 n=135138 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
18-Jan-2026 23:04:11 GMT  Closed output file "mpvmpr_monte-carlo-012080-001197_gen_g4_detsim.root"
18-Jan-2026 23:04:11 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/98/b3/mpvmpr_monte-carlo-012080-001197_gen_g4.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      46.1302       130.519       287.452       122.11        55.9359        50     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000732536   0.00613796     0.0456712    0.00441216    0.00705236       50     
simulate:tpcrawdecoder:WireCellToolkit          45.6378       128.349       282.914       120.182       54.9391        50     
simulate:opdigi:OpDetDigitizerDUNE             0.207809       0.96612       2.30425      0.891109      0.532856        50     
simulate:rns:RandomNumberSaver                2.7171e-05    5.10696e-05   0.000245351   4.44035e-05   2.98619e-05      50     
[art]:TriggerResults:TriggerResultInserter    1.3124e-05    2.17557e-05   6.7717e-05    1.9692e-05    9.62849e-06      50     
end_path:out1:RootOutput                       3.125e-06    6.76138e-06   2.2943e-05    6.5125e-06    2.62511e-06      50     
end_path:out1:RootOutput(write)                0.211333       1.1972        3.64401      0.979778      0.809735        50     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4439.57 MB
  Peak resident set size usage (VmHWM): 2727.27 MB
====================================================================================================
Art has completed and will exit with status 0.
-rw-r--r-- 1 pltdune04 pltdune 2.6G Jan 18 23:04 mpvmpr_monte-carlo-012080-001197_gen_g4_detsim.root
Wrote processed DID: usertests:mpvmpr_monte-carlo-012080-001197_gen_g4.root
=== DETSIM job completed successfully ===
justIN time: 2026-02-04 07:15:29 UTC       justIN version: 01.06.00