justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 46540.164@dunegpschedd01.fnal.gov

Jobsub ID46540.164@dunegpschedd01.fnal.gov
Workflow ID2607
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-15 08:53:30
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2025-09-15 11:11:39
From worker nodeHostnamehd79.dice.priv
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-09-15 09:17:31
Input filesvd-protodune:np02vd_raw_run039275_0257_df-s03-d2_dw_0_20250901T232803.hdf5
JobscriptExit code4
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-15 11:11:39
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

125 Tr=500 Nr=10112 Nout=10000 padding:linear 
[11:08:20.518] D [  aux   ] <Resampler:resmp0> resample 1536 traces at call=34 
[11:08:20.519] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 input frame: frame: ident=92868 time=152 tick=500 with 1536 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] 
[11:08:20.519] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 init nticks=10000 tbinmin=0 tbinmax=10000 
[11:08:20.559] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[11:08:22.988] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[11:08:25.450] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 
[11:08:30.549] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 save plane index: 0, Qtot=48088736 Qloss=-7631276, 1972 indices spanning [8251,10222] "wiener" 
[11:08:30.794] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 save plane index: 0, Qtot=46196594 Qloss=-6722929, 1649 indices spanning [10223,11871] "gauss" 
[11:08:31.268] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 save plane index: 1, Qtot=53366161 Qloss=-9041401, 1983 indices spanning [11872,13854] "wiener" 
[11:08:31.508] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 save plane index: 1, Qtot=50373661 Qloss=-8316653, 1626 indices spanning [13855,15480] "gauss" 
[11:08:31.974] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 save plane index: 2, Qtot=52576053 Qloss=-2226536, 3929 indices spanning [15481,19409] "wiener" 
[11:08:32.437] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 save plane index: 2, Qtot=51568321 Qloss=-1219108, 3162 indices spanning [19410,22571] "gauss" 
[11:08:32.438] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 produce 22572 traces: 7884 wiener0, 0 decon_charge0, 6437 gauss0, frame tag: sigproc 
[11:08:32.438] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 output frame: frame: ident=92868 time=152 tick=500 with 22572 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss0":6437 [0] "mp2_roi0":6661 [0] "mp3_roi0":1590 [0] "wiener0":7884 [7884] ] cmm:[ ] 
[11:08:32.483] D [  glue  ] <FrameFanin:nfsp> call=34 input 0: frame: ident=92868 time=152 tick=500 with 22572 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss0":6437 [0] "mp2_roi0":6661 [0] "mp3_roi0":1590 [0] "wiener0":7884 [7884] ] cmm:[ ] input 1: frame: ident=92868 time=152 tick=500 with 18242 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss1":5633 [0] "mp2_roi1":4311 [0] "mp3_roi1":1357 [0] "wiener1":6941 [6941] ] cmm:[ ] input 2: frame: ident=92868 time=152 tick=500 with 20026 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":5759 [0] "mp2_roi2":5791 [0] "mp3_roi2":1372 [0] "wiener2":7104 [7104] ] cmm:[ ] input 3: frame: ident=92868 time=152 tick=500 with 24150 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":7045 [0] "mp2_roi3":6261 [0] "mp3_roi3":1989 [0] "wiener3":8855 [8855] ] cmm:[ ] input 4: frame: ident=92868 time=152 tick=512 with 55853 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss4":14104 [0] "mp2_roi4":17720 [0] "mp3_roi4":5888 [0] "wiener4":18141 [18141] ] cmm:[ ] input 5: frame: ident=92868 time=152 tick=512 with 27219 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":7848 [0] "mp2_roi5":6274 [0] "mp3_roi5":2908 [0] "wiener5":10189 [10189] ] cmm:[ ] input 6: frame: ident=92868 time=152 tick=512 with 35697 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":9852 [0] "mp2_roi6":10822 [0] "mp3_roi6":2986 [0] "wiener6":12037 [12037] ] cmm:[ ] input 7: frame: ident=92868 time=152 tick=512 with 28841 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":8292 [0] "mp2_roi7":8445 [0] "mp3_roi7":2276 [0] "wiener7":9828 [9828] ] cmm:[ ] output: frame: ident=92868 time=152 tick=500 with 232600 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":6437 [0] "gauss1":5633 [0] "gauss2":5759 [0] "gauss3":7045 [0] "gauss4":14104 [0] "gauss5":7848 [0] "gauss6":9852 [0] "gauss7":8292 [0] "wiener0":7884 [7884] "wiener1":6941 [6941] "wiener2":7104 [7104] "wiener3":8855 [8855] "wiener4":18141 [18141] "wiener5":10189 [10189] "wiener6":12037 [12037] "wiener7":9828 [9828] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 80979 traces, 80979 summary
Retagger: tagging trace set: gauss with 64970 traces, 0 summary
[11:08:32.509] D [  glue  ] sink frame: #92868 @0.000152 with 232600 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[11:08:32.509] D [  glue  ] <FrameFanout:nfsp> call=35: see EOS 
[11:08:32.510] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=17 
[11:08:32.510] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=35 anode=7 
[11:08:32.510] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=17 
[11:08:32.510] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=35 anode=6 
[11:08:32.510] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=17 
[11:08:32.510] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=35 anode=5 
[11:08:32.510] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=17 
[11:08:32.510] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=35 anode=4 
[11:08:32.510] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=17 
[11:08:32.510] D [  aux   ] <Resampler:resmp3> EOS at call=35 
[11:08:32.510] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=35 anode=3 
[11:08:32.510] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=17 
[11:08:32.511] D [  aux   ] <Resampler:resmp2> EOS at call=35 
[11:08:32.511] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=35 anode=2 
[11:08:32.511] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=17 
[11:08:32.511] D [  aux   ] <Resampler:resmp1> EOS at call=35 
[11:08:32.511] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=35 anode=1 
[11:08:32.511] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=17 
[11:08:32.511] D [  aux   ] <Resampler:resmp0> EOS at call=35 
[11:08:32.511] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=35 anode=0 
[11:08:32.511] D [  glue  ] <FrameFanin:nfsp> EOS at call=35 with 8 
[11:08:32.511] D [  glue  ] frame sink sees EOS
[11:08:32.511] D [ pgraph ] <Pgrapher:> graph execution complete 
[11:08:32.511] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 18.25 sec
[11:08:32.511] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.54 sec
[11:08:32.511] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.49 sec
[11:08:32.511] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.72 sec
[11:08:32.511] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.92 sec
[11:08:32.511] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.78 sec
[11:08:32.511] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.26 sec
[11:08:32.511] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.19 sec
[11:08:32.511] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.27 sec
[11:08:32.511] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.25 sec
[11:08:32.511] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.24 sec
[11:08:32.511] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.23 sec
[11:08:32.511] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[11:08:32.512] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[11:08:32.512] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[11:08:32.512] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[11:08:32.512] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:32.512] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:32.512] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:32.512] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:32.512] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[11:08:32.512] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:32.512] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:32.512] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[11:08:32.512] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[11:08:32.512] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[11:08:32.512] I [ timer  ] Timer: Total node execution : 102.21999962627888 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 64970 traces tagged "gauss"
FrameSaver: q=1.13601e+07 n=1425874 tag=gauss
wclsFrameSaver: saving 80979 traces tagged "wiener"
FrameSaver: q=1.23377e+07 n=1364689 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 697 XUs and 991 XVs -> 23 XUVs
C:0 T:1 914 XUs and 861 XVs -> 36 XUVs
C:0 T:2 810 XUs and 804 XVs -> 104 XUVs
C:0 T:3 1431 XUs and 1008 XVs -> 41 XUVs
C:0 T:4 1301 XUs and 1339 XVs -> 63 XUVs
C:0 T:5 724 XUs and 652 XVs -> 23 XUVs
C:0 T:6 2957 XUs and 2726 XVs -> 163 XUVs
C:0 T:7 434 XUs and 416 XVs -> 16 XUVs
C:0 T:8 25706 XUs and 43626 XVs -> 8997 XUVs
C:0 T:9 23434 XUs and 42418 XVs -> 6114 XUVs
C:0 T:10 19102 XUs and 36744 XVs -> 7581 XUVs
C:0 T:11 6537 XUs and 9976 XVs -> 3107 XUVs
C:0 T:12 17638 XUs and 22169 XVs -> 3887 XUVs
C:0 T:13 1940 XUs and 2699 XVs -> 242 XUVs
C:0 T:14 1514 XUs and 2475 XVs -> 117 XUVs
C:0 T:15 2460 XUs and 2799 XVs -> 160 XUVs
30674 XUVs total
2765 collection wire objects
30674 potential space points
Neighbour search...
5529910 tests to find 2240816 neighbours
Iterating with no regularization...
Begin: 5.83094e+08
0 5.03625e+08
1 4.98557e+08
2 4.97494e+08
3 4.97069e+08
Now with regularization...
Begin: 4.86251e+08
0 4.85842e+08
../justin-jobscript: line 158:  1145 Bus error               (core dumped) LD_PRELOAD=$XROOTD_LIB/libXrdPosixPreload.so lar -c ${FCL1} -n ${nevents} ${extra_line} ${pfn}
../justin-jobscript: line 160: date: command not found
Error in reco1
../justin-jobscript: line 164: cat: command not found
justIN time: 2025-09-19 08:42:12 UTC       justIN version: 01.05.00