justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 38789.118@dunegpschedd02.fnal.gov

Jobsub ID38789.118@dunegpschedd02.fnal.gov
Workflow ID2523
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-12 06:29:32
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2025-09-12 09:00:31
From worker nodeHostnamehd79.dice.priv
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-09-12 06:57:26
Input filesvd-protodune:np02vd_raw_run039273_0777_df-s01-d3_dw_0_20250831T203341.hdf5
JobscriptExit code4
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-12 09:00:31
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

me tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] 
[08:54:57.137] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 init nticks=10000 tbinmin=0 tbinmax=10000 
[08:54:57.176] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[08:54:59.711] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[08:55:02.291] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 
[08:55:06.428] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 save plane index: 0, Qtot=49199756 Qloss=-6929243, 1557 indices spanning [6133,7689] "wiener" 
[08:55:06.680] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 save plane index: 0, Qtot=47148752 Qloss=-6110498, 1284 indices spanning [7690,8973] "gauss" 
[08:55:07.199] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 save plane index: 1, Qtot=48405192 Qloss=-6129022, 2544 indices spanning [8974,11517] "wiener" 
[08:55:07.453] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 save plane index: 1, Qtot=44879422 Qloss=-4532100, 1900 indices spanning [11518,13417] "gauss" 
[08:55:07.949] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 save plane index: 2, Qtot=45712607 Qloss=-2419791, 3465 indices spanning [13418,16882] "wiener" 
[08:55:08.441] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 save plane index: 2, Qtot=44210164 Qloss=-907131, 2550 indices spanning [16883,19432] "gauss" 
[08:55:08.449] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 produce 19433 traces: 7566 wiener0, 0 decon_charge0, 5734 gauss0, frame tag: sigproc 
[08:55:08.449] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=34 output frame: frame: ident=280061 time=43 tick=500 with 19433 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss0":5734 [0] "mp2_roi0":4602 [0] "mp3_roi0":1531 [0] "wiener0":7566 [7566] ] cmm:[ ] 
[08:55:08.503] D [  glue  ] <FrameFanin:nfsp> call=34 input 0: frame: ident=280061 time=43 tick=500 with 19433 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss0":5734 [0] "mp2_roi0":4602 [0] "mp3_roi0":1531 [0] "wiener0":7566 [7566] ] cmm:[ ] input 1: frame: ident=280061 time=43 tick=500 with 21894 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss1":6097 [0] "mp2_roi1":6843 [0] "mp3_roi1":1437 [0] "wiener1":7517 [7517] ] cmm:[ ] input 2: frame: ident=280061 time=43 tick=500 with 34904 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":9391 [0] "mp2_roi2":10785 [0] "mp3_roi2":2732 [0] "wiener2":11996 [11996] ] cmm:[ ] input 3: frame: ident=280061 time=43 tick=500 with 21008 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":5894 [0] "mp2_roi3":5527 [0] "mp3_roi3":1780 [0] "wiener3":7807 [7807] ] cmm:[ ] input 4: frame: ident=280061 time=43 tick=512 with 51365 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss4":12881 [0] "mp2_roi4":17797 [0] "mp3_roi4":4759 [0] "wiener4":15928 [15928] ] cmm:[ ] input 5: frame: ident=280061 time=43 tick=512 with 43281 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":11375 [0] "mp2_roi5":12385 [0] "mp3_roi5":4484 [0] "wiener5":15037 [15037] ] cmm:[ ] input 6: frame: ident=280061 time=43 tick=512 with 51960 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":13428 [0] "mp2_roi6":16403 [0] "mp3_roi6":4842 [0] "wiener6":17287 [17287] ] cmm:[ ] input 7: frame: ident=280061 time=43 tick=512 with 51289 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":12702 [0] "mp2_roi7":18354 [0] "mp3_roi7":4049 [0] "wiener7":16184 [16184] ] cmm:[ ] output: frame: ident=280061 time=43 tick=500 with 295134 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":5734 [0] "gauss1":6097 [0] "gauss2":9391 [0] "gauss3":5894 [0] "gauss4":12881 [0] "gauss5":11375 [0] "gauss6":13428 [0] "gauss7":12702 [0] "wiener0":7566 [7566] "wiener1":7517 [7517] "wiener2":11996 [11996] "wiener3":7807 [7807] "wiener4":15928 [15928] "wiener5":15037 [15037] "wiener6":17287 [17287] "wiener7":16184 [16184] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 99322 traces, 99322 summary
Retagger: tagging trace set: gauss with 77502 traces, 0 summary
[08:55:08.532] D [  glue  ] sink frame: #280061 @4.3e-05 with 295134 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[08:55:08.532] D [  glue  ] <FrameFanout:nfsp> call=35: see EOS 
[08:55:08.532] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=17 
[08:55:08.533] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=35 anode=7 
[08:55:08.533] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=17 
[08:55:08.533] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=35 anode=6 
[08:55:08.533] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=17 
[08:55:08.533] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=35 anode=5 
[08:55:08.533] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=17 
[08:55:08.533] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=35 anode=4 
[08:55:08.533] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=17 
[08:55:08.533] D [  aux   ] <Resampler:resmp3> EOS at call=35 
[08:55:08.533] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=35 anode=3 
[08:55:08.533] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=17 
[08:55:08.533] D [  aux   ] <Resampler:resmp2> EOS at call=35 
[08:55:08.533] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=35 anode=2 
[08:55:08.534] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=17 
[08:55:08.534] D [  aux   ] <Resampler:resmp1> EOS at call=35 
[08:55:08.534] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=35 anode=1 
[08:55:08.534] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=17 
[08:55:08.534] D [  aux   ] <Resampler:resmp0> EOS at call=35 
[08:55:08.534] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=35 anode=0 
[08:55:08.534] D [  glue  ] <FrameFanin:nfsp> EOS at call=35 with 8 
[08:55:08.534] D [  glue  ] frame sink sees EOS
[08:55:08.534] D [ pgraph ] <Pgrapher:> graph execution complete 
[08:55:08.534] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 16.52 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 15.63 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 15.09 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.31 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.55 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.78 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.13 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.11 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.28 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.28 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.26 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.25 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[08:55:08.534] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[08:55:08.535] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[08:55:08.535] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[08:55:08.535] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[08:55:08.535] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[08:55:08.535] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[08:55:08.535] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[08:55:08.535] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[08:55:08.535] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[08:55:08.535] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[08:55:08.535] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[08:55:08.535] I [ timer  ] Timer: Total node execution : 113.3000007700175 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 77502 traces tagged "gauss"
FrameSaver: q=1.42908e+07 n=1694181 tag=gauss
wclsFrameSaver: saving 99322 traces tagged "wiener"
FrameSaver: q=1.54078e+07 n=1617163 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 935 XUs and 1135 XVs -> 60 XUVs
C:0 T:1 1296 XUs and 2372 XVs -> 109 XUVs
C:0 T:2 1371 XUs and 1569 XVs -> 27 XUVs
C:0 T:3 826 XUs and 1088 XVs -> 32 XUVs
C:0 T:4 3643 XUs and 4479 XVs -> 209 XUVs
C:0 T:5 1976 XUs and 1850 XVs -> 164 XUVs
C:0 T:6 891 XUs and 1182 XVs -> 57 XUVs
C:0 T:7 1466 XUs and 2248 XVs -> 85 XUVs
C:0 T:8 12930 XUs and 23118 XVs -> 3826 XUVs
C:0 T:9 8208 XUs and 11162 XVs -> 1437 XUVs
C:0 T:10 9839 XUs and 29815 XVs -> 4136 XUVs
C:0 T:11 19918 XUs and 33107 XVs -> 5072 XUVs
C:0 T:12 18817 XUs and 35557 XVs -> 4090 XUVs
C:0 T:13 9908 XUs and 9752 XVs -> 1546 XUVs
C:0 T:14 10692 XUs and 10464 XVs -> 1363 XUVs
C:0 T:15 4856 XUs and 7192 XVs -> 588 XUVs
22801 XUVs total
3949 collection wire objects
22801 potential space points
Neighbour search...
2505695 tests to find 1119982 neighbours
Iterating with no regularization...
Begin: 1.12453e+09
0 1.022e+09
1 1.01526e+09
2 1.01416e+09
3 1.01377e+09
Now with regularization...
Begin: 9.92442e+08
0 9.92071e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
../justin-jobscript: line 158:  1145 Bus error               (core dumped) LD_PRELOAD=$XROOTD_LIB/libXrdPosixPreload.so lar -c ${FCL1} -n ${nevents} ${extra_line} ${pfn}
../justin-jobscript: line 160: date: command not found
Error in reco1
../justin-jobscript: line 164: cat: command not found
justIN time: 2025-11-03 18:51:09 UTC       justIN version: 01.05.01