justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234819.2@dunegpschedd02.fnal.gov

Jobsub ID234819.2@dunegpschedd02.fnal.gov
Workflow ID9373
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-29 15:47:33
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-10-29 19:39:13
From worker nodeHostnametd815.pic.es
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 15:48:42
Input filesvd-protodune:np02vd_raw_run040266_0078_df-s03-d1_dw_0_20251024T232303.hdf5
JobscriptExit code-7
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-29 19:39:13
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ame: frame: ident=14561 time=96 tick=500 with 1536 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] 
[19:38:07.840] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 init nticks=8608 tbinmin=0 tbinmax=8608 
[19:38:07.865] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 load plane index: 0, ntraces=1536, input bad regions: 0 
[19:38:10.098] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 load plane index: 1, ntraces=1536, input bad regions: 0 
[19:38:12.336] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 load plane index: 2, ntraces=1536, input bad regions: 0 
[19:38:14.774] D [sigproc ] ROI_refinement: local ch index 182 (plane 0), found max peaks 200 with threshold=0.1
[19:38:15.128] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 save plane index: 0, Qtot=94570340 Qloss=-16710722, 2121 indices spanning [8532,10652] "wiener" 
[19:38:15.375] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 save plane index: 0, Qtot=88548002 Qloss=-14647685, 1593 indices spanning [10653,12245] "gauss" 
[19:38:15.742] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 save plane index: 1, Qtot=82634538 Qloss=-10477137, 2556 indices spanning [12246,14801] "wiener" 
[19:38:15.979] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 save plane index: 1, Qtot=73538995 Qloss=-6955364, 2000 indices spanning [14802,16801] "gauss" 
[19:38:16.354] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 save plane index: 2, Qtot=68384018 Qloss=-3586343, 2392 indices spanning [16802,19193] "wiener" 
[19:38:16.726] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 save plane index: 2, Qtot=66614054 Qloss=-1791365, 1936 indices spanning [19194,21129] "gauss" 
[19:38:16.726] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 produce 21130 traces: 7069 wiener0, 0 decon_charge0, 5529 gauss0, frame tag: sigproc 
[19:38:16.726] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 output frame: frame: ident=14561 time=96 tick=500 with 21130 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss0":5529 [0] "mp2_roi0":6838 [0] "mp3_roi0":1694 [0] "wiener0":7069 [7069] ] cmm:[ ] 
[19:38:16.752] D [  glue  ] <FrameFanin:nfsp> call=44 input 0: frame: ident=14561 time=96 tick=500 with 21130 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss0":5529 [0] "mp2_roi0":6838 [0] "mp3_roi0":1694 [0] "wiener0":7069 [7069] ] cmm:[ ] input 1: frame: ident=14561 time=96 tick=500 with 25521 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss1":6810 [0] "mp2_roi1":8481 [0] "mp3_roi1":1985 [0] "wiener1":8245 [8245] ] cmm:[ ] input 2: frame: ident=14561 time=96 tick=500 with 39242 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":9690 [0] "mp2_roi2":13928 [0] "mp3_roi2":3292 [0] "wiener2":12332 [12332] ] cmm:[ ] input 3: frame: ident=14561 time=96 tick=500 with 14342 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":3753 [0] "mp2_roi3":4914 [0] "mp3_roi3":1036 [0] "wiener3":4639 [4639] ] cmm:[ ] input 4: frame: ident=14561 time=96 tick=512 with 22854 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss4":4914 [0] "mp2_roi4":10131 [0] "mp3_roi4":1842 [0] "wiener4":5967 [5967] ] cmm:[ ] input 5: frame: ident=14561 time=96 tick=512 with 9656 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":3257 [0] "mp2_roi5":2098 [0] "mp3_roi5":662 [0] "wiener5":3639 [3639] ] cmm:[ ] input 6: frame: ident=14561 time=96 tick=512 with 37689 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":9596 [0] "mp2_roi6":13663 [0] "mp3_roi6":2756 [0] "wiener6":11674 [11674] ] cmm:[ ] input 7: frame: ident=14561 time=96 tick=512 with 25208 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":7054 [0] "mp2_roi7":8153 [0] "mp3_roi7":1568 [0] "wiener7":8433 [8433] ] cmm:[ ] output: frame: ident=14561 time=96 tick=500 with 195642 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":5529 [0] "gauss1":6810 [0] "gauss2":9690 [0] "gauss3":3753 [0] "gauss4":4914 [0] "gauss5":3257 [0] "gauss6":9596 [0] "gauss7":7054 [0] "wiener0":7069 [7069] "wiener1":8245 [8245] "wiener2":12332 [12332] "wiener3":4639 [4639] "wiener4":5967 [5967] "wiener5":3639 [3639] "wiener6":11674 [11674] "wiener7":8433 [8433] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 61998 traces, 61998 summary
Retagger: tagging trace set: gauss with 50603 traces, 0 summary
[19:38:16.765] D [  glue  ] sink frame: #14561 @9.6e-05 with 195642 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[19:38:16.765] D [  glue  ] <FrameFanout:nfsp> call=45: see EOS 
[19:38:16.765] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=22 
[19:38:16.765] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=45 anode=7 
[19:38:16.765] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=22 
[19:38:16.765] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=45 anode=6 
[19:38:16.765] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=22 
[19:38:16.765] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=45 anode=5 
[19:38:16.765] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=22 
[19:38:16.765] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=45 anode=4 
[19:38:16.765] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=22 
[19:38:16.765] D [  aux   ] <Resampler:resmp3> EOS at call=45 
[19:38:16.765] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=45 anode=3 
[19:38:16.765] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=22 
[19:38:16.765] D [  aux   ] <Resampler:resmp2> EOS at call=45 
[19:38:16.765] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=45 anode=2 
[19:38:16.766] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=22 
[19:38:16.766] D [  aux   ] <Resampler:resmp1> EOS at call=45 
[19:38:16.766] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=45 anode=1 
[19:38:16.766] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=22 
[19:38:16.766] D [  aux   ] <Resampler:resmp0> EOS at call=45 
[19:38:16.766] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=45 anode=0 
[19:38:16.766] D [  glue  ] <FrameFanin:nfsp> EOS at call=45 with 8 
[19:38:16.766] D [  glue  ] frame sink sees EOS
[19:38:16.766] D [ pgraph ] <Pgrapher:> graph execution complete 
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.97 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.1 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.77 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.4 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.7 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.62 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.11 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.76 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.38 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.37 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.37 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.36 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:38:16.766] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[19:38:16.766] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[19:38:16.766] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[19:38:16.766] I [ timer  ] Timer: Total node execution : 62.96000079996884 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 50603 traces tagged "gauss"
FrameSaver: q=1.32739e+07 n=1122393 tag=gauss
wclsFrameSaver: saving 61998 traces tagged "wiener"
FrameSaver: q=1.42312e+07 n=1085344 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 525 XUs and 655 XVs -> 16 XUVs
C:0 T:1 1631 XUs and 1746 XVs -> 83 XUVs
C:0 T:2 1519 XUs and 1708 XVs -> 74 XUVs
C:0 T:3 937 XUs and 897 XVs -> 34 XUVs
C:0 T:4 16013 XUs and 16350 XVs -> 1155 XUVs
C:0 T:5 624 XUs and 592 XVs -> 49 XUVs
C:0 T:6 149 XUs and 156 XVs -> 3 XUVs
C:0 T:7 1883 XUs and 1941 XVs -> 106 XUVs
C:0 T:8 2179 XUs and 2663 XVs -> 159 XUVs
C:0 T:9 2053 XUs and 1684 XVs -> 178 XUVs
C:0 T:10 344 XUs and 445 XVs -> 21 XUVs
C:0 T:11 127 XUs and 187 XVs -> 16 XUVs
C:0 T:12 1874 XUs and 2100 XVs -> 99 XUVs
C:0 T:13 1732 XUs and 2069 XVs -> 95 XUVs
C:0 T:14 435 XUs and 539 XVs -> 40 XUVs
C:0 T:15 1535 XUs and 2058 XVs -> 98 XUVs
2226 XUVs total
1340 collection wire objects
2226 potential space points
Neighbour search...
42336 tests to find 13352 neighbours
Iterating with no regularization...
Begin: 1.44438e+10
0 1.40367e+10
1 1.40189e+10
2 1.40186e+10
Now with regularization...
Begin: 1.36505e+10
0 1.36501e+10
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
../justin-jobscript: line 158:  1146 Bus error               (core dumped) LD_PRELOAD=$XROOTD_LIB/libXrdPosixPreload.so lar -c ${FCL1} -n ${nevents} ${extra_line} ${pfn}
../justin-jobscript: line 160: date: command not found
Error in reco1
../justin-jobscript: line 164: cat: command not found
justIN time: 2025-11-04 01:31:14 UTC       justIN version: 01.05.01