justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 231093.147@dunegpschedd02.fnal.gov

Jobsub ID231093.147@dunegpschedd02.fnal.gov
Workflow ID9166
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-20 01:38:17
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2025-10-20 03:37:21
From worker nodeHostnamehd83.dice.priv
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-20 01:39:28
Input filesvd-protodune:np02vd_raw_run040140_1853_df-s04-d2_dw_0_20251019T190844.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-20 03:37:21
Saved logsjustin-logs:231093.147-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":3765 [0] "mp2_roi7":3189 [0] "mp3_roi7":780 [0] "wiener7":4478 [4478] ] cmm:[ ] output: frame: ident=325382 time=87 tick=500 with 221084 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":7265 [0] "gauss1":13764 [0] "gauss2":4392 [0] "gauss3":5155 [0] "gauss4":9151 [0] "gauss5":8209 [0] "gauss6":7535 [0] "gauss7":3765 [0] "wiener0":9225 [9225] "wiener1":17619 [17619] "wiener2":5278 [5278] "wiener3":6354 [6354] "wiener4":11129 [11129] "wiener5":9998 [9998] "wiener6":8924 [8924] "wiener7":4478 [4478] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 73005 traces, 73005 summary
Retagger: tagging trace set: gauss with 59236 traces, 0 summary
[03:35:12.465] D [  glue  ] sink frame: #325382 @8.7e-05 with 221084 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[03:35:12.465] D [  glue  ] <FrameFanout:nfsp> call=43: see EOS 
[03:35:12.465] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=21 
[03:35:12.465] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=43 anode=7 
[03:35:12.465] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=21 
[03:35:12.465] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=43 anode=6 
[03:35:12.465] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=21 
[03:35:12.466] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=43 anode=5 
[03:35:12.466] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=21 
[03:35:12.466] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=43 anode=4 
[03:35:12.466] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=21 
[03:35:12.466] D [  aux   ] <Resampler:resmp3> EOS at call=43 
[03:35:12.466] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=43 anode=3 
[03:35:12.466] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=21 
[03:35:12.466] D [  aux   ] <Resampler:resmp2> EOS at call=43 
[03:35:12.466] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=43 anode=2 
[03:35:12.466] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=21 
[03:35:12.466] D [  aux   ] <Resampler:resmp1> EOS at call=43 
[03:35:12.466] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=43 anode=1 
[03:35:12.466] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=21 
[03:35:12.466] D [  aux   ] <Resampler:resmp0> EOS at call=43 
[03:35:12.467] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=43 anode=0 
[03:35:12.467] D [  glue  ] <FrameFanin:nfsp> EOS at call=43 with 8 
[03:35:12.467] D [  glue  ] frame sink sees EOS
[03:35:12.467] D [ pgraph ] <Pgrapher:> graph execution complete 
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 16.88 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 15.08 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.78 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.48 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.16 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.03 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.63 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.48 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.61 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.6 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.6 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.59 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.03 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[03:35:12.467] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[03:35:12.467] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[03:35:12.467] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[03:35:12.467] I [ timer  ] Timer: Total node execution : 110.02999763004482 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 59236 traces tagged "gauss"
FrameSaver: q=1.50585e+07 n=1227813 tag=gauss
wclsFrameSaver: saving 73005 traces tagged "wiener"
FrameSaver: q=1.61827e+07 n=1179071 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 818 XUs and 989 XVs -> 31 XUVs
C:0 T:1 1899 XUs and 1767 XVs -> 68 XUVs
C:0 T:2 1406 XUs and 1991 XVs -> 67 XUVs
C:0 T:3 19088 XUs and 22422 XVs -> 1520 XUVs
C:0 T:4 1008 XUs and 1199 XVs -> 22 XUVs
C:0 T:5 248 XUs and 292 XVs -> 11 XUVs
C:0 T:6 720 XUs and 756 XVs -> 34 XUVs
C:0 T:7 1046 XUs and 1375 XVs -> 79 XUVs
C:0 T:8 1686 XUs and 1468 XVs -> 80 XUVs
C:0 T:9 2092 XUs and 1648 XVs -> 123 XUVs
C:0 T:10 4096 XUs and 4872 XVs -> 271 XUVs
C:0 T:11 2077 XUs and 2230 XVs -> 158 XUVs
C:0 T:12 1334 XUs and 1694 XVs -> 79 XUVs
C:0 T:13 649 XUs and 698 XVs -> 65 XUVs
C:0 T:14 2022 XUs and 947 XVs -> 132 XUVs
C:0 T:15 128 XUs and 145 XVs -> 13 XUVs
2753 XUVs total
1747 collection wire objects
2753 potential space points
Neighbour search...
66069 tests to find 16608 neighbours
Iterating with no regularization...
Begin: 2.92894e+10
0 2.90582e+10
1 2.90523e+10
Now with regularization...
Begin: 2.83775e+10
0 2.83771e+10
20-Oct-2025 03:36:47 UTC  Closed output file "np02vd_raw_run040140_1853_df-s04-d2_dw_0_20251019T190844_reco_stage1_20251020T033647_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        234.448       332.462       648.674        310.7        90.7418        21     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.7937e-05    0.000111174   0.00027023    9.4393e-05    4.00043e-05      22     
produce:tpcrawdecoder:PDVDTPCReader               13.7807       114.21        236.454       107.928       39.5153        22     
produce:triggerrawdecoder:PDVDTriggerReader4     0.332245      0.480709      0.734395      0.431241      0.117857        22     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000480487     7.79102       12.7145       8.01396       2.26236        22     
produce:ophit:OpHitFinder                       9.0571e-05     0.0428705     0.0536368     0.0448931    0.00991418       22     
produce:opflash:OpFlashFinderVerticalDrift      9.0992e-05    0.00542482     0.0120809    0.00438882    0.00343035       22     
produce:wclsdatavd:WireCellToolkit                3.02654       108.42        224.945       112.409       38.453         22     
produce:gaushit:GausHitFinder                    0.031922       1.76785       2.71916       1.88366       0.54604        22     
produce:nhitsfilter:NumberOfHitsFilter          5.0646e-05    0.00038854    0.00071538    0.000397635   0.000120573      22     
produce:reco3d:SpacePointSolver                  0.0195876      15.3823       27.2821       15.5886       5.39654        22     
produce:hitpdune:DisambigFromSpacePoints        0.00132997     0.297717      0.630482      0.292409      0.131828        22     
produce:pandora:StandardPandora                  0.0859962      60.4073       145.845       49.706        36.0791        22     
produce:pandoraTrack:LArPandoraTrackCreation    0.00489703      2.09531       5.33704       1.91336       1.17286        22     
produce:pandoraGnocalo:GnocchiCalorimetry       0.00146254     0.0486197     0.0766729     0.0507378     0.0148919       22     
[art]:TriggerResults:TriggerResultInserter      2.4907e-05     5.206e-05    0.000263647   3.51565e-05   4.94202e-05      22     
end_path:out1:RootOutput                         4.038e-06    8.71818e-06   3.5838e-05     7.173e-06    6.68159e-06      22     
end_path:out1:RootOutput(write)                  0.293742       6.32142       12.8161       6.26623       1.93403        22     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6979.63 MB
  Peak resident set size usage (VmHWM): 5025.3 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 20-Oct-2025 03:36:47 UTC ModuleEndJob
---- EventRangeOverlap BEGIN
  Attempt to merge event ranges that both contain one or more of the same events
   Run: 40140
    SubRun: 1 Event range: [325342,325343)  vs.
    SubRun: 1 Event range: [325342,325343)
---- EventRangeOverlap END
---- EventRangeOverlap BEGIN
  Attempt to merge event ranges that both contain one or more of the same events
   Run: 40140
    SubRun: 1 Event range: [325342,325343)  vs.
    SubRun: 1 Event range: [325342,325343)
---- EventRangeOverlap END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 04:48:32 UTC       justIN version: 01.05.01