justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 212449.55@dunegpschedd01.fnal.gov

Jobsub ID212449.55@dunegpschedd01.fnal.gov
Workflow ID8052
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-25 19:58:39
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2025-09-25 20:08:35
From worker nodeHostnamehd79.dice.priv
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-09-25 19:59:57
Input filesvd-protodune:np02vd_raw_run039636_0020_df-s04-d1_dw_0_20250925T185514.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-25 20:08:35
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

3809 time=0 tick=500 with 21683 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":6432 [0] "mp2_roi3":5488 [0] "mp3_roi3":1682 [0] "wiener3":8081 [8081] ] cmm:[ ] input 4: frame: ident=3809 time=0 tick=512 with 0 traces.  frame tags:[ ] 0 tagged trace sets:[ ] cmm:[ ] input 5: frame: ident=3809 time=0 tick=512 with 0 traces.  frame tags:[ ] 0 tagged trace sets:[ ] cmm:[ ] input 6: frame: ident=3809 time=0 tick=512 with 0 traces.  frame tags:[ ] 0 tagged trace sets:[ ] cmm:[ ] input 7: frame: ident=3809 time=0 tick=512 with 0 traces.  frame tags:[ ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3809 time=0 tick=500 with 101302 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":13055 [0] "gauss1":4263 [0] "gauss2":3698 [0] "gauss3":6432 [0] "wiener0":17059 [17059] "wiener1":5432 [5432] "wiener2":4508 [4508] "wiener3":8081 [8081] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 35080 traces, 35080 summary
Retagger: tagging trace set: gauss with 27448 traces, 0 summary
[20:06:46.202] D [  glue  ] sink frame: #3809 @0 with 101302 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[20:06:46.203] D [  glue  ] <FrameFanout:nfsp> call=3: see EOS 
[20:06:46.203] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=1 
[20:06:46.203] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=3 anode=7 
[20:06:46.203] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=1 
[20:06:46.203] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=3 anode=6 
[20:06:46.203] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=1 
[20:06:46.203] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=3 anode=5 
[20:06:46.203] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=1 
[20:06:46.203] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=3 anode=4 
[20:06:46.203] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=1 
[20:06:46.203] D [  aux   ] <Resampler:resmp3> EOS at call=3 
[20:06:46.203] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=3 anode=3 
[20:06:46.204] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=1 
[20:06:46.204] D [  aux   ] <Resampler:resmp2> EOS at call=3 
[20:06:46.204] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=3 anode=2 
[20:06:46.204] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=1 
[20:06:46.204] D [  aux   ] <Resampler:resmp1> EOS at call=3 
[20:06:46.204] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=3 anode=1 
[20:06:46.204] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=1 
[20:06:46.204] D [  aux   ] <Resampler:resmp0> EOS at call=3 
[20:06:46.204] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=3 anode=0 
[20:06:46.204] D [  glue  ] <FrameFanin:nfsp> EOS at call=3 with 8 
[20:06:46.204] D [  glue  ] frame sink sees EOS
[20:06:46.204] D [ pgraph ] <Pgrapher:> graph execution complete 
[20:06:46.204] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.15 sec
[20:06:46.204] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.19 sec
[20:06:46.204] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.9 sec
[20:06:46.204] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.78 sec
[20:06:46.204] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.31 sec
[20:06:46.204] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.31 sec
[20:06:46.204] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.3 sec
[20:06:46.204] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.3 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:06:46.205] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[20:06:46.205] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[20:06:46.205] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[20:06:46.205] I [ timer  ] Timer: Total node execution : 35.25999953225255 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 27448 traces tagged "gauss"
FrameSaver: q=4.31843e+06 n=657009 tag=gauss
wclsFrameSaver: saving 35080 traces tagged "wiener"
FrameSaver: q=4.54139e+06 n=629112 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 3096 XUs and 3162 XVs -> 99 XUVs
C:0 T:1 32718 XUs and 37442 XVs -> 1907 XUVs
C:0 T:2 1170 XUs and 1296 XVs -> 74 XUVs
C:0 T:3 410 XUs and 566 XVs -> 22 XUVs
C:0 T:4 852 XUs and 1040 XVs -> 44 XUVs
C:0 T:5 210 XUs and 219 XVs -> 10 XUVs
C:0 T:6 1390 XUs and 1578 XVs -> 54 XUVs
C:0 T:7 812 XUs and 970 XVs -> 55 XUVs
2265 XUVs total
1364 collection wire objects
2265 potential space points
Neighbour search...
36595 tests to find 10682 neighbours
Iterating with no regularization...
Begin: 3.68602e+09
0 3.63692e+09
1 3.63421e+09
Now with regularization...
Begin: 3.61751e+09
0 3.61746e+09
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        197.077       228.201       259.326       228.201       31.1245         2     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.0843e-05    0.000142954   0.000215066   0.000142954   7.21115e-05       2     
produce:tpcrawdecoder:PDVDTPCReader               44.5737       48.8704       53.1671       48.8704       4.29667         2     
produce:triggerrawdecoder:PDVDTriggerReader4     0.393225      0.409954      0.426683      0.409954      0.0167292        2     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000384525   0.000662325   0.000940125   0.000662325    0.0002778        2     
produce:ophit:OpHitFinder                       0.000215386   0.000570236   0.000925087   0.000570236   0.000354851       2     
produce:opflash:OpFlashFinderVerticalDrift       5.274e-05    0.000217936   0.000383133   0.000217936   0.000165197       2     
produce:wclsdatavd:WireCellToolkit                36.4776       37.5584       38.6391       37.5584       1.08077         2     
produce:gaushit:GausHitFinder                    0.985786       1.11788       1.24997       1.11788      0.132091         2     
produce:nhitsfilter:NumberOfHitsFilter          0.000205448   0.000346164   0.00048688    0.000346164   0.000140716       2     
produce:reco3d:SpacePointSolver                   9.05907       9.55317       10.0473       9.55317      0.494093         2     
produce:hitpdune:DisambigFromSpacePoints         0.181314      0.198431      0.215548      0.198431      0.0171167        2     
produce:pandora:StandardPandora                   94.4161       128.18        161.945       128.18        33.7643         2     
produce:pandoraTrack:LArPandoraTrackCreation     0.609374      0.853444       1.09751      0.853444      0.244071         2     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0243194     0.0275534     0.0307874     0.0275534    0.00323402        2     
[art]:TriggerResults:TriggerResultInserter      2.2352e-05    4.69435e-05   7.1535e-05    4.69435e-05   2.45915e-05       2     
end_path:out1:RootOutput                         3.767e-06    1.47875e-05   2.5808e-05    1.47875e-05   1.10205e-05       2     
end_path:out1:RootOutput(write)                   2.83319       2.83319       2.83319       2.83319          0            1     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3772.7 MB
  Peak resident set size usage (VmHWM): 1813.88 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 25-Sep-2025 20:08:33 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TFile::WriteBuffer
    error writing all requested bytes to file ./RootOutput-5ed4-4127-372d-ebe2.root, wrote 31566 of 147267
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while recob::Clusterrecob::Hitvoidart::Assns_pandora__pdvdkeepupstage1. has 2 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-03 18:49:46 UTC       justIN version: 01.05.01