justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 266216.7@dunegpschedd01.fnal.gov

Jobsub ID266216.7@dunegpschedd01.fnal.gov
Workflow ID11074
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-12-10 18:05:13
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce03
Last heartbeat2025-12-10 18:11:23
From worker nodeHostnamedune001-651405.0-lcg2718.gridpp.rl.ac.uk
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-12-10 18:05:33
Input filesvd-protodune:np02vd_raw_run041215_4852_df-s04-d1_dw_0_20251207T062906.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-12-10 18:11:23
Saved logsjustin-logs:266216.7-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

igproc ] <OmnibusSigProc:anode7sigproc7> call=4 init enlarge wire number in plane 0 from 476 to 496 
[18:10:56.192] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 init enlarge wire number in plane 1 from 476 to 496 
[18:10:56.192] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 init enlarge wire number in plane 2 from 584 to 604 
[18:10:56.203] D [sigproc ] <OmnibusSigProc:anode7sigproc7> rebase_waveform for plane 0 with m_rebase_nbins = 200 
[18:10:56.215] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 load plane index: 0, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      0.000162695     90.2812       139.743       131.101       63.9358         3     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      0.000151127   0.000203246   0.000295915   0.000162695   6.56971e-05       3     
produce:tpcrawdecoder:PDVDTPCReader               15.078        15.246        15.5329       15.1272      0.203843         3     
produce:triggerrawdecoder:PDVDTriggerReader4     0.394295      0.396156      0.399429      0.394744     0.00232174        3     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000429215   0.000601834   0.000933373   0.000442915    0.0002345        3     
produce:ophit:OpHitFinder                       0.000135644   0.000396486   0.000869417   0.000184397   0.000335004       3     
produce:opflash:OpFlashFinderVerticalDrift      8.3035e-05    0.000212142   0.000447833   0.000105558   0.000166912       3     
produce:wclsdatavd:WireCellToolkit                72.6256       73.1734       73.7212       73.1734      0.547817         2     
produce:gaushit:GausHitFinder                     1.47549       1.50604       1.53659       1.50604      0.0305502        2     
produce:nhitsfilter:NumberOfHitsFilter          0.000276936   0.000442059   0.000607182   0.000442059   0.000165123       2     
produce:reco3d:SpacePointSolver                   6.89452       7.51036       8.12621       7.51036      0.615846         2     
produce:hitpdune:DisambigFromSpacePoints         0.169206      0.172743      0.176281      0.172743      0.0035374        2     
produce:pandora:StandardPandora                   27.035        32.617        38.1989       32.617        5.58198         2     
produce:pandoraTrack:LArPandoraTrackCreation     0.470914      0.605258      0.739602      0.605258      0.134344         2     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0285857     0.0291033     0.0296209     0.0291033    0.000517612       2     
[art]:TriggerResults:TriggerResultInserter      6.0321e-05    8.1798e-05    0.000103275   8.1798e-05    2.1477e-05        2     
end_path:out1:RootOutput                         7.672e-06    2.00555e-05   3.2439e-05    2.00555e-05   1.23835e-05       2     
end_path:out1:RootOutput(write)                   4.19465       4.24731       4.29997       4.24731      0.0526625        2     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4823.36 MB
  Peak resident set size usage (VmHWM): 2477.1 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 10-Dec-2025 18:10:58 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module WireCellToolkit/wclsdatavd run: 41215 subRun: 1 event: 226383
      ../../source/wirecell-0.32.1/util/src/Waveform.cxx(96): Throw in function percentile
      Dynamic exception type: boost::wrapexcept<WireCell::ValueError>
      [tag_stacktrace*] =  0# WireCell::Waveform::percentile(std::vector<float, std::allocator<float> > const&, float) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellUtil.so
       1# WireCell::SigProc::OmnibusSigProc::restore_baseline(Eigen::Array<float, -1, -1, 0, -1, -1>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
       2# WireCell::SigProc::OmnibusSigProc::decon_2D_looseROI(int) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
       3# WireCell::SigProc::OmnibusSigProc::operator()(std::shared_ptr<WireCell::IFrame const> const&, std::shared_ptr<WireCell::IFrame const>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
       4# WireCell::IFunctionNode<WireCell::IFrame, WireCell::IFrame>::operator()(boost::any const&, boost::any&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellIface.so
       5# WireCell::Pgraph::Function::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
       6# WireCell::Pgraph::Graph::execute() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
       7# WireCell::Pgraph::Pgrapher::execute() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
       8# WireCell::Main::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellApps.so
       9# wcls::WCLS::process(art::Event&) in /cvmfs/larsoft.opensciencegrid.org/products/larwirecell/v10_01_24/slf7.x86_64.e26.prof/lib/liblarwirecell_Tools_WCLS_tool.so
      10# art::detail::Producer::doEvent(art::EventPrincipal&, art::ModuleContext const&, std::atomic<unsigned long>&, std::atomic<unsigned long>&, std::atomic<unsigned long>&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Core.so
      11# art::Worker::runWorker(art::EventPrincipal&, art::ModuleContext const&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
      12# 0x00007F05DE4407E3 in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
      13# tbb::detail::d1::function_task<hep::concurrency::SerialTaskQueue::QueuedTask>::execute(tbb::detail::d1::execution_data&) in /cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_09_02/slf7.x86_64.e26.prof/lib/libhep_concurrency.so
      14# 0x00007F05E38424F1 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
      15# 0x00007F05E3841437 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
      16# 0x00007F05E38408BA in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
      17# tbb::detail::r1::execute_and_wait(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
      18# void art::EventProcessor::process<(art::Level)4>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
      19# void art::EventProcessor::process<(art::Level)3>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
      20# void art::EventProcessor::process<(art::Level)2>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
      21# void art::EventProcessor::process<(art::Level)1>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
      22# void art::EventProcessor::process<(art::Level)0>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
      23# art::EventProcessor::runToCompletion() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
      24# art::run_art(int, char**, boost::program_options::options_description&, std::vector<std::unique_ptr<art::OptionsHandler, std::default_delete<art::OptionsHandler> >, std::allocator<std::unique_ptr<art::OptionsHandler, std::default_delete<art::OptionsHandler> > > >&&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
      25# artapp(int, char**, bool) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
      26# main in lar
      27# __libc_start_main in /lib64/libc.so.6
      28# 0x00000000004012D0 in lar
      
      [WireCell::tag_errmsg*] = empty waveform
    ---- StdException END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-12-19 07:22:52 UTC       justIN version: 01.05.03