Jobsub ID 264850.175@dunegpschedd01.fnal.gov
| Jobsub ID | 264850.175@dunegpschedd01.fnal.gov |
| Workflow ID | 11077 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 2 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-12-08 08:38:17 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce04 |
| Last heartbeat | 2025-12-08 09:17:00 |
| From worker node | Hostname | dune001-590683.0-lcg2682.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 2 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-12-08 09:09:35 |
| Input files | vd-protodune:np02vd_raw_run041215_8135_df-s04-d1_dw_0_20251208T080737.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-12-08 09:17:00 |
| Saved logs | justin-logs:264850.175-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
igproc ] <OmnibusSigProc:anode7sigproc7> call=4 init enlarge wire number in plane 2 from 584 to 604
[09:16:28.744] D [sigproc ] <OmnibusSigProc:anode7sigproc7> rebase_waveform for plane 0 with m_rebase_nbins = 200
[09:16:28.761] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 load plane index: 0, ntraces=1536, input bad regions: 0
[09:16:31.490] D [sigproc ] <OmnibusSigProc:anode7sigproc7> rebase_waveform for plane 1 with m_rebase_nbins = 200
[09:16:31.509] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 load plane index: 1, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 0.000157044 116.73 186.083 164.107 83.0268 3
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.00015459 0.000192468 0.000265769 0.000157044 5.18416e-05 3
produce:tpcrawdecoder:PDVDTPCReader 15.5313 15.7194 15.8592 15.7678 0.138171 3
produce:triggerrawdecoder:PDVDTriggerReader4 0.391123 0.393172 0.3945 0.393894 0.00147009 3
produce:pdvddaphne:DAPHNEReaderPDVD 0.000430057 0.00070715 0.00115287 0.00053852 0.000318269 3
produce:ophit:OpHitFinder 0.000134071 0.00165213 0.0046688 0.000153528 0.00213312 3
produce:opflash:OpFlashFinderVerticalDrift 9.3195e-05 0.000285289 0.000636835 0.000125837 0.000248937 3
produce:wclsdatavd:WireCellToolkit 95.8342 108.207 120.58 108.207 12.3731 2
produce:gaushit:GausHitFinder 1.64367 1.65462 1.66558 1.65462 0.0109558 2
produce:nhitsfilter:NumberOfHitsFilter 0.000324338 0.0004694 0.000614463 0.0004694 0.000145063 2
produce:reco3d:SpacePointSolver 9.19645 9.64326 10.0901 9.64326 0.446813 2
produce:hitpdune:DisambigFromSpacePoints 0.191802 0.194462 0.197121 0.194462 0.00265954 2
produce:pandora:StandardPandora 31.5158 33.1262 34.7366 33.1262 1.61038 2
produce:pandoraTrack:LArPandoraTrackCreation 0.511175 0.536 0.560825 0.536 0.0248248 2
produce:pandoraGnocalo:GnocchiCalorimetry 0.0198123 0.0199928 0.0201732 0.0199928 0.000180434 2
[art]:TriggerResults:TriggerResultInserter 2.5508e-05 5.4362e-05 8.3216e-05 5.4362e-05 2.8854e-05 2
end_path:out1:RootOutput 4.619e-06 1.57645e-05 2.691e-05 1.57645e-05 1.11455e-05 2
end_path:out1:RootOutput(write) 5.16945 5.45966 5.74987 5.45966 0.290208 2
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4752.06 MB
Peak resident set size usage (VmHWM): 2444.53 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 08-Dec-2025 09:16:34 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- StdException BEGIN
An exception was thrown while processing module WireCellToolkit/wclsdatavd run: 41215 subRun: 1 event: 379643
../../source/wirecell-0.32.1/util/src/Waveform.cxx(96): Throw in function percentile
Dynamic exception type: boost::wrapexcept<WireCell::ValueError>
[tag_stacktrace*] = 0# WireCell::Waveform::percentile(std::vector<float, std::allocator<float> > const&, float) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellUtil.so
1# WireCell::SigProc::OmnibusSigProc::restore_baseline(Eigen::Array<float, -1, -1, 0, -1, -1>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
2# WireCell::SigProc::OmnibusSigProc::decon_2D_looseROI(int) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
3# WireCell::SigProc::OmnibusSigProc::operator()(std::shared_ptr<WireCell::IFrame const> const&, std::shared_ptr<WireCell::IFrame const>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
4# WireCell::IFunctionNode<WireCell::IFrame, WireCell::IFrame>::operator()(boost::any const&, boost::any&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellIface.so
5# WireCell::Pgraph::Function::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
6# WireCell::Pgraph::Graph::execute() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
7# WireCell::Pgraph::Pgrapher::execute() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
8# WireCell::Main::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellApps.so
9# wcls::WCLS::process(art::Event&) in /cvmfs/larsoft.opensciencegrid.org/products/larwirecell/v10_01_24/slf7.x86_64.e26.prof/lib/liblarwirecell_Tools_WCLS_tool.so
10# art::detail::Producer::doEvent(art::EventPrincipal&, art::ModuleContext const&, std::atomic<unsigned long>&, std::atomic<unsigned long>&, std::atomic<unsigned long>&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Core.so
11# art::Worker::runWorker(art::EventPrincipal&, art::ModuleContext const&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
12# 0x00007F54590407E3 in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
13# tbb::detail::d1::function_task<hep::concurrency::SerialTaskQueue::QueuedTask>::execute(tbb::detail::d1::execution_data&) in /cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_09_02/slf7.x86_64.e26.prof/lib/libhep_concurrency.so
14# 0x00007F545E4424F1 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
15# 0x00007F545E441437 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
16# 0x00007F545E4408BA in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
17# tbb::detail::r1::execute_and_wait(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
18# void art::EventProcessor::process<(art::Level)4>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
19# void art::EventProcessor::process<(art::Level)3>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
20# void art::EventProcessor::process<(art::Level)2>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
21# void art::EventProcessor::process<(art::Level)1>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
22# void art::EventProcessor::process<(art::Level)0>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
23# art::EventProcessor::runToCompletion() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
24# art::run_art(int, char**, boost::program_options::options_description&, std::vector<std::unique_ptr<art::OptionsHandler, std::default_delete<art::OptionsHandler> >, std::allocator<std::unique_ptr<art::OptionsHandler, std::default_delete<art::OptionsHandler> > > >&&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
25# artapp(int, char**, bool) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
26# main in lar
27# __libc_start_main in /lib64/libc.so.6
28# 0x00000000004012D0 in lar
[WireCell::tag_errmsg*] = empty waveform
---- StdException END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 