Jobsub ID 264887.187@dunegpschedd01.fnal.gov
| Jobsub ID | 264887.187@dunegpschedd01.fnal.gov |
| Workflow ID | 11077 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 2 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-12-08 10:40:22 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce04 |
| Last heartbeat | 2025-12-08 10:58:16 |
| From worker node | Hostname | dune001-590683.0-lcg2682.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 2 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-12-08 10:46:07 |
| Input files | vd-protodune:np02vd_raw_run041215_8406_df-s04-d0_dw_0_20251208T101409.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-12-08 10:58:16 |
| Saved logs | justin-logs:264887.187-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
igproc ] <OmnibusSigProc:anode7sigproc7> call=4 init enlarge wire number in plane 2 from 584 to 604
[10:57:44.463] D [sigproc ] <OmnibusSigProc:anode7sigproc7> rebase_waveform for plane 0 with m_rebase_nbins = 200
[10:57:44.497] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 load plane index: 0, ntraces=1536, input bad regions: 0
[10:57:48.035] D [sigproc ] <OmnibusSigProc:anode7sigproc7> rebase_waveform for plane 1 with m_rebase_nbins = 200
[10:57:48.063] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 load plane index: 1, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 0.000117501 211.035 377.681 255.422 157.35 3
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000117501 0.000184623 0.000260368 0.000176 5.86431e-05 3
produce:tpcrawdecoder:PDVDTPCReader 16.4181 16.6115 16.8595 16.5569 0.18431 3
produce:triggerrawdecoder:PDVDTriggerReader4 0.393494 0.401003 0.405522 0.403994 0.0053464 3
produce:pdvddaphne:DAPHNEReaderPDVD 0.000518513 0.000720669 0.00102923 0.000614262 0.00022166 3
produce:ophit:OpHitFinder 0.000209333 0.000534299 0.000906561 0.000487003 0.0002866 3
produce:opflash:OpFlashFinderVerticalDrift 0.000110246 0.000286704 0.000545403 0.000204464 0.000186928 3
produce:wclsdatavd:WireCellToolkit 111.874 118.743 125.612 118.743 6.86902 2
produce:gaushit:GausHitFinder 2.2037 2.29664 2.38958 2.29664 0.0929381 2
produce:nhitsfilter:NumberOfHitsFilter 0.000502112 0.000587187 0.000672261 0.000587187 8.50745e-05 2
produce:reco3d:SpacePointSolver 17.0124 17.478 17.9436 17.478 0.465637 2
produce:hitpdune:DisambigFromSpacePoints 0.339511 0.370948 0.402385 0.370948 0.031437 2
produce:pandora:StandardPandora 83.7576 152.366 220.975 152.366 68.6087 2
produce:pandoraTrack:LArPandoraTrackCreation 1.25209 1.533 1.81392 1.533 0.280911 2
produce:pandoraGnocalo:GnocchiCalorimetry 0.0361286 0.0440543 0.0519801 0.0440543 0.00792575 2
[art]:TriggerResults:TriggerResultInserter 3.1649e-05 8.805e-05 0.000144451 8.805e-05 5.6401e-05 2
end_path:out1:RootOutput 1.061e-05 3.43695e-05 5.8129e-05 3.43695e-05 2.37595e-05 2
end_path:out1:RootOutput(write) 6.15167 6.54658 6.94149 6.54658 0.39491 2
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5092.41 MB
Peak resident set size usage (VmHWM): 2876.73 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 08-Dec-2025 10:57:51 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- StdException BEGIN
An exception was thrown while processing module WireCellToolkit/wclsdatavd run: 41215 subRun: 1 event: 392400
../../source/wirecell-0.32.1/util/src/Waveform.cxx(96): Throw in function percentile
Dynamic exception type: boost::wrapexcept<WireCell::ValueError>
[tag_stacktrace*] = 0# WireCell::Waveform::percentile(std::vector<float, std::allocator<float> > const&, float) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellUtil.so
1# WireCell::SigProc::OmnibusSigProc::restore_baseline(Eigen::Array<float, -1, -1, 0, -1, -1>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
2# WireCell::SigProc::OmnibusSigProc::decon_2D_looseROI(int) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
3# WireCell::SigProc::OmnibusSigProc::operator()(std::shared_ptr<WireCell::IFrame const> const&, std::shared_ptr<WireCell::IFrame const>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
4# WireCell::IFunctionNode<WireCell::IFrame, WireCell::IFrame>::operator()(boost::any const&, boost::any&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellIface.so
5# WireCell::Pgraph::Function::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
6# WireCell::Pgraph::Graph::execute() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
7# WireCell::Pgraph::Pgrapher::execute() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
8# WireCell::Main::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellApps.so
9# wcls::WCLS::process(art::Event&) in /cvmfs/larsoft.opensciencegrid.org/products/larwirecell/v10_01_24/slf7.x86_64.e26.prof/lib/liblarwirecell_Tools_WCLS_tool.so
10# art::detail::Producer::doEvent(art::EventPrincipal&, art::ModuleContext const&, std::atomic<unsigned long>&, std::atomic<unsigned long>&, std::atomic<unsigned long>&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Core.so
11# art::Worker::runWorker(art::EventPrincipal&, art::ModuleContext const&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
12# 0x00007FBACC8407E3 in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
13# tbb::detail::d1::function_task<hep::concurrency::SerialTaskQueue::QueuedTask>::execute(tbb::detail::d1::execution_data&) in /cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_09_02/slf7.x86_64.e26.prof/lib/libhep_concurrency.so
14# 0x00007FBAD1C424F1 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
15# 0x00007FBAD1C41437 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
16# 0x00007FBAD1C408BA in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
17# tbb::detail::r1::execute_and_wait(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
18# void art::EventProcessor::process<(art::Level)4>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
19# void art::EventProcessor::process<(art::Level)3>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
20# void art::EventProcessor::process<(art::Level)2>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
21# void art::EventProcessor::process<(art::Level)1>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
22# void art::EventProcessor::process<(art::Level)0>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
23# art::EventProcessor::runToCompletion() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
24# art::run_art(int, char**, boost::program_options::options_description&, std::vector<std::unique_ptr<art::OptionsHandler, std::default_delete<art::OptionsHandler> >, std::allocator<std::unique_ptr<art::OptionsHandler, std::default_delete<art::OptionsHandler> > > >&&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
25# artapp(int, char**, bool) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
26# main in lar
27# __libc_start_main in /lib64/libc.so.6
28# 0x00000000004012D0 in lar
[WireCell::tag_errmsg*] = empty waveform
---- StdException END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 