Jobsub ID 259192.178@dunegpschedd02.fnal.gov
| Jobsub ID | 259192.178@dunegpschedd02.fnal.gov |
| Workflow ID | 11100 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 2 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-12-08 20:46:53 |
| Site | NL_SURFsara |
| Entry | DUNE_SurfSARA_arc02 |
| Last heartbeat | 2025-12-08 23:37:27 |
| From worker node | Hostname | wn-da-04.gina.surf.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 2 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-12-08 22:18:12 |
| Input files | vd-protodune:np02vd_raw_run041242_0646_df-s04-d0_dw_0_20251208T211601.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-12-08 23:37:27 |
| Saved logs | justin-logs:259192.178-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
roc ] <OmnibusSigProc:anode4sigproc4> call=28 init enlarge wire number in plane 0 from 476 to 496
[00:36:48.796] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=28 init enlarge wire number in plane 1 from 476 to 496
[00:36:48.796] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=28 init enlarge wire number in plane 2 from 584 to 604
[00:36:48.828] D [sigproc ] <OmnibusSigProc:anode4sigproc4> rebase_waveform for plane 0 with m_rebase_nbins = 200
[00:36:48.883] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=28 load plane index: 0, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 8.8019e-05 296.2 420.916 312.164 87.1615 15
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.3351e-05 8.71479e-05 0.000114798 8.7786e-05 1.28019e-05 15
produce:tpcrawdecoder:PDVDTPCReader 105.45 131.988 151.242 130.076 14.1377 15
produce:triggerrawdecoder:PDVDTriggerReader4 0.554857 0.896618 2.1491 0.785985 0.386991 15
produce:pdvddaphne:DAPHNEReaderPDVD 0.000285895 0.000331558 0.000637564 0.00030991 8.25349e-05 15
produce:ophit:OpHitFinder 5.1919e-05 8.51982e-05 0.000479273 5.4123e-05 0.000105465 15
produce:opflash:OpFlashFinderVerticalDrift 3.8754e-05 5.95354e-05 0.000275905 4.0837e-05 5.826e-05 15
produce:wclsdatavd:WireCellToolkit 91.5758 122.028 134.311 124.387 11.8851 14
produce:gaushit:GausHitFinder 1.01028 1.38522 2.30719 1.36946 0.330314 14
produce:nhitsfilter:NumberOfHitsFilter 0.000210521 0.000326093 0.000465427 0.00033325 6.83072e-05 14
produce:reco3d:SpacePointSolver 3.67772 8.34697 11.6967 8.76424 2.45029 14
produce:hitpdune:DisambigFromSpacePoints 0.102204 0.179664 0.277699 0.185535 0.0499379 14
produce:pandora:StandardPandora 22.0985 48.7204 156.82 41.8063 32.3684 14
produce:pandoraTrack:LArPandoraTrackCreation 0.272149 0.675972 1.33789 0.682616 0.28351 14
produce:pandoraGnocalo:GnocchiCalorimetry 0.0118973 0.0218197 0.0300971 0.0216669 0.00484473 14
[art]:TriggerResults:TriggerResultInserter 1.623e-05 2.15374e-05 5.6378e-05 1.8445e-05 9.84116e-06 14
end_path:out1:RootOutput 4.087e-06 5.94214e-06 2.2674e-05 4.744e-06 4.65079e-06 14
end_path:out1:RootOutput(write) 3.7903 3.95689 4.11385 3.97361 0.120902 14
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5630.4 MB
Peak resident set size usage (VmHWM): 3605.24 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 09-Dec-2025 00:36:53 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- StdException BEGIN
An exception was thrown while processing module WireCellToolkit/wclsdatavd run: 41242 subRun: 1 event: 33890
../../source/wirecell-0.32.1/util/src/Waveform.cxx(96): Throw in function percentile
Dynamic exception type: boost::wrapexcept<WireCell::ValueError>
[tag_stacktrace*] = 0# WireCell::Waveform::percentile(std::vector<float, std::allocator<float> > const&, float) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellUtil.so
1# WireCell::SigProc::OmnibusSigProc::restore_baseline(Eigen::Array<float, -1, -1, 0, -1, -1>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
2# WireCell::SigProc::OmnibusSigProc::decon_2D_looseROI(int) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
3# WireCell::SigProc::OmnibusSigProc::operator()(std::shared_ptr<WireCell::IFrame const> const&, std::shared_ptr<WireCell::IFrame const>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
4# WireCell::IFunctionNode<WireCell::IFrame, WireCell::IFrame>::operator()(boost::any const&, boost::any&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellIface.so
5# WireCell::Pgraph::Function::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
6# WireCell::Pgraph::Graph::execute() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
7# WireCell::Pgraph::Pgrapher::execute() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
8# WireCell::Main::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellApps.so
9# wcls::WCLS::process(art::Event&) in /cvmfs/larsoft.opensciencegrid.org/products/larwirecell/v10_01_24/slf7.x86_64.e26.prof/lib/liblarwirecell_Tools_WCLS_tool.so
10# art::detail::Producer::doEvent(art::EventPrincipal&, art::ModuleContext const&, std::atomic<unsigned long>&, std::atomic<unsigned long>&, std::atomic<unsigned long>&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Core.so
11# art::Worker::runWorker(art::EventPrincipal&, art::ModuleContext const&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
12# 0x0000148ACEA407E3 in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
13# tbb::detail::d1::function_task<hep::concurrency::SerialTaskQueue::QueuedTask>::execute(tbb::detail::d1::execution_data&) in /cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_09_02/slf7.x86_64.e26.prof/lib/libhep_concurrency.so
14# 0x0000148AD3E424F1 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
15# 0x0000148AD3E41437 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
16# 0x0000148AD3E408BA in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
17# tbb::detail::r1::execute_and_wait(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
18# void art::EventProcessor::process<(art::Level)4>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
19# void art::EventProcessor::process<(art::Level)3>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
20# void art::EventProcessor::process<(art::Level)2>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
21# void art::EventProcessor::process<(art::Level)1>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
22# void art::EventProcessor::process<(art::Level)0>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
23# art::EventProcessor::runToCompletion() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
24# art::run_art(int, char**, boost::program_options::options_description&, std::vector<std::unique_ptr<art::OptionsHandler, std::default_delete<art::OptionsHandler> >, std::allocator<std::unique_ptr<art::OptionsHandler, std::default_delete<art::OptionsHandler> > > >&&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
25# artapp(int, char**, bool) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
26# main in lar
27# __libc_start_main in /lib64/libc.so.6
28# 0x00000000004012D0 in lar
[WireCell::tag_errmsg*] = empty waveform
---- StdException END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 