Jobsub ID 275657.141@dunegpschedd02.fnal.gov
| Jobsub ID | 275657.141@dunegpschedd02.fnal.gov |
| Workflow ID | 12080 |
| Stage ID | 3 |
| User name | usher@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 28800 (8 hours) |
| Submitted time | 2026-01-18 21:28:13 |
| Site | CA_SFU |
| Entry | DUNE_CA_SFU_lcg-ce3 |
| Last heartbeat | 2026-01-18 22:52:05 |
| From worker node | Hostname | fc30566.int.fir.alliancecan.ca |
| cpuinfo | AMD EPYC 9655 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 84598 (23 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-01-18 22:35:41 |
| Input files | usertests:mpvmpr_monte-carlo-012080-002121_gen_g4.root
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-01-18 22:52:05 |
| Saved logs | justin-logs:275657.141-dunegpschedd02.fnal.gov.logs.tgz |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
=================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 0.00296451 41.5211 87.347 37.5932 20.9209 20
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000682451 0.0108195 0.0884444 0.00210075 0.0254812 20
simulate:tpcrawdecoder:WireCellToolkit 20.4367 43.1037 85.9814 41.0855 18.8172 19
simulate:opdigi:OpDetDigitizerDUNE 0.111042 0.234881 0.400629 0.240448 0.0901345 19
simulate:rns:RandomNumberSaver 1.344e-05 2.39564e-05 0.00012905 1.751e-05 2.49296e-05 19
[art]:TriggerResults:TriggerResultInserter 7.84e-06 1.01526e-05 2.617e-05 9.42e-06 3.88531e-06 19
end_path:out1:RootOutput 2.6e-06 3.75905e-06 1.271e-05 3.01e-06 2.21688e-06 19
end_path:out1:RootOutput(write) 0.112818 0.35606 1.13631 0.254238 0.266981 19
================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4249.53 MB
Peak resident set size usage (VmHWM): 2557.09 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 18-Jan-2026 14:51:56 PST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- StdException BEGIN
An exception was thrown while processing module WireCellToolkit/tpcrawdecoder run: 212121 subRun: 0 event: 20
../../source/wirecell-0.32.1/util/src/Waveform.cxx(96): Throw in function percentile
Dynamic exception type: boost::wrapexcept<WireCell::ValueError>
[tag_stacktrace*] = 0# WireCell::Waveform::percentile(std::vector<float, std::allocator<float> > const&, float) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellUtil.so
1# WireCell::SigProc::OmnibusSigProc::restore_baseline(Eigen::Array<float, -1, -1, 0, -1, -1>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
2# WireCell::SigProc::OmnibusSigProc::decon_2D_looseROI(int) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
3# WireCell::SigProc::OmnibusSigProc::operator()(std::shared_ptr<WireCell::IFrame const> const&, std::shared_ptr<WireCell::IFrame const>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
4# WireCell::IFunctionNode<WireCell::IFrame, WireCell::IFrame>::operator()(boost::any const&, boost::any&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellIface.so
5# tbb::detail::d1::apply_body_task_bypass<tbb::detail::d1::function_input_base<std::pair<unsigned long, boost::any>, tbb::detail::d1::graph_policy_namespace::queueing, tbb::detail::d1::cache_aligned_allocator<std::pair<unsigned long, boost::any> >, tbb::detail::d1::function_input<std::pair<unsigned long, boost::any>, std::pair<unsigned long, boost::any>, tbb::detail::d1::graph_policy_namespace::queueing, tbb::detail::d1::cache_aligned_allocator<std::pair<unsigned long, boost::any> > > >, std::pair<unsigned long, boost::any> >::execute(tbb::detail::d1::execution_data&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellTbb.so
6# 0x00001512F44424F1 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
7# 0x00001512F4441437 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
8# 0x00001512F44408BA in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
9# tbb::detail::r1::wait(tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
10# tbb::detail::d1::task_arena_function<tbb::detail::d1::graph::wait_for_all()::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const in /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libImt.so.6.28
11# 0x00001512F4416647 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
12# tbb::detail::r1::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
13# WireCellTbb::DataFlowGraph::run() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellTbb.so
14# WireCell::Main::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellApps.so
15# wcls::WCLS::process(art::Event&) in /cvmfs/larsoft.opensciencegrid.org/products/larwirecell/v10_01_28/slf7.x86_64.e26.prof/lib/liblarwirecell_Tools_WCLS_tool.so
16# art::detail::Producer::doEvent(art::EventPrincipal&, art::ModuleContext const&, std::atomic<unsigned long>&, std::atomic<unsigned long>&, std::atomic<unsigned long>&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Core.so
17# art::Worker::runWorker(art::EventPrincipal&, art::ModuleContext const&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
18# 0x00001512EF8407E3 in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
19# tbb::detail::d1::function_task<hep::concurrency::SerialTaskQueue::QueuedTask>::execute(tbb::detail::d1::execution_data&) in /cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_09_02/slf7.x86_64.e26.prof/lib/libhep_concurrency.so
20# 0x00001512F44424F1 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
21# 0x00001512F4441437 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
22# 0x00001512F44408BA in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
23# tbb::detail::r1::execute_and_wait(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
24# void art::EventProcessor::process<(art::Level)4>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
25# void art::EventProcessor::process<(art::Level)3>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
26# void art::EventProcessor::process<(art::Level)2>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
27# void art::EventProcessor::process<(art::Level)1>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
28# void art::EventProcessor::process<(art::Level)0>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
29# art::EventProcessor::runToCompletion() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
30# art::run_art(int, char**, boost::program_options::options_description&, std::vector<std::unique_ptr<art::OptionsHandler, std::default_delete<art::OptionsHandler> >, std::allocator<std::unique_ptr<art::OptionsHandler, std::default_delete<art::OptionsHandler> > > >&&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
31# artapp(int, char**, bool) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
32# main in lar
33# __libc_start_main in /lib64/libc.so.6
34# 0x00000000004012D0 in lar
[WireCell::tag_errmsg*] = empty waveform
---- StdException END
Exception going through path simulate
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 19 entries while sim::SimEnergyDeposits_largeant_LArG4DetectorServicevolTPCPlaneVInner0_G4. has 50 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
ERROR: Output file mpvmpr_monte-carlo-012080-002121_gen_g4_detsim.root not produced! 