Jobsub ID 273875.0@dunegpschedd02.fnal.gov
| Jobsub ID | 273875.0@dunegpschedd02.fnal.gov |
| Workflow ID | 11995 |
| Stage ID | 3 |
| User name | usher@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 6291456000 (6000 MiB) |
| Wall seconds limit | 28800 (8 hours) |
| Submitted time | 2026-01-16 23:27:42 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce04 |
| Last heartbeat | 2026-01-17 00:38:24 |
| From worker node | Hostname | ccwcondor0615 |
| cpuinfo | AMD EPYC 7453 28-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 7864320000 (7500 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-01-16 23:30:34 |
| Input files | usertests:mpvmpr_monte-carlo-011995-000020_gen_g4.root
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-01-17 00:38:24 |
| Saved logs | justin-logs:273875.0-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
=================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 0.0157354 95.9104 209.886 97.3457 55.1274 41
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00271618 0.0103634 0.0422119 0.00683689 0.00817561 41
simulate:tpcrawdecoder:WireCellToolkit 13.1168 96.8506 207.07 101.974 53.094 40
simulate:opdigi:OpDetDigitizerDUNE 0.154103 0.771777 1.88669 0.627163 0.503638 40
simulate:rns:RandomNumberSaver 3.164e-05 7.29009e-05 0.000447398 5.16015e-05 6.69734e-05 40
[art]:TriggerResults:TriggerResultInserter 1.4086e-05 2.53692e-05 8.2264e-05 2.0178e-05 1.48706e-05 40
end_path:out1:RootOutput 3.587e-06 8.59632e-06 4.829e-05 5.4405e-06 8.23712e-06 40
end_path:out1:RootOutput(write) 0.0750113 0.673883 1.70066 0.580179 0.451154 40
================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4071.69 MB
Peak resident set size usage (VmHWM): 2302.77 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 17-Jan-2026 01:37:47 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- StdException BEGIN
An exception was thrown while processing module WireCellToolkit/tpcrawdecoder run: 200020 subRun: 0 event: 41
../../source/wirecell-0.32.1/util/src/Waveform.cxx(96): Throw in function percentile
Dynamic exception type: boost::wrapexcept<WireCell::ValueError>
[tag_stacktrace*] = 0# WireCell::Waveform::percentile(std::vector<float, std::allocator<float> > const&, float) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellUtil.so
1# WireCell::SigProc::OmnibusSigProc::restore_baseline(Eigen::Array<float, -1, -1, 0, -1, -1>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
2# WireCell::SigProc::OmnibusSigProc::decon_2D_looseROI(int) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
3# WireCell::SigProc::OmnibusSigProc::operator()(std::shared_ptr<WireCell::IFrame const> const&, std::shared_ptr<WireCell::IFrame const>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
4# WireCell::IFunctionNode<WireCell::IFrame, WireCell::IFrame>::operator()(boost::any const&, boost::any&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellIface.so
5# tbb::detail::d1::apply_body_task_bypass<tbb::detail::d1::function_input_base<std::pair<unsigned long, boost::any>, tbb::detail::d1::graph_policy_namespace::queueing, tbb::detail::d1::cache_aligned_allocator<std::pair<unsigned long, boost::any> >, tbb::detail::d1::function_input<std::pair<unsigned long, boost::any>, std::pair<unsigned long, boost::any>, tbb::detail::d1::graph_policy_namespace::queueing, tbb::detail::d1::cache_aligned_allocator<std::pair<unsigned long, boost::any> > > >, std::pair<unsigned long, boost::any> >::execute(tbb::detail::d1::execution_data&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellTbb.so
6# 0x0000154BFF0424F1 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
7# 0x0000154BFF041437 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
8# 0x0000154BFF0408BA in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
9# tbb::detail::r1::wait(tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
10# tbb::detail::d1::task_arena_function<tbb::detail::d1::graph::wait_for_all()::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const in /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libImt.so.6.28
11# 0x0000154BFF016647 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
12# tbb::detail::r1::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
13# WireCellTbb::DataFlowGraph::run() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellTbb.so
14# WireCell::Main::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_32_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellApps.so
15# wcls::WCLS::process(art::Event&) in /cvmfs/larsoft.opensciencegrid.org/products/larwirecell/v10_01_28/slf7.x86_64.e26.prof/lib/liblarwirecell_Tools_WCLS_tool.so
16# art::detail::Producer::doEvent(art::EventPrincipal&, art::ModuleContext const&, std::atomic<unsigned long>&, std::atomic<unsigned long>&, std::atomic<unsigned long>&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Core.so
17# art::Worker::runWorker(art::EventPrincipal&, art::ModuleContext const&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
18# 0x0000154BFA4407E3 in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
19# tbb::detail::d1::function_task<hep::concurrency::SerialTaskQueue::QueuedTask>::execute(tbb::detail::d1::execution_data&) in /cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_09_02/slf7.x86_64.e26.prof/lib/libhep_concurrency.so
20# 0x0000154BFF0424F1 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
21# 0x0000154BFF041437 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
22# 0x0000154BFF0408BA in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
23# tbb::detail::r1::execute_and_wait(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
24# void art::EventProcessor::process<(art::Level)4>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
25# void art::EventProcessor::process<(art::Level)3>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
26# void art::EventProcessor::process<(art::Level)2>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
27# void art::EventProcessor::process<(art::Level)1>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
28# void art::EventProcessor::process<(art::Level)0>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
29# art::EventProcessor::runToCompletion() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
30# art::run_art(int, char**, boost::program_options::options_description&, std::vector<std::unique_ptr<art::OptionsHandler, std::default_delete<art::OptionsHandler> >, std::allocator<std::unique_ptr<art::OptionsHandler, std::default_delete<art::OptionsHandler> > > >&&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
31# artapp(int, char**, bool) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
32# main in lar
33# __libc_start_main in /lib64/libc.so.6
34# 0x00000000004012D0 in lar
[WireCell::tag_errmsg*] = empty waveform
---- StdException END
Exception going through path simulate
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 40 entries while sim::SimEnergyDeposits_largeant_LArG4DetectorServicevolTPCPlaneVInner0_G4. has 50 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
ERROR: Output file mpvmpr_monte-carlo-011995-000020_gen_g4_detsim.root not produced! 