justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 279156.1@dunegpschedd01.fnal.gov

Jobsub ID279156.1@dunegpschedd01.fnal.gov
Workflow ID11962
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2026-01-15 14:01:55
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel
Last heartbeat2026-01-15 14:07:40
From worker nodeHostnamewn-snel-032.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2026-01-15 14:03:12
Input filesvd-protodune-det-reco:np02vd_raw_run039350_0735_df-s04-d2_dw_0_20250910T054753_reco_stage1_20250910T084724_keepup.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2026-01-15 14:07:40
Saved logsjustin-logs:279156.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

doraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 15:07:17 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
15-Jan-2026 15:07:17 CET  Opened output file with pattern "%ifb_%tc_reproc.root"
15-Jan-2026 15:07:17 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/vd-protodune-det-reco/02/fe/np02vd_raw_run039350_0735_df-s04-d2_dw_0_20250910T054753_reco_stage1_20250910T084724_keepup.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5034.47 MB
  Peak resident set size usage (VmHWM): 3908.79 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Jan-2026 15:07:18 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- ProductNotFound BEGIN
      Found zero products matching all selection criteria
        C++ type: std::vector<recob::Track>
        Module label: 'spacepts'
        Product instance name: ''
        Process name: (empty)
      The above exception was thrown while processing module Chi2ParticleID/pandorapid run: 39350 subRun: 1 event: 53846
    ---- ProductNotFound END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while dunedaq::trgdataformats2::TriggerActivityDatadunedaq::trgdataformats2::TriggerCandidateDatavoidart::Assns_triggerrawdecoder_daq_pdvdkeepupstage1. has 19 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2026-02-04 13:01:44 UTC       justIN version: 01.06.00