justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 279148.0@dunegpschedd01.fnal.gov

Jobsub ID	279148.0@dunegpschedd01.fnal.gov
Workflow ID	11962
Stage ID	1
User name	ykermaid@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	3600 (1 hours)
Submitted time	2026-01-15 13:49:54
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc02
Last heartbeat	2026-01-15 13:54:54
From worker node	Hostname	wn-db-07.gina.surf.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-01-15 13:51:00
Input files	vd-protodune-det-reco:np02vd_raw_run039350_0735_df-s04-d2_dw_0_20250910T054753_reco_stage1_20250910T084724_keepup.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-01-15 13:54:54
Saved logs	justin-logs:279148.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

doraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  15-Jan-2026 14:54:15 CET run: 39350 subRun: 1 event: 53846
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
15-Jan-2026 14:54:15 CET  Opened output file with pattern "%ifb_%tc_reproc.root"
15-Jan-2026 14:54:15 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/vd-protodune-det-reco/02/fe/np02vd_raw_run039350_0735_df-s04-d2_dw_0_20250910T054753_reco_stage1_20250910T084724_keepup.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5036.65 MB
  Peak resident set size usage (VmHWM): 3912.91 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Jan-2026 14:54:16 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- ProductNotFound BEGIN
      Found zero products matching all selection criteria
        C++ type: std::vector<recob::Track>
        Module label: 'spacepts'
        Product instance name: ''
        Process name: (empty)
      The above exception was thrown while processing module Chi2ParticleID/pandorapid run: 39350 subRun: 1 event: 53846
    ---- ProductNotFound END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while dunedaq::trgdataformats2::TriggerActivityDatadunedaq::trgdataformats2::TriggerCandidateDatavoidart::Assns_triggerrawdecoder_daq_pdvdkeepupstage1. has 19 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1

justIN time: 2026-02-04 07:44:47 UTC       justIN version: 01.06.00