Jobsub ID 297956.21@dunegpschedd02.fnal.gov

Jobsub ID	297956.21@dunegpschedd02.fnal.gov
Workflow ID	12718
Stage ID	2
User name	ykermaid@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
Submitted time	2026-02-04 00:12:27
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2026-02-04 00:18:39
From worker node	Hostname	dunegli-8534387-0-fnpc23011.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-04 00:18:09
Input files	usertests:np02vd_raw_run042422_0116_df-s02-d1_dw_0_20260203T161728_reco_stage0_20260203T234122_offline.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-04 00:18:39
Saved logs	justin-logs:297956.21-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

e Name:  * Type: art::Assns<recob::Shower, recob::Track, void>*              *
* Association Name:  ShowerTrackHitAssn       * Instance Name:  * Type: art::Assns<recob::Track, recob::Hit, void>*                 *
* Association Name:  clusterAssociationsbase  * Instance Name:  * Type: art::Assns<recob::Shower, recob::Cluster, void>*            *
* Association Name:  hitAssociationsbase      * Instance Name:  * Type: art::Assns<recob::Shower, recob::Hit, void>*                *
* Association Name:  pfShowerAssociationsbase * Instance Name:  * Type: art::Assns<recob::Shower, recob::PFParticle, void>*         *
* Association Name:  spShowerAssociationsbase * Instance Name:  * Type: art::Assns<recob::Shower, recob::SpacePoint, void>*         *
*************************************************************************************************************************************
      [92m[INFO][00m [0m [94m<ProcessDriver::configure::L174>[00m Start looping process list to instantiate processes
    [95m[NORMAL][00m [0m [94m<ProcessDriver::configure>[00m Instantiating Process ID=0 Type: SuperaBBoxInteraction w/ Name: SuperaBBoxInteraction
    [95m[NORMAL][00m [0m [94m<ProcessDriver::configure>[00m Instantiating Process ID=1 Type: SuperaSpacePoint w/ Name: SuperaSpacePoint
     [94m[DEBUG][00m [0m [94m<ProcessDriver::process_names>[00m ProcessDriver.cxx::L72 Called
     [94m[DEBUG][00m [0m [94m<ProcessDriver::override_output_file>[00m ProcessDriver.cxx::L45 Called
04-Feb-2026 00:18:25 UTC  Initiating request to open input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/ea/ff/np02vd_raw_run042422_0116_df-s02-d1_dw_0_20260203T161728_reco_stage0_20260203T234122_offline.root"
04-Feb-2026 00:18:26 UTC  Opened input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/ea/ff/np02vd_raw_run042422_0116_df-s02-d1_dw_0_20260203T161728_reco_stage0_20260203T234122_offline.root"
     [94m[DEBUG][00m [0m [94m<ProcessDriver::initialize>[00m ProcessDriver.cxx::L216 Called
      [92m[INFO][00m [0m [94m<ProcessDriver::initialize::L224>[00m Initializing IO 
     [94m[DEBUG][00m [0m [94m<IOManager::initialize>[00m IOManager.cxx::L131 start
      [92m[INFO][00m [0m [94m<IOManager::initialize::L136>[00m Opening an output file: larcv_stage1_20260204_001825_475175.root
      [92m[INFO][00m [0m [94m<ProcessDriver::initialize::L250>[00m Initializing: SuperaBBoxInteraction
      [92m[INFO][00m [0m [94m<ProcessDriver::initialize::L250>[00m Initializing: SuperaSpacePoint
      [92m[INFO][00m [0m [94m<ProcessDriver::initialize::L258>[00m Preparing access index vector
     [94m[DEBUG][00m [0m [94m<ProcessDriver::process_names>[00m ProcessDriver.cxx::L72 Called
     [94m[DEBUG][00m [0m [94m<ProcessDriver::process_ptr>[00m ProcessDriver.cxx::L84 Called
     [94m[DEBUG][00m [0m [94m<ProcessDriver::process_names>[00m ProcessDriver.cxx::L72 Called
     [94m[DEBUG][00m [0m [94m<ProcessDriver::process_ptr>[00m ProcessDriver.cxx::L84 Called
     [94m[DEBUG][00m [0m [94m<ProcessDriver::process_names>[00m ProcessDriver.cxx::L72 Called
db: runtime1770135447
Begin processing the 1st record. run: 42422 subRun: 1 event: 18805 at 04-Feb-2026 00:18:26 UTC
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_18_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/pandora/v04_17_05/src/pandora-v04-17-05/PandoraSDK-v04-01-00/src/Api/PandoraContentApiImpl.cc line#: 263
Failure in algorithm Alg0003, LArMaster, STATUS_CODE_NOT_INITIALIZED
     [94m[DEBUG][00m [0m [94m<ProcessDriver::process_ptr>[00m ProcessDriver.cxx::L84 Called
     [94m[DEBUG][00m [0m [94m<ProcessDriver::process_ptr>[00m ProcessDriver.cxx::L84 Called
Attempted to load data: cluster3d
      [92m[INFO][00m [0m [94m<IOManager::set_id::L569>[00m Request to set event id: 0042422_00001_018805
     [94m[DEBUG][00m [0m [94m<ProcessDriver::process_entry>[00m ProcessDriver.cxx::L314 Called
      [92m[INFO][00m [0m [94m<SuperaBBoxInteraction::process::L173>[00m 3D Meta:X range: -514.08 => 514.08 ... 1344 bins
Y range: -342.72 => 342.72 ... 896 bins
Z range: -21.71 => 321.01 ... 448 bins

     [94m[DEBUG][00m [0m [94m<IOManager::get_data>[00m IOManager.cxx::L499 start
     [94m[DEBUG][00m [0m [94m<IOManager::producer_id>[00m IOManager.cxx::L479 start
     [94m[DEBUG][00m [0m [94m<IOManager::register_producer>[00m IOManager.cxx::L178 start
      [92m[INFO][00m [0m [94m<IOManager::register_producer::L184>[00m Requested to register a producer: reco (TTree sparse3d_reco_tree)
      [92m[INFO][00m [0m [94m<IOManager::register_producer::L201>[00m It is a new producer registration (key=0)
      [92m[INFO][00m [0m [94m<IOManager::register_producer::L225>[00m kWRITE/kBOTH mode: creating an output TTree
     [94m[DEBUG][00m [0m [94m<IOManager::register_producer>[00m IOManager.cxx::L226 Branch name: sparse3d_reco_branch data pointer: 0xd55b380(0/1000)
     [94m[DEBUG][00m [0m [94m<IOManager::register_producer>[00m IOManager.cxx::L230 Created TTree @ 0xd95c300 ... TBranch @ 0x5786f20
    [95m[NORMAL][00m [0m [94m<IOManager::get_data>[00m Created TTree sparse3d_reco_tree (id=0) w/ 0 entries...
     [94m[DEBUG][00m [0m [94m<IOManager::get_data>[00m IOManager.cxx::L519 start
     [94m[DEBUG][00m [0m [94m<IOManager::get_data>[00m IOManager.cxx::L499 start
     [94m[DEBUG][00m [0m [94m<IOManager::producer_id>[00m IOManager.cxx::L479 start
     [94m[DEBUG][00m [0m [94m<IOManager::get_data>[00m IOManager.cxx::L519 start
      [92m[INFO][00m [0m [94m<SuperaSpacePoint::process::L83>[00m Voxel3DMeta: X range: -514.08 => 514.08 ... 1344 bins
Y range: -342.72 => 342.72 ... 896 bins
Z range: -21.71 => 321.01 ... 448 bins
04-Feb-2026 00:18:27 UTC  Opened output file with pattern "%ifb_reco_stage1_%tc_offline.root"
     [94m[DEBUG][00m [0m [94m<ProcessDriver::finalize>[00m ProcessDriver.cxx::L424 called
      [92m[INFO][00m [0m [94m<ProcessDriver::finalize::L427>[00m Finalizing: SuperaBBoxInteraction
      [92m[INFO][00m [0m [94m<ProcessDriver::finalize::L427>[00m Finalizing: SuperaSpacePoint
      [92m[INFO][00m [0m [94m<ProcessDriver::finalize::L434>[00m Compiling time profile...
      [92m[INFO][00m [0m [94m<ProcessDriver::finalize::L457>[00m Finalizing IO...
     [94m[DEBUG][00m [0m [94m<IOManager::finalize>[00m IOManager.cxx::L611 start
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tree with 0 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Closing output file
      [92m[INFO][00m [0m [94m<IOManager::finalize::L639>[00m Deleting data pointers
     [94m[DEBUG][00m [0m [94m<IOManager::reset>[00m IOManager.cxx::L647 start
      [92m[INFO][00m [0m [94m<ProcessDriver::finalize::L459>[00m Resetting...
     [94m[DEBUG][00m [0m [94m<ProcessDriver::reset>[00m ProcessDriver.cxx::L25 Called
     [94m[DEBUG][00m [0m [94m<IOManager::reset>[00m IOManager.cxx::L647 start
04-Feb-2026 00:18:30 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/ea/ff/np02vd_raw_run042422_0116_df-s02-d1_dw_0_20260203T161728_reco_stage0_20260203T234122_offline.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2035.18 MB
  Peak resident set size usage (VmHWM): 925.237 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 04-Feb-2026 00:18:30 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'cluster3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module LArSoftSuperaDriver/supera run: 42422 subRun: 1 event: 18805
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 28 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1