Jobsub ID 263240.101@dunegpschedd02.fnal.gov

Jobsub ID	263240.101@dunegpschedd02.fnal.gov
Workflow ID	11256
Stage ID	1
User name	avizcaya@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	5242880000 (5000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-12-18 15:40:57
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-12-18 16:33:08
From worker node	Hostname	dunegli-8107790-0-fnpc22005.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	5242880000 (5000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-12-18 16:31:20
Input files	usertests:pdhd_prod_beam__259199_160_1_20251208T214426Z_gen_g4_IonScintPDExt.root_260263_26_1_20251210T193905Z_PDInt.root_262713_2_1_20251215T204741Z_detsim_reco1.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-12-18 16:33:08
Saved logs	justin-logs:263240.101-dunegpschedd02.fnal.gov.logs.tgz
List job events (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

zer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4850
pandora2Shower object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4729
pandora2Track object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4850
pandora2Shower object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4729
pandora2Track object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4850
pandora2Shower object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4729
pandora2Track object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4850
pandora2Shower object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4729
pandora2Track object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4850
pandora2Shower object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4729
pandora2Track object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4850
pandora2Shower object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4729
pandora2Track object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:4850
pandora2Shower object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 16:32:26 UTC run: 20250627 subRun: 1 event: 163 PDSPAnalyzer_module.cc:5239
beam pandora2Track object not found, moving on
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 18-Dec-2025 16:32:32 UTC  run: 20250627 subRun: 1 event: 164
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.164
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 18-Dec-2025 16:32:38 UTC  run: 20250627 subRun: 1 event: 165
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.165
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 18-Dec-2025 16:32:47 UTC  run: 20250627 subRun: 1 event: 166
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.166
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 18-Dec-2025 16:32:52 UTC  run: 20250627 subRun: 1 event: 167
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.167
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 18-Dec-2025 16:32:59 UTC  run: 20250627 subRun: 1 event: 168
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.168
18-Dec-2025 16:32:59 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/a2/2a/pdhd_prod_beam__259199_160_1_20251208T214426Z_gen_g4_IonScintPDExt.root_260263_26_1_20251210T193905Z_PDInt.root_262713_2_1_20251215T204741Z_detsim_reco1.root"
18-Dec-2025 16:32:59 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/a2/2a/pdhd_prod_beam__259199_160_1_20251208T214426Z_gen_g4_IonScintPDExt.root_260263_26_1_20251210T193905Z_PDInt.root_262713_2_1_20251215T204741Z_detsim_reco1.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                        0.000357488     3.48515       5.5877        3.72604       1.56998         9     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)             0.0003163    0.00353951     0.0145693    0.000371003   0.00567881        9     
end_path:pduneana:PDSPAnalyzer      2.66647       3.91663       5.5872        3.78501       1.03016         8     
====================================================================================================================

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                        0.000357488     3.48515       5.5877        3.72604       1.56998         9     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)             0.0003163    0.00353951     0.0145693    0.000371003   0.00567881        9     
end_path:pduneana:PDSPAnalyzer      2.66647       3.91663       5.5872        3.78501       1.03016         8     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3970.72 MB
  Peak resident set size usage (VmHWM): 2929.3 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3970.72 MB
  Peak resident set size usage (VmHWM): 2929.3 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 9 passed = 9 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          9          8          1 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 56.228141 Real = 59.188827

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3970.72 VmHWM = 2929.3

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
%MSG-s ArtException:  PostEndJob 18-Dec-2025 16:33:00 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250627 subRun: 1 event: 168
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
%MSG-s ArtException:  PostEndJob 18-Dec-2025 16:33:00 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250627 subRun: 1 event: 168
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.