Jobsub ID 240990.0@dunegpschedd02.fnal.gov

Jobsub ID	240990.0@dunegpschedd02.fnal.gov
Workflow ID	9902
Stage ID	1
User name	mwrobel@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	5242880000 (5000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-10 19:05:18
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2025-11-10 19:10:01
From worker node	Hostname	dunegli-7562545-0-fnpc17124.fnal.gov
	cpuinfo	Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	5242880000 (5000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-11-10 19:05:56
Input files	usertests:pdhd_prod_beam__226232_91_1_20251008T182136Z_gen_g4_IonScintPDExt.root_226749_50_1_20251009T205337Z_PDInt.root_234889_15_1_20251016T190554Z_detsim_reco1.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-11-10 19:10:01
Saved logs	justin-logs:240990.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

PrepService::ctor:            digitReader
ToolBasedRawDigitPrepService::ctor:            adcSampleFiller
ToolBasedRawDigitPrepService::ctor:            adcScaleAdcToKe
ToolBasedRawDigitPrepService::ctor:            pdspTailPedRemovalZKe
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
11-Nov-2025 01:07:15 UTC  Initiating request to open input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/65/40/pdhd_prod_beam__226232_91_1_20251008T182136Z_gen_g4_IonScintPDExt.root_226749_50_1_20251009T205337Z_PDInt.root_234889_15_1_20251016T190554Z_detsim_reco1.root"
11-Nov-2025 01:07:15 UTC  Initiating request to open input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/65/40/pdhd_prod_beam__226232_91_1_20251008T182136Z_gen_g4_IonScintPDExt.root_226749_50_1_20251009T205337Z_PDInt.root_234889_15_1_20251016T190554Z_detsim_reco1.root"
11-Nov-2025 01:07:19 UTC  Opened input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/65/40/pdhd_prod_beam__226232_91_1_20251008T182136Z_gen_g4_IonScintPDExt.root_226749_50_1_20251009T205337Z_PDInt.root_234889_15_1_20251016T190554Z_detsim_reco1.root"
11-Nov-2025 01:07:19 UTC  Opened input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/65/40/pdhd_prod_beam__226232_91_1_20251008T182136Z_gen_g4_IonScintPDExt.root_226749_50_1_20251009T205337Z_PDInt.root_234889_15_1_20251016T190554Z_detsim_reco1.root"
Setting applicationFamily: "art"
Setting process_name: "PDHDAna"
db: runtime0
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2025 01:07:33 UTC  run: 20250627 subRun: 1 event: 60
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.60
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2025 01:07:58 UTC  run: 20250627 subRun: 1 event: 61
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.61
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2025 01:08:12 UTC  run: 20250627 subRun: 1 event: 62
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.62
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2025 01:08:27 UTC  run: 20250627 subRun: 1 event: 63
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.63
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2025 01:08:46 UTC  run: 20250627 subRun: 1 event: 64
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.64
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2025 01:09:08 UTC  run: 20250627 subRun: 1 event: 65
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.65
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2025 01:09:20 UTC  run: 20250627 subRun: 1 event: 66
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.66
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Nov-2025 01:09:45 UTC  run: 20250627 subRun: 1 event: 67
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.67
11-Nov-2025 01:09:46 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/65/40/pdhd_prod_beam__226232_91_1_20251008T182136Z_gen_g4_IonScintPDExt.root_226749_50_1_20251009T205337Z_PDInt.root_234889_15_1_20251016T190554Z_detsim_reco1.root"
11-Nov-2025 01:09:46 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/65/40/pdhd_prod_beam__226232_91_1_20251008T182136Z_gen_g4_IonScintPDExt.root_226749_50_1_20251009T205337Z_PDInt.root_234889_15_1_20251016T190554Z_detsim_reco1.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                        0.000651785     7.25249       15.6243       5.86795       4.98711         8     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000505394   0.00711092     0.0268217    0.000946541   0.00912178        8     
end_path:pduneana:PDSPAnalyzer      3.36636       8.28018       15.5968       7.17833       4.44667         7     
====================================================================================================================

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                        0.000651785     7.25249       15.6243       5.86795       4.98711         8     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000505394   0.00711092     0.0268217    0.000946541   0.00912178        8     
end_path:pduneana:PDSPAnalyzer      3.36636       8.28018       15.5968       7.17833       4.44667         7     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3767.42 MB
  Peak resident set size usage (VmHWM): 2724.25 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3767.42 MB
  Peak resident set size usage (VmHWM): 2724.25 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 8 passed = 8 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          8          7          1 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 129.324012 Real = 145.485011

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3767.42 VmHWM = 2724.25

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
%MSG-s ArtException:  PostEndJob 11-Nov-2025 01:09:46 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250627 subRun: 1 event: 67
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
%MSG-s ArtException:  PostEndJob 11-Nov-2025 01:09:46 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250627 subRun: 1 event: 67
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.