Jobsub ID 231743.0@dunegpschedd02.fnal.gov

Jobsub ID	231743.0@dunegpschedd02.fnal.gov
Workflow ID	9254
Stage ID	1
User name	avizcaya@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	5242880000 (5000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-21 15:51:25
Site	US_UChicago
Entry	Engage_US_MWT2_iut2_condce_mcore
Last heartbeat	2025-10-21 15:56:05
From worker node	Hostname	uct2-c502.mwt2.org
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	5242880000 (5000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-10-21 15:52:08
Input files	usertests:pdhd_prod_beam__226206_103_1_20251008T150916Z_gen_g4_IonScintPDExt.root_226551_15_1_20251009T040828Z_PDInt.root_229301_44_1_20251016T150913Z_detsim_reco1.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-10-21 15:56:05
Saved logs	justin-logs:231743.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

MaxTick: 21000
ExpTailPedRemover::ctor:              PedDegree: 0
ExpTailPedRemover::ctor:               PedTick0: 3000
ExpTailPedRemover::ctor:               PedFreqs: []
ExpTailPedRemover::ctor:   IncludeChannelRanges: [apa1z, apa2z, apa3z, apa4z, apa5z, apa6z, apa1c, apa2c, apa3c, apa4c, apa5c, apa6c]
ExpTailPedRemover::ctor:   NoWarnStatuses: [1]
ExpTailPedRemover::ctor:   ExcludeChannelRanges: []
ExpTailPedRemover::ctor: Channel checking enabled for 5760 channels.
ToolBasedRawDigitPrepService::ctor:     Found tool pdspTailPedRemovalZKe @ 0x10ee3650
ToolBasedRawDigitPrepService::ctor: Fetching wire building service.
ToolBasedRawDigitPrepService::ctor:   Wire building service: @0x5650c30
ToolBasedRawDigitPrepService::ctor: ToolBasedRawDigitPrepService:
ToolBasedRawDigitPrepService::ctor:                     LogLevel: 3
ToolBasedRawDigitPrepService::ctor:                      DoWires: 1
ToolBasedRawDigitPrepService::ctor:      ADC channel tools:
ToolBasedRawDigitPrepService::ctor:            digitReader
ToolBasedRawDigitPrepService::ctor:            adcSampleFiller
ToolBasedRawDigitPrepService::ctor:            adcScaleAdcToKe
ToolBasedRawDigitPrepService::ctor:            pdspTailPedRemovalZKe
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
21-Oct-2025 10:54:00 CDT  Initiating request to open input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/fc/c3/pdhd_prod_beam__226206_103_1_20251008T150916Z_gen_g4_IonScintPDExt.root_226551_15_1_20251009T040828Z_PDInt.root_229301_44_1_20251016T150913Z_detsim_reco1.root"
21-Oct-2025 10:54:00 CDT  Initiating request to open input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/fc/c3/pdhd_prod_beam__226206_103_1_20251008T150916Z_gen_g4_IonScintPDExt.root_226551_15_1_20251009T040828Z_PDInt.root_229301_44_1_20251016T150913Z_detsim_reco1.root"
21-Oct-2025 10:54:06 CDT  Opened input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/fc/c3/pdhd_prod_beam__226206_103_1_20251008T150916Z_gen_g4_IonScintPDExt.root_226551_15_1_20251009T040828Z_PDInt.root_229301_44_1_20251016T150913Z_detsim_reco1.root"
21-Oct-2025 10:54:06 CDT  Opened input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/fc/c3/pdhd_prod_beam__226206_103_1_20251008T150916Z_gen_g4_IonScintPDExt.root_226551_15_1_20251009T040828Z_PDInt.root_229301_44_1_20251016T150913Z_detsim_reco1.root"
Setting applicationFamily: "art"
Setting process_name: "PDHDAna"
db: runtime0
%MSG-w PhotonBackTrackerService:  ProcessEvent 21-Oct-2025 10:54:18 CDT  run: 20250627 subRun: 1 event: 50
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.50
No track calorimetry object found... returning empty vector
No shower calorimetry object found... returning empty vector
No track calorimetry object found... returning empty vector
No shower calorimetry object found... returning empty vector
No track calorimetry object found... returning empty vector
No shower calorimetry object found... returning empty vector
No track calorimetry object found... returning empty vector
No shower calorimetry object found... returning empty vector
No track calorimetry object found... returning empty vector
%MSG-w PhotonBackTrackerService:  ProcessEvent 21-Oct-2025 10:55:21 CDT  run: 20250627 subRun: 1 event: 51
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.51
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 21-Oct-2025 10:55:48 CDT  run: 20250627 subRun: 1 event: 52
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.52
21-Oct-2025 10:55:48 CDT  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/fc/c3/pdhd_prod_beam__226206_103_1_20251008T150916Z_gen_g4_IonScintPDExt.root_226551_15_1_20251009T040828Z_PDInt.root_229301_44_1_20251016T150913Z_detsim_reco1.root"
21-Oct-2025 10:55:48 CDT  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/fc/c3/pdhd_prod_beam__226206_103_1_20251008T150916Z_gen_g4_IonScintPDExt.root_226551_15_1_20251009T040828Z_PDInt.root_229301_44_1_20251016T150913Z_detsim_reco1.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0237709      22.1117       51.1622       15.1491       21.4498         3     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)             0.001044      0.0252706     0.0509968     0.0237709     0.0204207        3     
end_path:pduneana:PDSPAnalyzer      15.1478       33.1286       51.1095       33.1286       17.9809         2     
====================================================================================================================

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0237709      22.1117       51.1622       15.1491       21.4498         3     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)             0.001044      0.0252706     0.0509968     0.0237709     0.0204207        3     
end_path:pduneana:PDSPAnalyzer      15.1478       33.1286       51.1095       33.1286       17.9809         2     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3764.48 MB
  Peak resident set size usage (VmHWM): 2705.03 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3764.48 MB
  Peak resident set size usage (VmHWM): 2705.03 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 3 passed = 3 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          3          2          1 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 91.085904 Real = 100.825514

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3764.48 VmHWM = 2705.03

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
%MSG-s ArtException:  PostEndJob 21-Oct-2025 10:55:50 CDT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250627 subRun: 1 event: 52
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
%MSG-s ArtException:  PostEndJob 21-Oct-2025 10:55:50 CDT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250627 subRun: 1 event: 52
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.