justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 231460.0@dunegpschedd02.fnal.gov

Jobsub ID231460.0@dunegpschedd02.fnal.gov
Workflow ID9219
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes5242880000 (5000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-20 21:30:17
SiteUS_UChicago
EntryEngage_US_MWT2_iut2_condce_mcore
Last heartbeat2025-10-20 21:34:01
From worker nodeHostnameiut2-c375.iu.edu
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes5242880000 (5000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-20 21:31:45
Input filesusertests:pdhd_prod_beam__226002_22_1_20251007T163213Z_gen_g4_IonScintPDExt.root_226385_0_1_20251008T194733Z_PDInt.root_232179_78_1_20251009T181912Z_detsim_reco1.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-20 21:34:01
Saved logsjustin-logs:231460.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

PedDegree: 0
ExpTailPedRemover::ctor:               PedTick0: 3000
ExpTailPedRemover::ctor:               PedFreqs: []
ExpTailPedRemover::ctor:   IncludeChannelRanges: [apa1z, apa2z, apa3z, apa4z, apa5z, apa6z, apa1c, apa2c, apa3c, apa4c, apa5c, apa6c]
ExpTailPedRemover::ctor:   NoWarnStatuses: [1]
ExpTailPedRemover::ctor:   ExcludeChannelRanges: []
ExpTailPedRemover::ctor: Channel checking enabled for 5760 channels.
ToolBasedRawDigitPrepService::ctor:     Found tool pdspTailPedRemovalZKe @ 0xcbe70b0
ToolBasedRawDigitPrepService::ctor: Fetching wire building service.
ToolBasedRawDigitPrepService::ctor:   Wire building service: @0x438a2b0
ToolBasedRawDigitPrepService::ctor: ToolBasedRawDigitPrepService:
ToolBasedRawDigitPrepService::ctor:                     LogLevel: 3
ToolBasedRawDigitPrepService::ctor:                      DoWires: 1
ToolBasedRawDigitPrepService::ctor:      ADC channel tools:
ToolBasedRawDigitPrepService::ctor:            digitReader
ToolBasedRawDigitPrepService::ctor:            adcSampleFiller
ToolBasedRawDigitPrepService::ctor:            adcScaleAdcToKe
ToolBasedRawDigitPrepService::ctor:            pdspTailPedRemovalZKe
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
20-Oct-2025 17:32:51 EDT  Initiating request to open input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/b2/36/pdhd_prod_beam__226002_22_1_20251007T163213Z_gen_g4_IonScintPDExt.root_226385_0_1_20251008T194733Z_PDInt.root_232179_78_1_20251009T181912Z_detsim_reco1.root"
20-Oct-2025 17:32:51 EDT  Initiating request to open input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/b2/36/pdhd_prod_beam__226002_22_1_20251007T163213Z_gen_g4_IonScintPDExt.root_226385_0_1_20251008T194733Z_PDInt.root_232179_78_1_20251009T181912Z_detsim_reco1.root"
20-Oct-2025 17:32:54 EDT  Opened input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/b2/36/pdhd_prod_beam__226002_22_1_20251007T163213Z_gen_g4_IonScintPDExt.root_226385_0_1_20251008T194733Z_PDInt.root_232179_78_1_20251009T181912Z_detsim_reco1.root"
20-Oct-2025 17:32:54 EDT  Opened input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/b2/36/pdhd_prod_beam__226002_22_1_20251007T163213Z_gen_g4_IonScintPDExt.root_226385_0_1_20251008T194733Z_PDInt.root_232179_78_1_20251009T181912Z_detsim_reco1.root"
Setting applicationFamily: "art"
Setting process_name: "PDHDAna"
db: runtime0
%MSG-w PhotonBackTrackerService:  ProcessEvent 20-Oct-2025 17:33:04 EDT  run: 20250627 subRun: 1 event: 10
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.10
No track calorimetry object found... returning empty vector
No shower calorimetry object found... returning empty vector
No track calorimetry object found... returning empty vector
No shower calorimetry object found... returning empty vector
No track calorimetry object found... returning empty vector
No shower calorimetry object found... returning empty vector
No track calorimetry object found... returning empty vector
%MSG-w PhotonBackTrackerService:  ProcessEvent 20-Oct-2025 17:33:20 EDT  run: 20250627 subRun: 1 event: 11
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.11
4
2
Getting reco beam calo
No track calorimetry object found... returning empty vector
No shower calorimetry object found... returning empty vector
No track calorimetry object found... returning empty vector
%MSG-w PhotonBackTrackerService:  ProcessEvent 20-Oct-2025 17:33:44 EDT  run: 20250627 subRun: 1 event: 12
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.12
20-Oct-2025 17:33:44 EDT  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/b2/36/pdhd_prod_beam__226002_22_1_20251007T163213Z_gen_g4_IonScintPDExt.root_226385_0_1_20251008T194733Z_PDInt.root_232179_78_1_20251009T181912Z_detsim_reco1.root"
20-Oct-2025 17:33:44 EDT  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/b2/36/pdhd_prod_beam__226002_22_1_20251007T163213Z_gen_g4_IonScintPDExt.root_226385_0_1_20251008T194733Z_PDInt.root_232179_78_1_20251009T181912Z_detsim_reco1.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                        0.00654954      6.0283        13.7494       4.32892       5.73775         3     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000603706   0.00527487    0.00867137    0.00654954    0.00341471        3     
end_path:pduneana:PDSPAnalyzer      4.31949       9.03405       13.7486       9.03405       4.71456         2     
====================================================================================================================

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                        0.00654954      6.0283        13.7494       4.32892       5.73775         3     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000603706   0.00527487    0.00867137    0.00654954    0.00341471        3     
end_path:pduneana:PDSPAnalyzer      4.31949       9.03405       13.7486       9.03405       4.71456         2     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4567.58 MB
  Peak resident set size usage (VmHWM): 3531.13 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4567.58 MB
  Peak resident set size usage (VmHWM): 3531.13 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 3 passed = 3 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          3          2          1 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 45.319940 Real = 48.508316

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4567.58 VmHWM = 3531.13

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
%MSG-s ArtException:  PostEndJob 20-Oct-2025 17:33:45 EDT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250627 subRun: 1 event: 12
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
%MSG-s ArtException:  PostEndJob 20-Oct-2025 17:33:45 EDT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250627 subRun: 1 event: 12
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
justIN time: 2025-11-05 05:21:31 UTC       justIN version: 01.05.01