Jobsub ID 269102.0@dunegpschedd01.fnal.gov

Jobsub ID	269102.0@dunegpschedd01.fnal.gov
Workflow ID	11255
Stage ID	1
User name	avizcaya@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	5242880000 (5000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-12-18 17:13:00
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw6
Last heartbeat	2025-12-18 17:26:20
From worker node	Hostname	mh-7763-2.t2.ucsd.edu
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	5242880000 (5000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-12-18 17:13:59
Input files	usertests:pdhd_prod_beam__259173_22_1_20251208T180908Z_gen_g4_IonScintPDExt.root_260261_66_1_20251210T184625Z_PDInt.root_262711_0_1_20251215T193705Z_detsim_reco1.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-12-18 17:26:20
Saved logs	justin-logs:269102.0-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

DDING 0.003543 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 5.80906e+13 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.2256e+06  1.0000e+00  1.0000e+00  1.2256e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1226872.130683 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.0992e-07  3.8269e-08  1.4817e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.4106e-08  3.6125e-08  1.4676e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004000 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 1153.19 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.6729e+00  1.0000e+00  1.0000e+00  3.6729e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 1.676585 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.0840e-04 -6.3978e-06  1.9770e+00  2.0231e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002132 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.0563e-06  8.9529e-07  4.7162e-01  3.5284e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.003529 TO DIAGONAL.
Getting reco beam calo
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 09:19:53 PST run: 20250627 subRun: 1 event: 146 PDSPAnalyzer_module.cc:4729
pandora2Track object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 09:19:53 PST run: 20250627 subRun: 1 event: 146 PDSPAnalyzer_module.cc:4850
pandora2Shower object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 09:19:54 PST run: 20250627 subRun: 1 event: 146 PDSPAnalyzer_module.cc:4729
pandora2Track object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 09:19:54 PST run: 20250627 subRun: 1 event: 146 PDSPAnalyzer_module.cc:4850
pandora2Shower object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 09:19:54 PST run: 20250627 subRun: 1 event: 146 PDSPAnalyzer_module.cc:4729
pandora2Track object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 09:19:54 PST run: 20250627 subRun: 1 event: 146 PDSPAnalyzer_module.cc:4850
pandora2Shower object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 09:19:54 PST run: 20250627 subRun: 1 event: 146 PDSPAnalyzer_module.cc:4729
pandora2Track object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 09:19:54 PST run: 20250627 subRun: 1 event: 146 PDSPAnalyzer_module.cc:4850
pandora2Shower object not found, moving on
%MSG
%MSG-w PDSPAnalyzer:  PDSPAnalyzer:pduneana@BeginModule  18-Dec-2025 09:19:54 PST run: 20250627 subRun: 1 event: 146 PDSPAnalyzer_module.cc:5239
beam pandora2Track object not found, moving on
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 18-Dec-2025 09:21:24 PST  run: 20250627 subRun: 1 event: 147
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.147
18-Dec-2025 09:21:24 PST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/49/dc/pdhd_prod_beam__259173_22_1_20251208T180908Z_gen_g4_IonScintPDExt.root_260261_66_1_20251210T184625Z_PDInt.root_262711_0_1_20251215T193705Z_detsim_reco1.root"
18-Dec-2025 09:21:24 PST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/49/dc/pdhd_prod_beam__259173_22_1_20251208T180908Z_gen_g4_IonScintPDExt.root_260261_66_1_20251210T184625Z_PDInt.root_262711_0_1_20251215T193705Z_detsim_reco1.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                        0.000565921     6.86638       11.1423       6.30226       3.48484         8     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000537981    0.0255819     0.0676353    0.000611646    0.0322862        8     
end_path:pduneana:PDSPAnalyzer      5.18001       7.81777       11.0748       6.58592       2.48602         7     
====================================================================================================================

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                        0.000565921     6.86638       11.1423       6.30226       3.48484         8     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000537981    0.0255819     0.0676353    0.000611646    0.0322862        8     
end_path:pduneana:PDSPAnalyzer      5.18001       7.81777       11.0748       6.58592       2.48602         7     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4581.4 MB
  Peak resident set size usage (VmHWM): 3518.63 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4581.4 MB
  Peak resident set size usage (VmHWM): 3518.63 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 8 passed = 8 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          8          7          1 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 74.733084 Real = 394.082297

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4581.4 VmHWM = 3518.63

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
%MSG-s ArtException:  PostEndJob 18-Dec-2025 09:21:25 PST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250627 subRun: 1 event: 147
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
%MSG-s ArtException:  PostEndJob 18-Dec-2025 09:21:25 PST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250627 subRun: 1 event: 147
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.