Jobsub ID 248252.1@dunegpschedd01.fnal.gov

Jobsub ID	248252.1@dunegpschedd01.fnal.gov
Workflow ID	10080
Stage ID	1
User name	mwrobel@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	5242880000 (5000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-12 23:11:39
Site	CA_SFU
Entry	DUNE_CA_SFU_lcg-ce3
Last heartbeat	2025-11-12 23:19:16
From worker node	Hostname	fc30563.int.fir.alliancecan.ca
	cpuinfo	AMD EPYC 9655 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	5242880000 (5000 MiB)
	Wall seconds limit	84598 (23 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-11-12 23:12:27
Input files	usertests:pdhd_prod_beam__231691_38_1_20251008T183100Z_gen_g4_IonScintPDExt.root_227038_39_1_20251010T161153Z_PDInt.root_240525_31_1_20251029T204722Z_detsim_reco1.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-11-12 23:19:16
Saved logs	justin-logs:248252.1-dunegpschedd01.fnal.gov.logs.tgz
List job events (HTCondor job logs unavailable)
Jobscript log (last 10,000 characters)

======== 22413.9 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.0259e+01  1.0000e+00  1.0001e+00  1.2258e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 10.270767 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 199.036 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.7617e+00  1.0005e+00  1.0030e+00  9.7582e+00
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 7.771498 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 6.61915e+14 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.3574e+06  1.0000e+00  1.0024e+00  2.3574e+06
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2359737.757205 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.4796e-06 -4.5504e-07  1.3145e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.4501e-06 -3.1486e-07  1.2661e-06  4.0000e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.004001 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 28934.1 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.8559e+01  1.0000e+00  1.0001e+00  3.0558e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 28.589070 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -6.4817e-06  3.1924e-07  4.4763e-01  3.5524e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.003559 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 2.36815e+06 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.2733e+02  1.0000e+00  1.0001e+00  7.2933e+02
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 728.062553 TO DIAGONAL.
Getting reco beam calo
No track calorimetry object found... returning empty vector
No shower calorimetry object found... returning empty vector
No track calorimetry object found... returning empty vector
No shower calorimetry object found... returning empty vector
No track calorimetry object found... returning empty vector
No shower calorimetry object found... returning empty vector
No track calorimetry object found... returning empty vector
%MSG-w PhotonBackTrackerService:  ProcessEvent 12-Nov-2025 15:15:16 PST  run: 20250627 subRun: 1 event: 89
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.89
12-Nov-2025 15:15:17 PST  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/7c/a0/pdhd_prod_beam__231691_38_1_20251008T183100Z_gen_g4_IonScintPDExt.root_227038_39_1_20251010T161153Z_PDInt.root_240525_31_1_20251029T204722Z_detsim_reco1.root"
12-Nov-2025 15:15:17 PST  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/7c/a0/pdhd_prod_beam__231691_38_1_20251008T183100Z_gen_g4_IonScintPDExt.root_227038_39_1_20251010T161153Z_PDInt.root_240525_31_1_20251029T204722Z_detsim_reco1.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                        0.000478769     8.59111       18.2844       8.74121       4.72955        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000434499    0.0273773     0.0538602     0.0286748     0.026231        10     
end_path:pduneana:PDSPAnalyzer      4.41467       9.51505        18.23        8.77162       3.96642         9     
====================================================================================================================

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                        0.000478769     8.59111       18.2844       8.74121       4.72955        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000434499    0.0273773     0.0538602     0.0286748     0.026231        10     
end_path:pduneana:PDSPAnalyzer      4.41467       9.51505        18.23        8.77162       3.96642         9     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4004.96 MB
  Peak resident set size usage (VmHWM): 2967.85 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4004.96 MB
  Peak resident set size usage (VmHWM): 2967.85 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10          9          1 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 95.840466 Real = 138.641488

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4004.96 VmHWM = 2967.85

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
%MSG-s ArtException:  PostEndJob 12-Nov-2025 15:15:17 PST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250627 subRun: 1 event: 89
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
%MSG-s ArtException:  PostEndJob 12-Nov-2025 15:15:17 PST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250627 subRun: 1 event: 89
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.