justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 250891.3@dunegpschedd01.fnal.gov

Jobsub ID250891.3@dunegpschedd01.fnal.gov
Workflow ID10203
Stage ID1
User namecalcuttj@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-14 22:19:59
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-11-14 22:34:22
From worker nodeHostnamedunegli-7757240-0-fnpc23038.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-14 22:21:50
Input fileshd-protodune:pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_248431_36_1_20251113T033923Z_detsim_reco1.root
hd-protodune:pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_248431_128_1_20251113T034000Z_detsim_reco1.root
hd-protodune:pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_242669_78_1_20251113T040458Z_detsim_reco1.root
hd-protodune:pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_248395_67_1_20251113T033123Z_detsim_reco1.root
hd-protodune:pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_242659_87_1_20251113T033940Z_detsim_reco1.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-14 22:34:22
Saved logsjustin-logs:250891.3-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

ix
 MINUIT WARNING IN MIGRAD  
 ============== 3.19094e+06 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -4.1711e+01  1.0000e+00  1.0003e+00  4.3711e+01
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 41.755134 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -9.3936e-06  4.5811e-06  3.6197e-01  3.6380e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.003647 TO DIAGONAL.
Getting reco beam calo
No track calorimetry object found... returning empty vector
No shower calorimetry object found... returning empty vector
No track calorimetry object found... returning empty vector
%MSG-w PhotonBackTrackerService:  ProcessEvent 14-Nov-2025 22:27:46 UTC  run: 20250623 subRun: 1 event: 102
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.102
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 14-Nov-2025 22:27:50 UTC  run: 20250623 subRun: 1 event: 103
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.103
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 14-Nov-2025 22:27:56 UTC  run: 20250623 subRun: 1 event: 104
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.104
We found no beam particles for this event... moving on
%MSG-w PhotonBackTrackerService:  ProcessEvent 14-Nov-2025 22:28:03 UTC  run: 20250623 subRun: 1 event: 105
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.105
105
2
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -1.6293e-04  1.6491e-05  1.3312e+00  2.6689e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002832 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 4.52496e+13 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.3789e+05  1.0000e+00  1.0055e+00  7.3789e+05
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 738626.828948 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -6.4978e-08  4.6751e-06  4.3013e-02  3.9570e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.003957 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.6317e-04  1.0210e-05  1.3321e+00  2.6682e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.002931 TO DIAGONAL.
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -7.2741e-09  3.9323e-06  3.9339e-02  3.9607e+00
 MINUIT WARNING IN HESSE   
 ============== MATRIX FORCED POS-DEF BY ADDING 0.003961 TO DIAGONAL.
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 1 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 4 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 8.14781e+08 added to diagonal of error matrix
 EIGENVALUES OF SECOND-DERIVATIVE MATRIX:
        -2.9666e+03  1.0000e+00  1.0014e+00  2.9686e+03
 MINUIT WARNING IN MIGRAD  
 ============== MATRIX FORCED POS-DEF BY ADDING 2969.571771 TO DIAGONAL.
Getting reco beam calo
No track calorimetry object found... returning empty vector
%MSG-w PhotonBackTrackerService:  ProcessEvent 14-Nov-2025 22:28:13 UTC  run: 20250623 subRun: 1 event: 106
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
########## EvtNo.106
14-Nov-2025 22:28:13 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/be/2f/pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_248395_67_1_20251113T033123Z_detsim_reco1.root"
14-Nov-2025 22:28:13 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/be/2f/pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_248395_67_1_20251113T033123Z_detsim_reco1.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                        0.00134172      6.06216       12.2772       5.49869       3.19701        37     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000414488   0.00339517     0.0226714    0.00115904    0.00600282       37     
end_path:pduneana:PDSPAnalyzer      1.32433       6.22695       12.2677       5.50629       3.07525        36     
====================================================================================================================

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                        0.00134172      6.06216       12.2772       5.49869       3.19701        37     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000414488   0.00339517     0.0226714    0.00115904    0.00600282       37     
end_path:pduneana:PDSPAnalyzer      1.32433       6.22695       12.2677       5.50629       3.07525        36     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4946.12 MB
  Peak resident set size usage (VmHWM): 3899.83 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4946.12 MB
  Peak resident set size usage (VmHWM): 3899.83 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 37 passed = 37 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         37         36          1 pduneana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 320.231140 Real = 337.924155

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4946.12 VmHWM = 3899.83

ToolBasedRawDigitPrepService:dtor: Event count: 0
ToolBasedRawDigitPrepService:dtor:  Call count: 0
ToolBasedRawDigitPrepService:dtor: Time report for 4 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.00 sec
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   0.00 sec
%MSG-s ArtException:  PostEndJob 14-Nov-2025 22:28:13 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250623 subRun: 1 event: 106
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
%MSG-s ArtException:  PostEndJob 14-Nov-2025 22:28:13 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::SpacePoint>
          Module label: 'reco3d'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module PDSPAnalyzer/pduneana run: 20250623 subRun: 1 event: 106
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
justIN time: 2026-02-09 13:54:53 UTC       justIN version: 01.06.00