justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 212346.0@dunegpschedd01.fnal.gov

Jobsub ID212346.0@dunegpschedd01.fnal.gov
Workflow ID8047
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-09-25 17:48:32
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp_multicore
Last heartbeat2025-09-26 12:04:55
From worker nodeHostnamewn-sate-020.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes7864320000 (7500 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-25 17:54:39
Input files
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-26 12:04:55
Saved logsjustin-logs:212346.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

13:20:28 CEST  run: 1 subRun: 0 event: 6
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:23:07 CEST  run: 1 subRun: 0 event: 7
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:26:06 CEST  run: 1 subRun: 0 event: 8
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:28:08 CEST  run: 1 subRun: 0 event: 9
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:30:01 CEST  run: 1 subRun: 0 event: 10
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
26-Sep-2025 13:31:18 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/cd/a1/prod_cosmics_radiologicals_protodunehd_1_205934_4_1758725473_reco_nest.root"
26-Sep-2025 13:31:18 CEST  Initiating request to open input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/af/57/prod_cosmics_radiologicals_protodunehd_1_205934_42_1758725468_reco_nest.root"
26-Sep-2025 13:31:20 CEST  Opened input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/af/57/prod_cosmics_radiologicals_protodunehd_1_205934_42_1758725468_reco_nest.root"
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:31:38 CEST  run: 1 subRun: 0 event: 1
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:32:50 CEST  run: 1 subRun: 0 event: 2
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:35:35 CEST  run: 1 subRun: 0 event: 3
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:36:27 CEST  run: 1 subRun: 0 event: 4
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:38:20 CEST  run: 1 subRun: 0 event: 5
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:39:54 CEST  run: 1 subRun: 0 event: 6
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:42:17 CEST  run: 1 subRun: 0 event: 7
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:43:30 CEST  run: 1 subRun: 0 event: 8
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:45:25 CEST  run: 1 subRun: 0 event: 9
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:46:25 CEST  run: 1 subRun: 0 event: 10
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
26-Sep-2025 13:48:20 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/af/57/prod_cosmics_radiologicals_protodunehd_1_205934_42_1758725468_reco_nest.root"
26-Sep-2025 13:48:20 CEST  Initiating request to open input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/df/27/prod_cosmics_radiologicals_protodunehd_1_205934_43_1758725435_reco_nest.root"
26-Sep-2025 13:48:21 CEST  Opened input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/df/27/prod_cosmics_radiologicals_protodunehd_1_205934_43_1758725435_reco_nest.root"
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:48:40 CEST  run: 1 subRun: 0 event: 1
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:50:13 CEST  run: 1 subRun: 0 event: 2
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:52:00 CEST  run: 1 subRun: 0 event: 3
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:53:32 CEST  run: 1 subRun: 0 event: 4
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:54:57 CEST  run: 1 subRun: 0 event: 5
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:56:15 CEST  run: 1 subRun: 0 event: 6
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 13:59:09 CEST  run: 1 subRun: 0 event: 7
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 14:00:44 CEST  run: 1 subRun: 0 event: 8
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 14:02:24 CEST  run: 1 subRun: 0 event: 9
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 26-Sep-2025 14:04:28 CEST  run: 1 subRun: 0 event: 10
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
26-Sep-2025 14:04:33 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/df/27/prod_cosmics_radiologicals_protodunehd_1_205934_43_1758725435_reco_nest.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          4682.58       5255.81       5866.95       5281.7        352.81         10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000401523    0.0228086     0.110966     0.00480747     0.0372306       610    
end_path:r1:SingleHit               7.87727       22.5702       123.318       20.7954       10.0063        609    
end_path:r2:SingleHit               3.98206       14.8631       112.436       13.0026       9.19316        609    
end_path:r3:SingleHit               4.16056       15.4061       112.28        13.5286       9.23428        609    
end_path:r4:SingleHit               4.49903       16.3189       112.537       14.3418       9.30079        609    
end_path:r5:SingleHit               4.9991        17.1192       113.38        15.0818       9.40698        609    
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9764.11 MB
  Peak resident set size usage (VmHWM): 7895.51 MB
  Details saved in: 'mem.db'
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 610 passed = 610 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        610        609          1 r1
TrigReport        609        609          0 r2
TrigReport        609        609          0 r3
TrigReport        609        609          0 r4
TrigReport        609        609          0 r5

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 54348.649431 Real = 65376.755679

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 9764.11 VmHWM = 7895.51

%MSG-s ArtException:  PostEndJob 26-Sep-2025 14:04:37 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: std::vector<recob::Hit>
          Module label: 'hitpdune'
          Product instance name: ''
          Process name: (empty)
        The above exception was thrown while processing module SingleHit/r1 run: 1 subRun: 0 event: 10
      ---- ProductNotFound END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
justIN time: 2025-11-05 10:33:02 UTC       justIN version: 01.05.01