Jobsub ID 213472.0@dunegpschedd01.fnal.gov
| Jobsub ID | 213472.0@dunegpschedd01.fnal.gov |
| Workflow ID | 8100 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 6815744000 (6500 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-09-27 05:47:26 |
| Site | US_FNAL-FermiGrid |
| Entry | FNAL_GPGrid_ce03_mcore_op_duneonly |
| Last heartbeat | 2025-09-27 19:55:27 |
| From worker node | Hostname | dunegli-7032722-0-fnpc9067.fnal.gov |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 7864320000 (7500 MiB) |
| Wall seconds limit | 172800 (48 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-09-27 05:49:30 |
| Input files | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-09-27 19:55:27 |
| Saved logs | justin-logs:213472.0-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
rService: ProcessEvent 27-Sep-2025 19:17:34 UTC run: 1 subRun: 0 event: 6
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:19:46 UTC run: 1 subRun: 0 event: 7
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:22:23 UTC run: 1 subRun: 0 event: 8
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:24:06 UTC run: 1 subRun: 0 event: 9
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:25:44 UTC run: 1 subRun: 0 event: 10
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
27-Sep-2025 19:26:51 UTC Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/cd/a1/prod_cosmics_radiologicals_protodunehd_1_205934_4_1758725473_reco_nest.root"
27-Sep-2025 19:26:51 UTC Initiating request to open input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/af/57/prod_cosmics_radiologicals_protodunehd_1_205934_42_1758725468_reco_nest.root"
27-Sep-2025 19:26:51 UTC Opened input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/af/57/prod_cosmics_radiologicals_protodunehd_1_205934_42_1758725468_reco_nest.root"
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:26:55 UTC run: 1 subRun: 0 event: 1
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:27:51 UTC run: 1 subRun: 0 event: 2
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:30:26 UTC run: 1 subRun: 0 event: 3
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:31:09 UTC run: 1 subRun: 0 event: 4
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:32:53 UTC run: 1 subRun: 0 event: 5
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:34:10 UTC run: 1 subRun: 0 event: 6
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:36:11 UTC run: 1 subRun: 0 event: 7
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:37:14 UTC run: 1 subRun: 0 event: 8
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:38:53 UTC run: 1 subRun: 0 event: 9
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:39:42 UTC run: 1 subRun: 0 event: 10
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
27-Sep-2025 19:41:20 UTC Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/af/57/prod_cosmics_radiologicals_protodunehd_1_205934_42_1758725468_reco_nest.root"
27-Sep-2025 19:41:20 UTC Initiating request to open input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/df/27/prod_cosmics_radiologicals_protodunehd_1_205934_43_1758725435_reco_nest.root"
27-Sep-2025 19:41:20 UTC Opened input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/df/27/prod_cosmics_radiologicals_protodunehd_1_205934_43_1758725435_reco_nest.root"
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:41:25 UTC run: 1 subRun: 0 event: 1
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:42:42 UTC run: 1 subRun: 0 event: 2
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:44:13 UTC run: 1 subRun: 0 event: 3
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:45:34 UTC run: 1 subRun: 0 event: 4
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:46:48 UTC run: 1 subRun: 0 event: 5
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:47:56 UTC run: 1 subRun: 0 event: 6
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:50:27 UTC run: 1 subRun: 0 event: 7
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:51:49 UTC run: 1 subRun: 0 event: 8
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:53:15 UTC run: 1 subRun: 0 event: 9
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 27-Sep-2025 19:54:56 UTC run: 1 subRun: 0 event: 10
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
27-Sep-2025 19:55:00 UTC Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/df/27/prod_cosmics_radiologicals_protodunehd_1_205934_43_1758725435_reco_nest.root"
====================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
====================================================================================================================
Full event 4234.37 4759.64 5331.14 4814.62 355.951 10
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000375749 0.00430783 0.206732 0.00406276 0.0086231 610
end_path:r1:SingleHit 6.4844 18.627 96.1075 16.822 8.53781 609
end_path:r2:SingleHit 3.96186 13.9728 91.2993 12.0683 8.24956 609
end_path:r3:SingleHit 4.03028 14.397 92.5459 12.5409 8.25812 609
end_path:r4:SingleHit 4.49318 15.2172 92.7816 13.3293 8.3337 609
end_path:r5:SingleHit 4.78396 15.9001 93.6128 13.9648 8.40379 609
====================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 9763.55 MB
Peak resident set size usage (VmHWM): 8148.8 MB
Details saved in: 'mem.db'
====================================================================================================
TrigReport ---------- Event summary -------------
TrigReport Events total = 610 passed = 610 failed = 0
TrigReport ---------- Modules in End-path ----------
TrigReport Run Success Error Name
TrigReport 610 609 1 r1
TrigReport 609 609 0 r2
TrigReport 609 609 0 r3
TrigReport 609 609 0 r4
TrigReport 609 609 0 r5
TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 50496.928010 Real = 50708.816293
MemReport ---------- Memory summary [base-10 MB] ------
MemReport VmPeak = 9763.55 VmHWM = 8148.8
%MSG-s ArtException: PostEndJob 27-Sep-2025 19:55:17 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- EventProcessorFailure BEGIN
EndPathExecutor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- ProductNotFound BEGIN
Found zero products matching all selection criteria
C++ type: std::vector<recob::Hit>
Module label: 'hitpdune'
Product instance name: ''
Process name: (empty)
The above exception was thrown while processing module SingleHit/r1 run: 1 subRun: 0 event: 10
---- ProductNotFound END
Exception going through path end_path
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1. 