Jobsub ID 228838.49@dunegpschedd02.fnal.gov

Jobsub ID	228838.49@dunegpschedd02.fnal.gov
Workflow ID	9021
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-15 16:11:35
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce03
Last heartbeat	2025-10-15 17:35:22
From worker node	Hostname	dune004-9145974.0-lcg2705.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-15 16:13:04
Input files	hd-protodune:np04hd_raw_run028891_0025_dataflow4_datawriter_0_20240823T155258.hdf5 hd-protodune:np04hd_raw_run028891_0029_dataflow4_datawriter_0_20240823T155846.hdf5 hd-protodune:np04hd_raw_run028891_0031_dataflow0_datawriter_0_20240823T160135.hdf5 hd-protodune:np04hd_raw_run028891_0035_dataflow2_datawriter_0_20240823T160637.hdf5 hd-protodune:np04hd_raw_run028891_0037_dataflow2_datawriter_0_20240823T160907.hdf5 hd-protodune:np04hd_raw_run028891_0040_dataflow6_datawriter_0_20240823T161322.hdf5 hd-protodune:np04hd_raw_run028891_0861_dataflow6_datawriter_0_20240824T131149.hdf5 hd-protodune:np04hd_raw_run028891_0871_dataflow2_datawriter_0_20240824T132544.hdf5 hd-protodune:np04hd_raw_run028891_0882_dataflow4_datawriter_0_20240824T134143.hdf5 hd-protodune:np04hd_raw_run028891_0881_dataflow1_datawriter_0_20240824T134014.hdf5 hd-protodune:np04hd_raw_run028891_0865_dataflow3_datawriter_0_20240824T131720.hdf5 hd-protodune:np04hd_raw_run028891_0884_dataflow4_datawriter_0_20240824T134427.hdf5
Jobscript	Exit code	0
	Real time	1h (4917s)
	CPU time	1m (87s = 1%)
	Max RSS bytes	657485824 (627 MiB)
Outputting started	2025-10-15 17:35:02
Output files
Finished	2025-10-15 17:35:22
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 17:34:56 UTC  run: 28891 subRun: 1 event: 219460 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 17:34:56 UTC  run: 28891 subRun: 1 event: 219460 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 95.9746 -1
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 17:34:56 UTC  run: 28891 subRun: 1 event: 219460 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 219468 5
%MSG
Timing trigger: 8
Matched: 0
CKovs: 1 0
TOF, P: 0 -1
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 17:34:57 UTC  run: 28891 subRun: 1 event: 219468 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 219476 5
%MSG
Timing trigger: 8
Matched: 0
CKovs: 1 0
TOF, P: 0 -1
15-Oct-2025 17:35:00 UTC  Closed output file "pdhd_228838_49_1_20251015T161318Z_beam.root"
15-Oct-2025 17:35:00 UTC  Closed output file "pdhd_228838_49_1_20251015T161318Z_beam.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.872344       12.6507       2392.87      0.970263       141.675        372    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.5253e-05    2.41028e-05   0.000130957   2.1783e-05    9.75548e-06      372    
produce:triggerrawdecoder:PDHDTriggerReader3      0.622979      0.692698       1.01622      0.684587      0.0411981       372    
produce:ctbrawdecoder:PDHDCTBRawDecoder            0.11376      0.126506      0.256606      0.115683      0.0171516       372    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0227959     0.0232832     0.0366943     0.0231851    0.000804786      372    
produce:beamevent:BeamEvent                      0.000382697     11.6933       2391.9      0.00825733      141.674        372    
[art]:TriggerResults:TriggerResultInserter       1.2068e-05    2.2221e-05     0.0003118    2.0165e-05    1.67126e-05      372    
end_path:out1:RootOutput                          2.614e-06    3.89899e-06   2.9545e-05    3.5355e-06    1.98341e-06      372    
end_path:out1:RootOutput(write)                   0.0620322     0.114541      0.182653      0.115612      0.0207216       372    
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.872344       12.6507       2392.87      0.970263       141.675        372    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.5253e-05    2.41028e-05   0.000130957   2.1783e-05    9.75548e-06      372    
produce:triggerrawdecoder:PDHDTriggerReader3      0.622979      0.692698       1.01622      0.684587      0.0411981       372    
produce:ctbrawdecoder:PDHDCTBRawDecoder            0.11376      0.126506      0.256606      0.115683      0.0171516       372    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0227959     0.0232832     0.0366943     0.0231851    0.000804786      372    
produce:beamevent:BeamEvent                      0.000382697     11.6933       2391.9      0.00825733      141.674        372    
[art]:TriggerResults:TriggerResultInserter       1.2068e-05    2.2221e-05     0.0003118    2.0165e-05    1.67126e-05      372    
end_path:out1:RootOutput                          2.614e-06    3.89899e-06   2.9545e-05    3.5355e-06    1.98341e-06      372    
end_path:out1:RootOutput(write)                   0.0620322     0.114541      0.182653      0.115612      0.0207216       372    
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1879.57 MB
  Peak resident set size usage (VmHWM): 967.541 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1879.57 MB
  Peak resident set size usage (VmHWM): 967.541 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 372 passed = 372 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        372        372          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 79.443700 Real = 4898.448894

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1879.57 VmHWM = 967.541

Art has completed and will exit with status 0.
gen lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 168708
-rw-r--r-- 1 dune004 dune 13174045 Oct 15 17:35 jobscript.log
-rw-r--r-- 1 dune004 dune     1956 Oct 15 17:35 justin-processed-pfns.txt
-rw-r--r-- 1 dune004 dune 74681102 Oct 15 17:35 pdhd_228838_49_1_20251015T161318Z_beam_hist.root
-rw-r--r-- 1 dune004 dune 12892809 Oct 15 17:35 debug.log
-rw-r--r-- 1 dune004 dune 71081808 Oct 15 17:35 pdhd_228838_49_1_20251015T161318Z_beam.root
-rw-r--r-- 1 dune004 dune      996 Oct 15 16:13 all-input-dids.txt