Jobsub ID 228838.106@dunegpschedd02.fnal.gov

Jobsub ID	228838.106@dunegpschedd02.fnal.gov
Workflow ID	9021
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-15 16:11:35
Site	UK_Sheffield
Entry	DUNE_UK_Sheffield_lcgce2
Last heartbeat	2025-10-15 17:47:56
From worker node	Hostname	wn035.hep
	cpuinfo	AMD EPYC 7352 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-15 16:13:06
Input files	hd-protodune:np04hd_raw_run028891_0029_dataflow2_datawriter_0_20240823T155846.hdf5 hd-protodune:np04hd_raw_run028891_0032_dataflow2_datawriter_0_20240823T160249.hdf5 hd-protodune:np04hd_raw_run028891_0034_dataflow6_datawriter_0_20240823T160524.hdf5 hd-protodune:np04hd_raw_run028891_0037_dataflow4_datawriter_0_20240823T160910.hdf5 hd-protodune:np04hd_raw_run028891_0859_dataflow2_datawriter_0_20240824T130900.hdf5 hd-protodune:np04hd_raw_run028891_0877_dataflow7_datawriter_0_20240824T133424.hdf5 hd-protodune:np04hd_raw_run028891_0862_dataflow3_datawriter_0_20240824T131320.hdf5 hd-protodune:np04hd_raw_run028891_0872_dataflow4_datawriter_0_20240824T132709.hdf5 hd-protodune:np04hd_raw_run028891_0886_dataflow5_datawriter_0_20240824T134709.hdf5 hd-protodune:np04hd_raw_run028891_0874_dataflow0_datawriter_0_20240824T133009.hdf5 hd-protodune:np04hd_raw_run028891_0875_dataflow6_datawriter_0_20240824T133145.hdf5
Jobscript	Exit code	0
	Real time	1h (5648s)
	CPU time	1m (82s = 1%)
	Max RSS bytes	962564096 (917 MiB)
Outputting started	2025-10-15 17:47:15
Output files
Finished	2025-10-15 17:47:56
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

RNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 18:47:11 BST  run: 28891 subRun: 1 event: 217238 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 18:47:11 BST  run: 28891 subRun: 1 event: 217238 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: -1 -1
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 18:47:11 BST  run: 28891 subRun: 1 event: 217238 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 217246 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 267.386 5.11269
15-Oct-2025 18:47:14 BST  Closed output file "pdhd_228838_106_1_20251015T161318Z_beam.root"
15-Oct-2025 18:47:14 BST  Closed output file "pdhd_228838_106_1_20251015T161318Z_beam.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.960784       15.9425       1683.9        1.24891       128.639        341    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.4127e-05    1.95245e-05   0.000124467   1.8405e-05    7.45281e-06      341    
produce:triggerrawdecoder:PDHDTriggerReader3      0.687683       1.05286       2.55617       0.84205      0.413185        341    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.123924      0.140715      0.233269      0.129735      0.0194566       341    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0248701     0.0257485     0.082893      0.0251299    0.00367797       341    
produce:beamevent:BeamEvent                      0.000411785     14.6101       1682.83     0.00975238      128.652        341    
[art]:TriggerResults:TriggerResultInserter        9.558e-06    1.90724e-05   5.5616e-05    1.8185e-05    5.22756e-06      341    
end_path:out1:RootOutput                          2.835e-06    3.6776e-06    2.8795e-05     3.437e-06    2.10461e-06      341    
end_path:out1:RootOutput(write)                   0.0702799     0.112738      0.311852      0.110508      0.0221332       341    
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.960784       15.9425       1683.9        1.24891       128.639        341    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.4127e-05    1.95245e-05   0.000124467   1.8405e-05    7.45281e-06      341    
produce:triggerrawdecoder:PDHDTriggerReader3      0.687683       1.05286       2.55617       0.84205      0.413185        341    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.123924      0.140715      0.233269      0.129735      0.0194566       341    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0248701     0.0257485     0.082893      0.0251299    0.00367797       341    
produce:beamevent:BeamEvent                      0.000411785     14.6101       1682.83     0.00975238      128.652        341    
[art]:TriggerResults:TriggerResultInserter        9.558e-06    1.90724e-05   5.5616e-05    1.8185e-05    5.22756e-06      341    
end_path:out1:RootOutput                          2.835e-06    3.6776e-06    2.8795e-05     3.437e-06    2.10461e-06      341    
end_path:out1:RootOutput(write)                   0.0702799     0.112738      0.311852      0.110508      0.0221332       341    
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1851.08 MB
  Peak resident set size usage (VmHWM): 962.564 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1851.08 MB
  Peak resident set size usage (VmHWM): 962.564 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 341 passed = 341 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        341        341          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 74.895952 Real = 5632.404618

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1851.08 VmHWM = 962.564

Art has completed and will exit with status 0.
gen lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 149036
-rw-r--r--. 1 dune004 dune 11957845 Oct 15 18:47 jobscript.log
-rw-r--r--. 1 dune004 dune     1793 Oct 15 18:47 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 63276069 Oct 15 18:47 pdhd_228838_106_1_20251015T161318Z_beam_hist.root
-rw-r--r--. 1 dune004 dune 11705018 Oct 15 18:47 debug.log
-rw-r--r--. 1 dune004 dune 65025278 Oct 15 18:47 pdhd_228838_106_1_20251015T161318Z_beam.root
-rw-r--r--. 1 dune004 dune      913 Oct 15 17:13 all-input-dids.txt