Jobsub ID 228838.91@dunegpschedd02.fnal.gov

Jobsub ID	228838.91@dunegpschedd02.fnal.gov
Workflow ID	9021
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-15 16:11:35
Site	UK_Sheffield
Entry	DUNE_UK_Sheffield_lcgce2
Last heartbeat	2025-10-15 18:20:26
From worker node	Hostname	wn032.hep
	cpuinfo	Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-15 16:13:06
Input files	hd-protodune:np04hd_raw_run028891_0027_dataflow5_datawriter_0_20240823T155551.hdf5 hd-protodune:np04hd_raw_run028891_0030_dataflow0_datawriter_0_20240823T160010.hdf5 hd-protodune:np04hd_raw_run028891_0033_dataflow4_datawriter_0_20240823T160404.hdf5 hd-protodune:np04hd_raw_run028891_0036_dataflow2_datawriter_0_20240823T160800.hdf5 hd-protodune:np04hd_raw_run028891_0039_dataflow1_datawriter_0_20240823T161154.hdf5 hd-protodune:np04hd_raw_run028891_0861_dataflow0_datawriter_0_20240824T131149.hdf5 hd-protodune:np04hd_raw_run028891_0862_dataflow7_datawriter_0_20240824T131320.hdf5 hd-protodune:np04hd_raw_run028891_0878_dataflow4_datawriter_0_20240824T133538.hdf5 hd-protodune:np04hd_raw_run028891_0863_dataflow3_datawriter_0_20240824T131431.hdf5 hd-protodune:np04hd_raw_run028891_0866_dataflow4_datawriter_0_20240824T131852.hdf5 hd-protodune:np04hd_raw_run028891_0883_dataflow2_datawriter_0_20240824T134311.hdf5
Jobscript	Exit code	0
	Real time	2h (7620s)
	CPU time	1m (95s = 1%)
	Max RSS bytes	968142848 (923 MiB)
Outputting started	2025-10-15 18:20:07
Output files
Finished	2025-10-15 18:20:26
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

G: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 19:19:57 BST  run: 28891 subRun: 1 event: 219194 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 19:19:57 BST  run: 28891 subRun: 1 event: 219194 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 92.7346 4.77214
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 19:19:57 BST  run: 28891 subRun: 1 event: 219194 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 219202 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 97.2823 4.88889
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 19:19:59 BST  run: 28891 subRun: 1 event: 219202 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 219210 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 98.0409 4.80506
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 19:20:00 BST  run: 28891 subRun: 1 event: 219210 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 219218 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 98.1601 -1
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 19:20:02 BST  run: 28891 subRun: 1 event: 219218 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 219226 5
%MSG
Timing trigger: 8
Matched: 0
CKovs: 0 0
TOF, P: 0 -1
15-Oct-2025 19:20:05 BST  Closed output file "pdhd_228838_91_1_20251015T161317Z_beam.root"
15-Oct-2025 19:20:05 BST  Closed output file "pdhd_228838_91_1_20251015T161317Z_beam.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.967938       21.3373       1927.47       1.58404       157.994        341    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.3906e-05    2.16004e-05   0.000184201   2.0202e-05    9.68272e-06      341    
produce:triggerrawdecoder:PDHDTriggerReader3       0.69222       1.36286       4.16227       1.17766      0.690552        341    
produce:ctbrawdecoder:PDHDCTBRawDecoder            0.12402      0.147531      0.551959       0.13359      0.0448764       341    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.024945      0.0259728     0.0715498     0.0251423    0.00469373       341    
produce:beamevent:BeamEvent                      0.000500395     19.6711       1926.31      0.0144775      158.005        341    
[art]:TriggerResults:TriggerResultInserter       1.6945e-05    2.55593e-05   0.000131718   2.4386e-05    9.42345e-06      341    
end_path:out1:RootOutput                          2.718e-06    3.57402e-06    2.625e-05     3.432e-06    1.32879e-06      341    
end_path:out1:RootOutput(write)                   0.0641381      0.12941      0.201291      0.128888      0.0227812       341    
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.967938       21.3373       1927.47       1.58404       157.994        341    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.3906e-05    2.16004e-05   0.000184201   2.0202e-05    9.68272e-06      341    
produce:triggerrawdecoder:PDHDTriggerReader3       0.69222       1.36286       4.16227       1.17766      0.690552        341    
produce:ctbrawdecoder:PDHDCTBRawDecoder            0.12402      0.147531      0.551959       0.13359      0.0448764       341    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.024945      0.0259728     0.0715498     0.0251423    0.00469373       341    
produce:beamevent:BeamEvent                      0.000500395     19.6711       1926.31      0.0144775      158.005        341    
[art]:TriggerResults:TriggerResultInserter       1.6945e-05    2.55593e-05   0.000131718   2.4386e-05    9.42345e-06      341    
end_path:out1:RootOutput                          2.718e-06    3.57402e-06    2.625e-05     3.432e-06    1.32879e-06      341    
end_path:out1:RootOutput(write)                   0.0641381      0.12941      0.201291      0.128888      0.0227812       341    
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1820.93 MB
  Peak resident set size usage (VmHWM): 968.143 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1820.93 MB
  Peak resident set size usage (VmHWM): 968.143 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 341 passed = 341 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        341        341          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 88.736242 Real = 7603.768480

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1820.93 VmHWM = 968.143

Art has completed and will exit with status 0.
gen lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 151172
-rw-r--r--. 1 dune004 dune 12160377 Oct 15 19:20 jobscript.log
-rw-r--r--. 1 dune004 dune     1793 Oct 15 19:20 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 64158524 Oct 15 19:20 pdhd_228838_91_1_20251015T161317Z_beam_hist.root
-rw-r--r--. 1 dune004 dune 11895976 Oct 15 19:20 debug.log
-rw-r--r--. 1 dune004 dune 65357258 Oct 15 19:20 pdhd_228838_91_1_20251015T161317Z_beam.root
-rw-r--r--. 1 dune004 dune      913 Oct 15 17:13 all-input-dids.txt