Jobsub ID 228838.71@dunegpschedd02.fnal.gov

Jobsub ID	228838.71@dunegpschedd02.fnal.gov
Workflow ID	9021
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-15 16:11:35
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2025-10-15 16:54:22
From worker node	Hostname	comp23-04
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-15 16:13:08
Input files	hd-protodune:np04hd_raw_run028891_0039_dataflow0_datawriter_0_20240823T161155.hdf5 hd-protodune:np04hd_raw_run028891_0860_dataflow1_datawriter_0_20240824T131021.hdf5 hd-protodune:np04hd_raw_run028891_0730_dataflow0_datawriter_0_20240824T100017.hdf5 hd-protodune:np04hd_raw_run028891_0863_dataflow0_datawriter_0_20240824T131431.hdf5 hd-protodune:np04hd_raw_run028891_0880_dataflow0_datawriter_0_20240824T133835.hdf5 hd-protodune:np04hd_raw_run028891_0864_dataflow4_datawriter_0_20240824T131613.hdf5 hd-protodune:np04hd_raw_run028891_0888_dataflow7_datawriter_0_20240824T135009.hdf5 hd-protodune:np04hd_raw_run028891_0886_dataflow6_datawriter_0_20240824T134709.hdf5 hd-protodune:np04hd_raw_run028891_0874_dataflow3_datawriter_0_20240824T133009.hdf5
Jobscript	Exit code	0
	Real time	40m (2455s)
	CPU time	1m (60s = 2%)
	Max RSS bytes	900964352 (859 MiB)
Outputting started	2025-10-15 16:54:04
Output files
Finished	2025-10-15 16:54:22
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

RO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 17:53:56 BST  run: 28891 subRun: 1 event: 216979 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 17:53:56 BST  run: 28891 subRun: 1 event: 216979 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 99.0136 4.467
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 17:53:56 BST  run: 28891 subRun: 1 event: 216979 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 216987 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 95.0061 4.57434
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 17:53:59 BST  run: 28891 subRun: 1 event: 216987 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 216995 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: -1 5.49279
15-Oct-2025 17:54:02 BST  Closed output file "pdhd_228838_71_1_20251015T161318Z_beam.root"
15-Oct-2025 17:54:02 BST  Closed output file "pdhd_228838_71_1_20251015T161318Z_beam.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.952452       8.16225       903.848       1.64077       67.0788        279    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.9704e-05    3.10893e-05   0.000103946   3.0306e-05    5.51192e-06      279    
produce:triggerrawdecoder:PDHDTriggerReader3      0.678362       1.37792       3.69733       1.3298       0.585877        279    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.122833      0.143474      0.417592      0.129435      0.034671        279    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0246365     0.0261942     0.059206      0.0250545    0.00516776       279    
produce:beamevent:BeamEvent                      0.000641423     6.52569       902.777      0.0181146      67.1141        279    
[art]:TriggerResults:TriggerResultInserter       1.2582e-05    2.00837e-05   0.000112608   1.8058e-05    7.78212e-06      279    
end_path:out1:RootOutput                          2.223e-06    3.53639e-06   2.7588e-05     3.187e-06    1.7003e-06       279    
end_path:out1:RootOutput(write)                   0.0414483     0.0885621     0.248413      0.0879134     0.0187886       279    
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.952452       8.16225       903.848       1.64077       67.0788        279    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.9704e-05    3.10893e-05   0.000103946   3.0306e-05    5.51192e-06      279    
produce:triggerrawdecoder:PDHDTriggerReader3      0.678362       1.37792       3.69733       1.3298       0.585877        279    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.122833      0.143474      0.417592      0.129435      0.034671        279    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0246365     0.0261942     0.059206      0.0250545    0.00516776       279    
produce:beamevent:BeamEvent                      0.000641423     6.52569       902.777      0.0181146      67.1141        279    
[art]:TriggerResults:TriggerResultInserter       1.2582e-05    2.00837e-05   0.000112608   1.8058e-05    7.78212e-06      279    
end_path:out1:RootOutput                          2.223e-06    3.53639e-06   2.7588e-05     3.187e-06    1.7003e-06       279    
end_path:out1:RootOutput(write)                   0.0414483     0.0885621     0.248413      0.0879134     0.0187886       279    
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1758.36 MB
  Peak resident set size usage (VmHWM): 900.841 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1758.36 MB
  Peak resident set size usage (VmHWM): 900.841 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 279 passed = 279 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        279        279          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 55.233521 Real = 2441.201871

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1758.36 VmHWM = 900.903

Art has completed and will exit with status 0.
gen lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 113778
-rw-r--r-- 1 pltdune004 pltdune 10198265 Oct 15 17:54 jobscript.log
-rw-r--r-- 1 pltdune004 pltdune     1467 Oct 15 17:54 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune004 pltdune 43131786 Oct 15 17:54 pdhd_228838_71_1_20251015T161318Z_beam_hist.root
-rw-r--r-- 1 pltdune004 pltdune  9981314 Oct 15 17:54 debug.log
-rw-r--r-- 1 pltdune004 pltdune 53193631 Oct 15 17:54 pdhd_228838_71_1_20251015T161318Z_beam.root
-rw-r--r-- 1 pltdune004 pltdune      747 Oct 15 17:13 all-input-dids.txt