Jobsub ID 228838.59@dunegpschedd02.fnal.gov

Jobsub ID	228838.59@dunegpschedd02.fnal.gov
Workflow ID	9021
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-15 16:11:35
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce04
Last heartbeat	2025-10-15 18:32:14
From worker node	Hostname	dune004-8085185.0-lcg2648.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-15 16:12:58
Input files	hd-protodune:np04hd_raw_run028891_0013_dataflow2_datawriter_0_20240823T153442.hdf5 hd-protodune:np04hd_raw_run028891_0014_dataflow0_datawriter_0_20240823T153613.hdf5 hd-protodune:np04hd_raw_run028891_0017_dataflow3_datawriter_0_20240823T154031.hdf5 hd-protodune:np04hd_raw_run028891_0019_dataflow2_datawriter_0_20240823T154335.hdf5 hd-protodune:np04hd_raw_run028891_0022_dataflow7_datawriter_0_20240823T154810.hdf5 hd-protodune:np04hd_raw_run028891_0024_dataflow1_datawriter_0_20240823T155135.hdf5 hd-protodune:np04hd_raw_run028891_0034_dataflow5_datawriter_0_20240823T160524.hdf5 hd-protodune:np04hd_raw_run028891_0027_dataflow1_datawriter_0_20240823T155549.hdf5 hd-protodune:np04hd_raw_run028891_0031_dataflow5_datawriter_0_20240823T160135.hdf5 hd-protodune:np04hd_raw_run028891_0036_dataflow3_datawriter_0_20240823T160800.hdf5 hd-protodune:np04hd_raw_run028891_0038_dataflow7_datawriter_0_20240823T161036.hdf5 hd-protodune:np04hd_raw_run028891_0040_dataflow1_datawriter_0_20240823T161321.hdf5 hd-protodune:np04hd_raw_run028891_0861_dataflow4_datawriter_0_20240824T131149.hdf5 hd-protodune:np04hd_raw_run028891_0861_dataflow5_datawriter_0_20240824T131149.hdf5 hd-protodune:np04hd_raw_run028891_0881_dataflow5_datawriter_0_20240824T134016.hdf5 hd-protodune:np04hd_raw_run028891_0879_dataflow2_datawriter_0_20240824T133704.hdf5 hd-protodune:np04hd_raw_run028891_0886_dataflow4_datawriter_0_20240824T134709.hdf5 hd-protodune:np04hd_raw_run028891_0889_dataflow3_datawriter_0_20240824T135134.hdf5
Jobscript	Exit code	0
	Real time	2h (8328s)
	CPU time	2m (159s = 1%)
	Max RSS bytes	923258880 (880 MiB)
Outputting started	2025-10-15 18:31:48
Output files
Finished	2025-10-15 18:32:14
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 18:31:40 UTC  run: 28891 subRun: 1 event: 220707 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 18:31:40 UTC  run: 28891 subRun: 1 event: 220707 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 95.501 4.51332
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 18:31:42 UTC  run: 28891 subRun: 1 event: 220707 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 220715 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 94.8379 5.17757
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 18:31:44 UTC  run: 28891 subRun: 1 event: 220715 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 220723 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: -1 5.5919
15-Oct-2025 18:31:47 UTC  Closed output file "pdhd_228838_59_1_20251015T161318Z_beam.root"
15-Oct-2025 18:31:47 UTC  Closed output file "pdhd_228838_59_1_20251015T161318Z_beam.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.891468       14.354        3487.03       1.0469         162.7         558    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.7583e-05    2.63683e-05   0.000233841   2.3019e-05    1.4191e-05       558    
produce:triggerrawdecoder:PDHDTriggerReader3       0.62671      0.729058       1.44571      0.714911      0.0674362       558    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.113561      0.130378       0.23695      0.121143      0.0186294       558    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0227281      0.02337      0.0393377      0.02319     0.00105402       558    
produce:beamevent:BeamEvent                      0.000367684     13.3318       3485.85     0.00742231      162.691        558    
[art]:TriggerResults:TriggerResultInserter       1.4307e-05    2.86837e-05   0.000391359   2.4887e-05    2.08484e-05      558    
end_path:out1:RootOutput                          3.016e-06    4.62521e-06   4.3532e-05    4.4285e-06    1.86848e-06      558    
end_path:out1:RootOutput(write)                   0.0656548     0.138893       0.35124      0.134104      0.034594        558    
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.891468       14.354        3487.03       1.0469         162.7         558    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.7583e-05    2.63683e-05   0.000233841   2.3019e-05    1.4191e-05       558    
produce:triggerrawdecoder:PDHDTriggerReader3       0.62671      0.729058       1.44571      0.714911      0.0674362       558    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.113561      0.130378       0.23695      0.121143      0.0186294       558    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0227281      0.02337      0.0393377      0.02319     0.00105402       558    
produce:beamevent:BeamEvent                      0.000367684     13.3318       3485.85     0.00742231      162.691        558    
[art]:TriggerResults:TriggerResultInserter       1.4307e-05    2.86837e-05   0.000391359   2.4887e-05    2.08484e-05      558    
end_path:out1:RootOutput                          3.016e-06    4.62521e-06   4.3532e-05    4.4285e-06    1.86848e-06      558    
end_path:out1:RootOutput(write)                   0.0656548     0.138893       0.35124      0.134104      0.034594        558    
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1935.12 MB
  Peak resident set size usage (VmHWM): 970.985 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1935.12 MB
  Peak resident set size usage (VmHWM): 970.985 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 558 passed = 558 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        558        558          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 148.771122 Real = 8304.325718

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1935.12 VmHWM = 970.985

Art has completed and will exit with status 0.
gen lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 307752
-rw-r--r-- 1 dune004 dune  19931216 Oct 15 18:31 jobscript.log
-rw-r--r-- 1 dune004 dune      2934 Oct 15 18:31 justin-processed-pfns.txt
-rw-r--r-- 1 dune004 dune 165546343 Oct 15 18:31 pdhd_228838_59_1_20251015T161318Z_beam_hist.root
-rw-r--r-- 1 dune004 dune  19502102 Oct 15 18:31 debug.log
-rw-r--r-- 1 dune004 dune 109360482 Oct 15 18:31 pdhd_228838_59_1_20251015T161318Z_beam.root
-rw-r--r-- 1 dune004 dune      1494 Oct 15 16:13 all-input-dids.txt