Jobsub ID 228838.48@dunegpschedd02.fnal.gov

Jobsub ID	228838.48@dunegpschedd02.fnal.gov
Workflow ID	9021
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-15 16:11:35
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce02
Last heartbeat	2025-10-15 17:46:45
From worker node	Hostname	dune004-8505282.0-lcg2703.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-15 16:13:03
Input files	hd-protodune:np04hd_raw_run028891_0024_dataflow5_datawriter_0_20240823T155136.hdf5 hd-protodune:np04hd_raw_run028891_0026_dataflow4_datawriter_0_20240823T155427.hdf5 hd-protodune:np04hd_raw_run028891_0027_dataflow7_datawriter_0_20240823T155551.hdf5 hd-protodune:np04hd_raw_run028891_0030_dataflow6_datawriter_0_20240823T160010.hdf5 hd-protodune:np04hd_raw_run028891_0032_dataflow4_datawriter_0_20240823T160249.hdf5 hd-protodune:np04hd_raw_run028891_0036_dataflow1_datawriter_0_20240823T160800.hdf5 hd-protodune:np04hd_raw_run028891_0038_dataflow5_datawriter_0_20240823T161026.hdf5 hd-protodune:np04hd_raw_run028891_0859_dataflow7_datawriter_0_20240824T130900.hdf5 hd-protodune:np04hd_raw_run028891_0870_dataflow1_datawriter_0_20240824T132419.hdf5 hd-protodune:np04hd_raw_run028891_0882_dataflow3_datawriter_0_20240824T134143.hdf5 hd-protodune:np04hd_raw_run028891_0872_dataflow5_datawriter_0_20240824T132709.hdf5 hd-protodune:np04hd_raw_run028891_0889_dataflow1_datawriter_0_20240824T135126.hdf5 hd-protodune:np04hd_raw_run028891_0883_dataflow0_datawriter_0_20240824T134312.hdf5 hd-protodune:np04hd_raw_run028891_0890_dataflow2_datawriter_0_20240824T135255.hdf5
Jobscript	Exit code	0
	Real time	1h (5601s)
	CPU time	1m (105s = 1%)
	Max RSS bytes	635645952 (606 MiB)
Outputting started	2025-10-15 17:46:24
Output files
Finished	2025-10-15 17:46:45
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

2.7923e+06

%MSG
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 17:46:21 UTC  run: 28891 subRun: 1 event: 220954 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 17:46:21 UTC  run: 28891 subRun: 1 event: 220954 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 95.6487 4.78364
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 17:46:21 UTC  run: 28891 subRun: 1 event: 220954 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 220970 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 94.9365 6.73922
15-Oct-2025 17:46:23 UTC  Closed output file "pdhd_228838_48_1_20251015T161318Z_beam.root"
15-Oct-2025 17:46:23 UTC  Closed output file "pdhd_228838_48_1_20251015T161318Z_beam.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.868493       12.3232       1881.14      0.992984       114.302        434    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.5624e-05    2.53352e-05   0.000398973   2.17575e-05   2.09656e-05      434    
produce:triggerrawdecoder:PDHDTriggerReader3      0.625141      0.715669       2.00308      0.693461      0.0909758       434    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.113419       0.12763      0.254903      0.116097      0.0181913       434    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0228034     0.0233624     0.0458254     0.0231782    0.00131284       434    
produce:beamevent:BeamEvent                      0.000312632     11.3388       1880.22     0.00749211      114.281        434    
[art]:TriggerResults:TriggerResultInserter       1.1547e-05    2.24347e-05   9.0337e-05    2.10015e-05   8.18284e-06      434    
end_path:out1:RootOutput                          3.215e-06    4.21553e-06   7.3281e-05     3.906e-06    3.63105e-06      434    
end_path:out1:RootOutput(write)                   0.066301      0.117335      0.196345      0.116478      0.0227678       434    
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.868493       12.3232       1881.14      0.992984       114.302        434    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.5624e-05    2.53352e-05   0.000398973   2.17575e-05   2.09656e-05      434    
produce:triggerrawdecoder:PDHDTriggerReader3      0.625141      0.715669       2.00308      0.693461      0.0909758       434    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.113419       0.12763      0.254903      0.116097      0.0181913       434    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0228034     0.0233624     0.0458254     0.0231782    0.00131284       434    
produce:beamevent:BeamEvent                      0.000312632     11.3388       1880.22     0.00749211      114.281        434    
[art]:TriggerResults:TriggerResultInserter       1.1547e-05    2.24347e-05   9.0337e-05    2.10015e-05   8.18284e-06      434    
end_path:out1:RootOutput                          3.215e-06    4.21553e-06   7.3281e-05     3.906e-06    3.63105e-06      434    
end_path:out1:RootOutput(write)                   0.066301      0.117335      0.196345      0.116478      0.0227678       434    
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1905.51 MB
  Peak resident set size usage (VmHWM): 963.99 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1905.51 MB
  Peak resident set size usage (VmHWM): 963.99 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 434 passed = 434 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        434        434          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 96.928123 Real = 5582.132043

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1905.51 VmHWM = 963.99

Art has completed and will exit with status 0.
gen lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 216084
-rw-r--r-- 1 dune004 dune  16008674 Oct 15 17:46 jobscript.log
-rw-r--r-- 1 dune004 dune      2282 Oct 15 17:46 justin-processed-pfns.txt
-rw-r--r-- 1 dune004 dune 102667824 Oct 15 17:46 pdhd_228838_48_1_20251015T161318Z_beam_hist.root
-rw-r--r-- 1 dune004 dune  15663185 Oct 15 17:46 debug.log
-rw-r--r-- 1 dune004 dune  83983961 Oct 15 17:46 pdhd_228838_48_1_20251015T161318Z_beam.root
-rw-r--r-- 1 dune004 dune      1162 Oct 15 16:13 all-input-dids.txt