Jobsub ID 228838.76@dunegpschedd02.fnal.gov

Jobsub ID	228838.76@dunegpschedd02.fnal.gov
Workflow ID	9021
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-15 16:11:35
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel_multicore
Last heartbeat	2025-10-15 18:10:15
From worker node	Hostname	wn-sate-004.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-15 16:12:59
Input files	hd-protodune:np04hd_raw_run028891_0015_dataflow7_datawriter_0_20240823T153749.hdf5 hd-protodune:np04hd_raw_run028891_0015_dataflow4_datawriter_0_20240823T153738.hdf5 hd-protodune:np04hd_raw_run028891_0018_dataflow0_datawriter_0_20240823T154208.hdf5 hd-protodune:np04hd_raw_run028891_0020_dataflow6_datawriter_0_20240823T154502.hdf5 hd-protodune:np04hd_raw_run028891_0024_dataflow7_datawriter_0_20240823T155136.hdf5 hd-protodune:np04hd_raw_run028891_0033_dataflow3_datawriter_0_20240823T160404.hdf5 hd-protodune:np04hd_raw_run028891_0027_dataflow6_datawriter_0_20240823T155551.hdf5 hd-protodune:np04hd_raw_run028891_0030_dataflow2_datawriter_0_20240823T160010.hdf5 hd-protodune:np04hd_raw_run028891_0034_dataflow0_datawriter_0_20240823T160524.hdf5 hd-protodune:np04hd_raw_run028891_0037_dataflow5_datawriter_0_20240823T160910.hdf5 hd-protodune:np04hd_raw_run028891_0039_dataflow7_datawriter_0_20240823T161155.hdf5 hd-protodune:np04hd_raw_run028891_0858_dataflow1_datawriter_0_20240824T130734.hdf5 hd-protodune:np04hd_raw_run028891_0862_dataflow2_datawriter_0_20240824T131319.hdf5 hd-protodune:np04hd_raw_run028891_0879_dataflow4_datawriter_0_20240824T133704.hdf5 hd-protodune:np04hd_raw_run028891_0887_dataflow0_datawriter_0_20240824T134843.hdf5 hd-protodune:np04hd_raw_run028891_0865_dataflow1_datawriter_0_20240824T131720.hdf5 hd-protodune:np04hd_raw_run028891_0889_dataflow0_datawriter_0_20240824T135134.hdf5
Jobscript	Exit code	0
	Real time	1h (7008s)
	CPU time	2m (130s = 1%)
	Max RSS bytes	981487616 (936 MiB)
Outputting started	2025-10-15 18:09:48
Output files
Finished	2025-10-15 18:10:15
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

F
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 20:09:41 CEST  run: 28891 subRun: 1 event: 220704 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 20:09:41 CEST  run: 28891 subRun: 1 event: 220704 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 96.3135 5.83681
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 20:09:41 CEST  run: 28891 subRun: 1 event: 220704 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 220712 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 96.4717 5.13023
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 20:09:43 CEST  run: 28891 subRun: 1 event: 220712 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 220720 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 95.4209 5.49102
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 20:09:44 CEST  run: 28891 subRun: 1 event: 220720 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 220728 5
%MSG
Timing trigger: 8
Matched: 0
CKovs: 0 0
TOF, P: 0 -1
15-Oct-2025 20:09:47 CEST  Closed output file "pdhd_228838_76_1_20251015T161318Z_beam.root"
15-Oct-2025 20:09:47 CEST  Closed output file "pdhd_228838_76_1_20251015T161318Z_beam.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.722199       12.7972       1577.21       1.20041       96.6034        527    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.6261e-05    2.24934e-05   0.000152126    2.122e-05    7.52433e-06      527    
produce:triggerrawdecoder:PDHDTriggerReader3      0.486393      0.988049       6.86086      0.900812      0.503964        527    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0863573     0.106793       2.02017      0.0931974     0.0903393       527    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0173516     0.0183149     0.0596627     0.0176835    0.00417831       527    
produce:beamevent:BeamEvent                      0.000202018     11.5633       1576.45     0.00735244      96.6297        527    
[art]:TriggerResults:TriggerResultInserter       1.3706e-05    2.42866e-05   0.000103845   2.3424e-05    7.30331e-06      527    
end_path:out1:RootOutput                          3.016e-06    4.31348e-06    2.122e-05     4.117e-06    1.35725e-06      527    
end_path:out1:RootOutput(write)                   0.0632153     0.120385      0.257136      0.120526      0.0239622       527    
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.722199       12.7972       1577.21       1.20041       96.6034        527    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.6261e-05    2.24934e-05   0.000152126    2.122e-05    7.52433e-06      527    
produce:triggerrawdecoder:PDHDTriggerReader3      0.486393      0.988049       6.86086      0.900812      0.503964        527    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0863573     0.106793       2.02017      0.0931974     0.0903393       527    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0173516     0.0183149     0.0596627     0.0176835    0.00417831       527    
produce:beamevent:BeamEvent                      0.000202018     11.5633       1576.45     0.00735244      96.6297        527    
[art]:TriggerResults:TriggerResultInserter       1.3706e-05    2.42866e-05   0.000103845   2.3424e-05    7.30331e-06      527    
end_path:out1:RootOutput                          3.016e-06    4.31348e-06    2.122e-05     4.117e-06    1.35725e-06      527    
end_path:out1:RootOutput(write)                   0.0632153     0.120385      0.257136      0.120526      0.0239622       527    
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1858.89 MB
  Peak resident set size usage (VmHWM): 981.488 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1858.89 MB
  Peak resident set size usage (VmHWM): 981.488 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 527 passed = 527 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        527        527          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 121.929980 Real = 6983.729681

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1858.89 VmHWM = 981.488

Art has completed and will exit with status 0.
gen lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 266056
-rw-r--r--. 1 dune090 dune  19297886 Oct 15 20:09 jobscript.log
-rw-r--r--. 1 dune090 dune      2771 Oct 15 20:09 justin-processed-pfns.txt
-rw-r--r--. 1 dune090 dune 135180298 Oct 15 20:09 pdhd_228838_76_1_20251015T161318Z_beam_hist.root
-rw-r--r--. 1 dune090 dune  18885548 Oct 15 20:09 debug.log
-rw-r--r--. 1 dune090 dune  99043358 Oct 15 20:09 pdhd_228838_76_1_20251015T161318Z_beam.root
-rw-r--r--. 1 dune090 dune      1411 Oct 15 18:13 all-input-dids.txt