Jobsub ID 228842.54@dunegpschedd02.fnal.gov

Jobsub ID	228842.54@dunegpschedd02.fnal.gov
Workflow ID	9021
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-15 16:17:36
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce2
Last heartbeat	2025-10-15 18:00:18
From worker node	Hostname	milan21.farm.particle.cz
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-15 16:18:45
Input files	hd-protodune:np04hd_raw_run028891_0062_dataflow7_datawriter_0_20240823T164629.hdf5 hd-protodune:np04hd_raw_run028891_0087_dataflow6_datawriter_0_20240823T172908.hdf5 hd-protodune:np04hd_raw_run028891_0090_dataflow1_datawriter_0_20240823T173616.hdf5 hd-protodune:np04hd_raw_run028891_0095_dataflow1_datawriter_0_20240823T174810.hdf5 hd-protodune:np04hd_raw_run028891_0099_dataflow2_datawriter_0_20240823T175751.hdf5 hd-protodune:np04hd_raw_run028891_0107_dataflow5_datawriter_0_20240823T181737.hdf5 hd-protodune:np04hd_raw_run028891_0115_dataflow0_datawriter_0_20240823T183230.hdf5 hd-protodune:np04hd_raw_run028891_0118_dataflow7_datawriter_0_20240823T183716.hdf5 hd-protodune:np04hd_raw_run028891_0123_dataflow1_datawriter_0_20240823T184445.hdf5
Jobscript	Exit code	0
	Real time	1h (6070s)
	CPU time	1m (61s = 1%)
	Max RSS bytes	884518912 (843 MiB)
Outputting started	2025-10-15 17:59:56
Output files
Finished	2025-10-15 18:00:18
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 19:59:54 CEST  run: 28891 subRun: 1 event: 30745 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 19:59:54 CEST  run: 28891 subRun: 1 event: 30745 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 97.7968 4.43884
15-Oct-2025 19:59:55 CEST  Closed output file "pdhd_228842_54_1_20251015T161858Z_beam.root"
15-Oct-2025 19:59:55 CEST  Closed output file "pdhd_228842_54_1_20251015T161858Z_beam.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.622849       21.3023       2441.8        0.73381       161.672        279    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        1.367e-05    2.13468e-05    8.419e-05     2.003e-05    6.84417e-06      279    
produce:triggerrawdecoder:PDHDTriggerReader3      0.432458      0.511663      0.972862      0.497105      0.0582227       279    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0771713     0.0897782     0.164506      0.0818179     0.0157874       279    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0154835     0.0158686     0.0220633     0.0158322    0.000421398      279    
produce:beamevent:BeamEvent                      0.000261201     20.5829       2441.06      0.0116093      161.668        279    
[art]:TriggerResults:TriggerResultInserter        1.131e-05    2.26791e-05    8.809e-05     2.034e-05    9.50887e-06      279    
end_path:out1:RootOutput                           2.8e-06     4.01631e-06    2.292e-05     3.88e-06     1.29072e-06      279    
end_path:out1:RootOutput(write)                   0.0602297     0.101704      0.224319      0.100315      0.0219337       279    
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.622849       21.3023       2441.8        0.73381       161.672        279    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        1.367e-05    2.13468e-05    8.419e-05     2.003e-05    6.84417e-06      279    
produce:triggerrawdecoder:PDHDTriggerReader3      0.432458      0.511663      0.972862      0.497105      0.0582227       279    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0771713     0.0897782     0.164506      0.0818179     0.0157874       279    
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0154835     0.0158686     0.0220633     0.0158322    0.000421398      279    
produce:beamevent:BeamEvent                      0.000261201     20.5829       2441.06      0.0116093      161.668        279    
[art]:TriggerResults:TriggerResultInserter        1.131e-05    2.26791e-05    8.809e-05     2.034e-05    9.50887e-06      279    
end_path:out1:RootOutput                           2.8e-06     4.01631e-06    2.292e-05     3.88e-06     1.29072e-06      279    
end_path:out1:RootOutput(write)                   0.0602297     0.101704      0.224319      0.100315      0.0219337       279    
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1782.71 MB
  Peak resident set size usage (VmHWM): 884.519 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1782.71 MB
  Peak resident set size usage (VmHWM): 884.519 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 279 passed = 279 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        279        279          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 54.614764 Real = 6053.035978

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1782.71 VmHWM = 884.519

Art has completed and will exit with status 0.
gen lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 98368
-rw-r--r-- 1 fermilab users  8091752 Oct 15 19:59 jobscript.log
-rw-r--r-- 1 fermilab users     1467 Oct 15 19:59 justin-processed-pfns.txt
-rw-r--r-- 1 fermilab users 32368532 Oct 15 19:59 pdhd_228842_54_1_20251015T161858Z_beam_hist.root
-rw-r--r-- 1 fermilab users  7919658 Oct 15 19:59 debug.log
-rw-r--r-- 1 fermilab users 52314526 Oct 15 19:59 pdhd_228842_54_1_20251015T161858Z_beam.root
-rw-r--r-- 1 fermilab users      747 Oct 15 18:18 all-input-dids.txt