justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 228838.92@dunegpschedd02.fnal.gov

Jobsub ID228838.92@dunegpschedd02.fnal.gov
Workflow ID9021
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-15 16:11:35
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2025-10-15 16:16:51
From worker nodeHostnamehd80.dice.priv
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-15 16:13:10
Input fileshd-protodune:np04hd_raw_run028891_0866_dataflow2_datawriter_0_20240824T131850.hdf5
hd-protodune:np04hd_raw_run028891_0874_dataflow7_datawriter_0_20240824T133009.hdf5
JobscriptExit code0
Real time3m (199s)
CPU time0m (21s = 10%)
Max RSS bytes760729600 (725 MiB)
Outputting started2025-10-15 16:16:29
Output files
Finished2025-10-15 16:16:51
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 16:16:22 UTC  run: 28891 subRun: 1 event: 216975 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 16:16:22 UTC  run: 28891 subRun: 1 event: 216975 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: -1 5.69126
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 16:16:22 UTC  run: 28891 subRun: 1 event: 216975 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 216983 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 97.2002 2.31855
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 16:16:23 UTC  run: 28891 subRun: 1 event: 216983 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 216991 5
%MSG
Timing trigger: 8
Matched: 0
CKovs: 0 0
TOF, P: 0 -1
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 16:16:25 UTC  run: 28891 subRun: 1 event: 216991 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 216999 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: -1 -1
15-Oct-2025 16:16:29 UTC  Closed output file "pdhd_228838_92_1_20251015T161317Z_beam.root"
15-Oct-2025 16:16:29 UTC  Closed output file "pdhd_228838_92_1_20251015T161317Z_beam.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.784296        2.491        47.0572       1.43602       6.00363        62     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.7012e-05    2.18491e-05   0.000138841   1.9712e-05    1.51442e-05      62     
produce:triggerrawdecoder:PDHDTriggerReader3      0.547801       1.19442       2.85239       1.10523      0.585567        62     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0964751     0.145768       2.07752      0.0973941     0.250925        62     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0193169     0.0195477     0.0198405     0.0195288    0.000124401      62     
produce:beamevent:BeamEvent                      0.000190137     1.01175        45.78       0.0060681      6.01856        62     
[art]:TriggerResults:TriggerResultInserter       1.4147e-05    1.98564e-05   5.5664e-05    1.84445e-05   6.30436e-06      62     
end_path:out1:RootOutput                          3.126e-06    4.03502e-06   2.1069e-05     3.662e-06    2.23384e-06      62     
end_path:out1:RootOutput(write)                   0.072976      0.119138      0.166674       0.12136      0.0237468       62     
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.784296        2.491        47.0572       1.43602       6.00363        62     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.7012e-05    2.18491e-05   0.000138841   1.9712e-05    1.51442e-05      62     
produce:triggerrawdecoder:PDHDTriggerReader3      0.547801       1.19442       2.85239       1.10523      0.585567        62     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0964751     0.145768       2.07752      0.0973941     0.250925        62     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0193169     0.0195477     0.0198405     0.0195288    0.000124401      62     
produce:beamevent:BeamEvent                      0.000190137     1.01175        45.78       0.0060681      6.01856        62     
[art]:TriggerResults:TriggerResultInserter       1.4147e-05    1.98564e-05   5.5664e-05    1.84445e-05   6.30436e-06      62     
end_path:out1:RootOutput                          3.126e-06    4.03502e-06   2.1069e-05     3.662e-06    2.23384e-06      62     
end_path:out1:RootOutput(write)                   0.072976      0.119138      0.166674       0.12136      0.0237468       62     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1593.1 MB
  Peak resident set size usage (VmHWM): 760.73 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1593.1 MB
  Peak resident set size usage (VmHWM): 760.73 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 62 passed = 62 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         62         62          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 14.787384 Real = 186.835737

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1593.1 VmHWM = 760.73

Art has completed and will exit with status 0.
gen lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 19616
-rw-r--r-- 1 dnpil001 users  2121065 Oct 15 16:16 jobscript.log
-rw-r--r-- 1 dnpil001 users      326 Oct 15 16:16 justin-processed-pfns.txt
-rw-r--r-- 1 dnpil001 users  2202962 Oct 15 16:16 pdhd_228838_92_1_20251015T161317Z_beam_hist.root
-rw-r--r-- 1 dnpil001 users  2074588 Oct 15 16:16 debug.log
-rw-r--r-- 1 dnpil001 users 11603962 Oct 15 16:16 pdhd_228838_92_1_20251015T161317Z_beam.root
-rw-r--r-- 1 dnpil001 users      166 Oct 15 16:13 all-input-dids.txt
justIN time: 2025-11-03 18:49:52 UTC       justIN version: 01.05.01