justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234369.43@dunegpschedd01.fnal.gov

Jobsub ID234369.43@dunegpschedd01.fnal.gov
Workflow ID9021
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-15 16:39:36
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce02
Last heartbeat2025-10-15 18:07:47
From worker nodeHostnameccwcondor0687
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-15 16:41:18
Input fileshd-protodune:np04hd_raw_run028891_0937_dataflow3_datawriter_0_20240824T150509.hdf5
hd-protodune:np04hd_raw_run028891_0950_dataflow4_datawriter_0_20240824T152929.hdf5
hd-protodune:np04hd_raw_run028891_0969_dataflow1_datawriter_0_20240824T155944.hdf5
hd-protodune:np04hd_raw_run028891_0914_dataflow7_datawriter_0_20240824T142845.hdf5
JobscriptExit code0
Real time1h (5157s)
CPU time0m (34s = 0%)
Max RSS bytes727502848 (693 MiB)
Outputting started2025-10-15 18:07:16
Output files
Finished2025-10-15 18:07:47
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

F
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 20:07:11 CEST  run: 28891 subRun: 1 event: 226911 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 20:07:11 CEST  run: 28891 subRun: 1 event: 226911 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 0 0
TOF, P: 94.6905 -1
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 20:07:11 CEST  run: 28891 subRun: 1 event: 226911 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 226919 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 95.4491 4.23902
15-Oct-2025 20:07:15 CEST  Closed output file "pdhd_234369_43_1_20251015T164127Z_beam.root"
15-Oct-2025 20:07:15 CEST  Closed output file "pdhd_234369_43_1_20251015T164127Z_beam.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.197657       41.2861       2897.29      0.312859       285.151        124    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        1.612e-05    2.3264e-05    0.000192875    2.118e-05    1.58396e-05      124    
produce:triggerrawdecoder:PDHDTriggerReader3      0.0991961     0.156745       0.61144      0.146806      0.0565369       124    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0140293     0.0257734     0.102433      0.0204325     0.0159813       124    
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00286509    0.00350137     0.0112434     0.0035622    0.00083217       124    
produce:beamevent:BeamEvent                      0.000533909     40.9911       2897.01     0.00958838      285.152        124    
[art]:TriggerResults:TriggerResultInserter       1.3846e-05    2.45456e-05    7.917e-05    2.4176e-05    7.9208e-06       124    
end_path:out1:RootOutput                          3.246e-06    4.81455e-06    3.13e-05      4.313e-06    2.88244e-06      124    
end_path:out1:RootOutput(write)                   0.0630893     0.108609      0.171074      0.106617      0.0211116       124    
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.197657       41.2861       2897.29      0.312859       285.151        124    
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        1.612e-05    2.3264e-05    0.000192875    2.118e-05    1.58396e-05      124    
produce:triggerrawdecoder:PDHDTriggerReader3      0.0991961     0.156745       0.61144      0.146806      0.0565369       124    
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0140293     0.0257734     0.102433      0.0204325     0.0159813       124    
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00286509    0.00350137     0.0112434     0.0035622    0.00083217       124    
produce:beamevent:BeamEvent                      0.000533909     40.9911       2897.01     0.00958838      285.152        124    
[art]:TriggerResults:TriggerResultInserter       1.3846e-05    2.45456e-05    7.917e-05    2.4176e-05    7.9208e-06       124    
end_path:out1:RootOutput                          3.246e-06    4.81455e-06    3.13e-05      4.313e-06    2.88244e-06      124    
end_path:out1:RootOutput(write)                   0.0630893     0.108609      0.171074      0.106617      0.0211116       124    
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1641.2 MB
  Peak resident set size usage (VmHWM): 726.454 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1641.2 MB
  Peak resident set size usage (VmHWM): 726.454 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 124 passed = 124 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        124        124          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 28.259569 Real = 5144.334704

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1641.2 VmHWM = 726.454

Art has completed and will exit with status 0.
gen lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 40952
-rw-r--r-- 1 dune001 lbno  4549073 Oct 15 20:07 jobscript.log
-rw-r--r-- 1 dune001 lbno      652 Oct 15 20:07 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno  8866144 Oct 15 20:07 pdhd_234369_43_1_20251015T164127Z_beam_hist.root
-rw-r--r-- 1 dune001 lbno  4448621 Oct 15 20:07 debug.log
-rw-r--r-- 1 dune001 lbno 23491258 Oct 15 20:07 pdhd_234369_43_1_20251015T164127Z_beam.root
-rw-r--r-- 1 dune001 lbno      332 Oct 15 18:41 all-input-dids.txt
justIN time: 2025-11-04 01:31:16 UTC       justIN version: 01.05.01