justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234359.23@dunegpschedd01.fnal.gov

Jobsub ID234359.23@dunegpschedd01.fnal.gov
Workflow ID9021
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-15 16:15:35
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_brug
Last heartbeat2025-10-15 16:18:33
From worker nodeHostnamewn-pijl-023.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-15 16:16:44
Input fileshd-protodune:np04hd_raw_run028891_0875_dataflow3_datawriter_0_20240824T133137.hdf5
JobscriptExit code0
Real time1m (71s)
CPU time0m (18s = 25%)
Max RSS bytes781893632 (745 MiB)
Outputting started2025-10-15 16:17:57
Output files
Finished2025-10-15 16:18:33
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
WARNING: NONZERO HALF
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 18:17:52 CEST  run: 28891 subRun: 1 event: 217243 BeamEvent_module.cc:1828
Could not get XCET1 info

%MSG
%MSG-w BeamEvent:  BeamEvent:beamevent@BeginModule 15-Oct-2025 18:17:52 CEST  run: 28891 subRun: 1 event: 217243 BeamEvent_module.cc:1846
Could not get XCET2 info

%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 97.0683 4.77362
%MSG-w HDF5FileLayout.cpp:  PostProcessEvent 15-Oct-2025 18:17:52 CEST  run: 28891 subRun: 1 event: 217243 HDF5FileLayout.cpp:70
 File Layout Not Enough Digits For Path 217251 5
%MSG
Timing trigger: 12
Matched: 1
CKovs: 1 0
TOF, P: 96.3897 -1
15-Oct-2025 18:17:56 CEST  Closed output file "pdhd_234359_23_1_20251015T161653Z_beam.root"
15-Oct-2025 18:17:56 CEST  Closed output file "pdhd_234359_23_1_20251015T161653Z_beam.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.759044       1.09079       6.83778      0.857409       1.05554        31     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.9079e-05    2.64092e-05   0.000154934   2.1883e-05    2.37232e-05      31     
produce:triggerrawdecoder:PDHDTriggerReader3      0.497046      0.554438       0.66093      0.543633      0.0368257       31     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0879745     0.0984605     0.144081      0.0889291     0.0153792       31     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176733     0.0182238     0.0289191     0.0178258    0.00195785       31     
produce:beamevent:BeamEvent                      0.00045717      0.24772       5.94081     0.00543857      1.04503        31     
[art]:TriggerResults:TriggerResultInserter       1.2799e-05    2.67851e-05   0.000131548   2.3665e-05    1.94024e-05      31     
end_path:out1:RootOutput                          3.635e-06    5.01029e-06   2.4457e-05     4.276e-06    3.58093e-06      31     
end_path:out1:RootOutput(write)                   0.122114      0.171521      0.269734      0.157931      0.0378987       31     
===================================================================================================================================

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                        0.759044       1.09079       6.83778      0.857409       1.05554        31     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       1.9079e-05    2.64092e-05   0.000154934   2.1883e-05    2.37232e-05      31     
produce:triggerrawdecoder:PDHDTriggerReader3      0.497046      0.554438       0.66093      0.543633      0.0368257       31     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0879745     0.0984605     0.144081      0.0889291     0.0153792       31     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176733     0.0182238     0.0289191     0.0178258    0.00195785       31     
produce:beamevent:BeamEvent                      0.00045717      0.24772       5.94081     0.00543857      1.04503        31     
[art]:TriggerResults:TriggerResultInserter       1.2799e-05    2.67851e-05   0.000131548   2.3665e-05    1.94024e-05      31     
end_path:out1:RootOutput                          3.635e-06    5.01029e-06   2.4457e-05     4.276e-06    3.58093e-06      31     
end_path:out1:RootOutput(write)                   0.122114      0.171521      0.269734      0.157931      0.0378987       31     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1587.27 MB
  Peak resident set size usage (VmHWM): 781.894 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1587.27 MB
  Peak resident set size usage (VmHWM): 781.894 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 31 passed = 31 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         31         31          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 10.085256 Real = 50.318503

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1587.27 VmHWM = 781.894

Art has completed and will exit with status 0.
gen lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 9232
-rw-r--r-- 1 dune090 dune 1248173 Oct 15 18:17 jobscript.log
-rw-r--r-- 1 dune090 dune     163 Oct 15 18:17 justin-processed-pfns.txt
-rw-r--r-- 1 dune090 dune  586018 Oct 15 18:17 pdhd_234359_23_1_20251015T161653Z_beam_hist.root
-rw-r--r-- 1 dune090 dune 1219104 Oct 15 18:17 debug.log
-rw-r--r-- 1 dune090 dune 6373138 Oct 15 18:17 pdhd_234359_23_1_20251015T161653Z_beam.root
-rw-r--r-- 1 dune090 dune      83 Oct 15 18:16 all-input-dids.txt
justIN time: 2025-11-03 18:44:36 UTC       justIN version: 01.05.01