Jobsub ID 46968.92@dunegpschedd01.fnal.gov
| Jobsub ID | 46968.92@dunegpschedd01.fnal.gov |
| Workflow ID | 2650 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-16 11:42:55 |
| Site | NL_SURFsara |
| Entry | DUNE_SurfSARA_arc03 |
| Last heartbeat | 2025-09-16 13:27:58 |
| From worker node | Hostname | wn-la-04.gina.surf.nl |
| cpuinfo | AMD EPYC 9754 128-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-09-16 11:59:31 |
| Input files | vd-protodune:np02vd_raw_run039275_0357_df-s04-d0_dw_0_20250902T025008.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-09-16 13:27:58 |
| Saved logs | justin-logs:46968.92-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
Timer: WireCell::Gen::FrameFanout : 0 sec
[15:22:10.424] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[15:22:10.424] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[15:22:10.424] I [ timer ] Timer: Total node execution : 98.07999895699322 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 65368 traces tagged "gauss"
FrameSaver: q=1.2429e+07 n=1439514 tag=gauss
wclsFrameSaver: saving 81157 traces tagged "wiener"
FrameSaver: q=1.33181e+07 n=1377323 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 843 XUs and 1161 XVs -> 30 XUVs
C:0 T:1 424 XUs and 527 XVs -> 19 XUVs
C:0 T:2 690 XUs and 1028 XVs -> 36 XUVs
C:0 T:3 545 XUs and 622 XVs -> 15 XUVs
C:0 T:4 335 XUs and 478 XVs -> 19 XUVs
C:0 T:5 1215 XUs and 1480 XVs -> 48 XUVs
C:0 T:6 337 XUs and 334 XVs -> 18 XUVs
C:0 T:7 897 XUs and 1119 XVs -> 80 XUVs
C:0 T:8 19887 XUs and 54108 XVs -> 8540 XUVs
C:0 T:9 19595 XUs and 63224 XVs -> 7429 XUVs
C:0 T:10 15577 XUs and 32604 XVs -> 3761 XUVs
C:0 T:11 5368 XUs and 6880 XVs -> 991 XUVs
C:0 T:12 16704 XUs and 31671 XVs -> 4204 XUVs
C:0 T:13 3861 XUs and 3999 XVs -> 388 XUVs
C:0 T:14 2451 XUs and 1757 XVs -> 99 XUVs
C:0 T:15 5127 XUs and 5806 XVs -> 694 XUVs
26371 XUVs total
2985 collection wire objects
26371 potential space points
Neighbour search...
4481169 tests to find 1815118 neighbours
Iterating with no regularization...
Begin: 7.74662e+08
0 6.89272e+08
1 6.83692e+08
2 6.82653e+08
3 6.82284e+08
Now with regularization...
Begin: 6.63851e+08
0 6.63529e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
#000: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dio.c line 179 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#001: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 2011 in H5VL_dataset_read(): dataset read failed
major: Virtual Object Layer
minor: Read failed
#002: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 1978 in H5VL__dataset_read(): dataset read failed
major: Virtual Object Layer
minor: Read failed
#003: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLnative_dataset.c line 166 in H5VL__native_dataset_read(): can't read data
major: Dataset
minor: Read failed
#004: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dio.c line 545 in H5D__read(): can't read data
major: Dataset
minor: Read failed
#005: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dcontig.c line 600 in H5D__contig_read(): contiguous read failed
major: Dataset
minor: Read failed
#006: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dselect.c line 465 in H5D__select_read(): read error
major: Dataspace
minor: Read failed
#007: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dselect.c line 220 in H5D__select_io(): read error
major: Dataspace
minor: Read failed
#008: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dcontig.c line 924 in H5D__contig_readvv(): can't perform vectorized sieve buffer read
major: Dataset
minor: Can't operate on object
#009: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VM.c line 1401 in H5VM_opvv(): can't perform operation
major: Internal error (too specific to document in detail)
minor: Can't operate on object
#010: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dcontig.c line 729 in H5D__contig_readvv_sieve_cb(): block read failed
major: Dataset
minor: Read failed
#011: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fio.c line 105 in H5F_shared_block_read(): read through page buffer failed
major: Low-level I/O
minor: Read failed
#012: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PB.c line 721 in H5PB_read(): read through metadata accumulator failed
major: Page Buffering
minor: Read failed
#013: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Faccum.c line 252 in H5F__accum_read(): driver read request failed
major: Low-level I/O
minor: Read failed
#014: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDint.c line 189 in H5FD_read(): driver read request failed
major: Virtual File Layer
minor: Read failed
#015: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDsec2.c line 755 in H5FD__sec2_read(): file read failed: time = Tue Sep 16 15:27:10 2025
, filename = 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/vd-protodune/d1/58/np02vd_raw_run039275_0357_df-s04-d0_dw_0_20250902T025008.hdf5', file descriptor = 24, errno = 113, error message = 'No route to host', buf = 0x17f444d0, total read size = 1137672, bytes this sub-read = 1137672, bytes actually read = 18446744073709551615, offset = 0
major: Low-level I/O
minor: Read failed
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 7.9017e-05 300.885 446.957 299.836 90.1341 17
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.3854e-05 8.45522e-05 0.000190094 7.7377e-05 2.79842e-05 17
produce:tpcrawdecoder:PDVDTPCReader 66.5381 95.2619 125.229 94.0896 15.3528 16
produce:triggerrawdecoder:PDVDTriggerReader4 0.0331174 0.0410078 0.0935248 0.0376062 0.0137509 16
produce:pdvddaphne:DAPHNEReaderPDVD 9.81929 15.7437 45.2609 14.4947 7.82864 16
produce:ophit:OpHitFinder 0.0500555 0.065205 0.0790093 0.0644443 0.0072019 16
produce:opflash:OpFlashFinderVerticalDrift 0.00739825 0.0142848 0.0230844 0.0136157 0.0043045 16
produce:wclsdatavd:WireCellToolkit 73.8434 91.2408 116.909 88.8717 9.90477 16
produce:gaushit:GausHitFinder 1.02918 1.56621 2.20569 1.52942 0.29465 16
produce:nhitsfilter:NumberOfHitsFilter 0.0003145 0.00059491 0.0011194 0.00050569 0.000261009 16
produce:reco3d:SpacePointSolver 11.1951 22.0997 36.7694 21.5073 6.34618 16
produce:hitpdune:DisambigFromSpacePoints 0.176835 0.332453 0.578098 0.310656 0.102942 16
produce:pandora:StandardPandora 31.8448 85.4459 162.237 80.625 37.6708 16
produce:pandoraTrack:LArPandoraTrackCreation 1.039 1.98816 3.45803 1.69981 0.816327 16
produce:pandoraGnocalo:GnocchiCalorimetry 0.0222848 0.0356362 0.0584358 0.0334995 0.0107266 16
[art]:TriggerResults:TriggerResultInserter 2.8433e-05 3.82747e-05 0.000119478 3.19175e-05 2.14186e-05 16
end_path:out1:RootOutput 5.519e-06 8.64737e-06 3.6164e-05 6.7655e-06 7.14458e-06 16
end_path:out1:RootOutput(write) 4.91082 5.76278 7.19967 5.31781 0.805162 16
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5204.34 MB
Peak resident set size usage (VmHWM): 3319.57 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 16-Sep-2025 15:27:11 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- StdException BEGIN
An exception was thrown while processing module PDVDTPCReader/tpcrawdecoder run: 39275 subRun: 1 event: 128850
Error during HDF5 Read:
---- StdException END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 