Jobsub ID 46971.13@dunegpschedd01.fnal.gov
| Jobsub ID | 46971.13@dunegpschedd01.fnal.gov |
| Workflow ID | 2641 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-16 11:54:53 |
| Site | NL_SURFsara |
| Entry | DUNE_SurfSARA_arc02 |
| Last heartbeat | 2025-09-16 13:27:38 |
| From worker node | Hostname | wn-db-10.gina.surf.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-09-16 12:14:36 |
| Input files | vd-protodune:np02vd_raw_run039353_0943_df-s05-d0_dw_0_20250916T105125.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-09-16 13:27:38 |
| Saved logs | justin-logs:46971.13-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
: 0 sec
[15:24:27.967] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[15:24:27.967] I [ timer ] Timer: Total node execution : 83.5299993623048 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 51187 traces tagged "gauss"
FrameSaver: q=8.84642e+06 n=1047184 tag=gauss
wclsFrameSaver: saving 61525 traces tagged "wiener"
FrameSaver: q=9.23754e+06 n=1010290 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 550 XUs and 626 XVs -> 13 XUVs
C:0 T:1 500 XUs and 585 XVs -> 25 XUVs
C:0 T:2 2070 XUs and 1839 XVs -> 81 XUVs
C:0 T:3 742 XUs and 722 XVs -> 47 XUVs
C:0 T:4 926 XUs and 828 XVs -> 34 XUVs
C:0 T:5 308 XUs and 345 XVs -> 17 XUVs
C:0 T:6 296 XUs and 425 XVs -> 17 XUVs
C:0 T:7 366 XUs and 385 XVs -> 15 XUVs
C:0 T:8 4853 XUs and 5082 XVs -> 371 XUVs
C:0 T:9 2995 XUs and 3264 XVs -> 164 XUVs
C:0 T:10 4776 XUs and 4564 XVs -> 298 XUVs
C:0 T:11 870 XUs and 859 XVs -> 35 XUVs
C:0 T:12 1239 XUs and 1481 XVs -> 72 XUVs
C:0 T:13 3545 XUs and 5648 XVs -> 147 XUVs
C:0 T:14 318 XUs and 724 XVs -> 25 XUVs
C:0 T:15 1389 XUs and 1741 XVs -> 78 XUVs
1439 XUVs total
1217 collection wire objects
1439 potential space points
Neighbour search...
12115 tests to find 5570 neighbours
Iterating with no regularization...
Begin: 7.0868e+08
0 7.00532e+08
1 7.00387e+08
Now with regularization...
Begin: 6.92847e+08
0 6.92835e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
#000: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dio.c line 179 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#001: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 2011 in H5VL_dataset_read(): dataset read failed
major: Virtual Object Layer
minor: Read failed
#002: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 1978 in H5VL__dataset_read(): dataset read failed
major: Virtual Object Layer
minor: Read failed
#003: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLnative_dataset.c line 166 in H5VL__native_dataset_read(): can't read data
major: Dataset
minor: Read failed
#004: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dio.c line 545 in H5D__read(): can't read data
major: Dataset
minor: Read failed
#005: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dcontig.c line 600 in H5D__contig_read(): contiguous read failed
major: Dataset
minor: Read failed
#006: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dselect.c line 465 in H5D__select_read(): read error
major: Dataspace
minor: Read failed
#007: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dselect.c line 220 in H5D__select_io(): read error
major: Dataspace
minor: Read failed
#008: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dcontig.c line 924 in H5D__contig_readvv(): can't perform vectorized sieve buffer read
major: Dataset
minor: Can't operate on object
#009: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VM.c line 1401 in H5VM_opvv(): can't perform operation
major: Internal error (too specific to document in detail)
minor: Can't operate on object
#010: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Dcontig.c line 729 in H5D__contig_readvv_sieve_cb(): block read failed
major: Dataset
minor: Read failed
#011: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fio.c line 105 in H5F_shared_block_read(): read through page buffer failed
major: Low-level I/O
minor: Read failed
#012: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PB.c line 721 in H5PB_read(): read through metadata accumulator failed
major: Page Buffering
minor: Read failed
#013: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Faccum.c line 252 in H5F__accum_read(): driver read request failed
major: Low-level I/O
minor: Read failed
#014: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDint.c line 189 in H5FD_read(): driver read request failed
major: Virtual File Layer
minor: Read failed
#015: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDsec2.c line 755 in H5FD__sec2_read(): file read failed: time = Tue Sep 16 15:27:10 2025
, filename = 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/vd-protodune/b5/ca/np02vd_raw_run039353_0943_df-s05-d0_dw_0_20250916T105125.hdf5', file descriptor = 24, errno = 113, error message = 'No route to host', buf = 0x34c06dc0, total read size = 1137672, bytes this sub-read = 1137672, bytes actually read = 18446744073709551615, offset = 0
major: Low-level I/O
minor: Read failed
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 0.000114629 264.606 362.888 278.273 95.7267 16
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 7.9973e-05 0.000109397 0.000260287 8.90645e-05 4.4446e-05 16
produce:tpcrawdecoder:PDVDTPCReader 39.5655 91.7234 155.765 91.1061 26.4526 15
produce:triggerrawdecoder:PDVDTriggerReader4 0.313346 0.375949 0.545023 0.362603 0.0545483 15
produce:pdvddaphne:DAPHNEReaderPDVD 0.0003561 0.000455542 0.000929253 0.000390024 0.000148546 15
produce:ophit:OpHitFinder 0.00014217 0.000258897 0.00100327 0.000173351 0.000209777 15
produce:opflash:OpFlashFinderVerticalDrift 4.8563e-05 8.41619e-05 0.000427695 5.6408e-05 9.23015e-05 15
produce:wclsdatavd:WireCellToolkit 16.4526 92.5092 142.664 88.8872 26.8934 15
produce:gaushit:GausHitFinder 0.508539 1.62105 2.12648 1.72992 0.403282 15
produce:nhitsfilter:NumberOfHitsFilter 0.000297808 0.00127713 0.00228798 0.00120548 0.000477594 15
produce:reco3d:SpacePointSolver 2.2395 17.5024 25.0041 18.4255 5.23605 15
produce:hitpdune:DisambigFromSpacePoints 0.0238242 0.245964 0.354748 0.254557 0.0755493 15
produce:pandora:StandardPandora 2.58655 70.9554 152.086 65.5418 33.8306 15
produce:pandoraTrack:LArPandoraTrackCreation 0.157893 2.21946 4.38258 2.25744 1.0929 15
produce:pandoraGnocalo:GnocchiCalorimetry 0.0119866 0.0424308 0.0594447 0.0443762 0.0110034 15
[art]:TriggerResults:TriggerResultInserter 2.6761e-05 5.58137e-05 0.000129828 4.178e-05 3.1211e-05 15
end_path:out1:RootOutput 6.412e-06 1.21105e-05 3.2512e-05 7.464e-06 8.51703e-06 15
end_path:out1:RootOutput(write) 1.15347 4.94681 5.44557 5.21258 1.02543 15
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4641.17 MB
Peak resident set size usage (VmHWM): 2748.61 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 16-Sep-2025 15:27:11 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- StdException BEGIN
An exception was thrown while processing module PDVDTPCReader/tpcrawdecoder run: 39353 subRun: 1 event: 72712
Error during HDF5 Read:
---- StdException END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 