Jobsub ID 297917.171@dunegpschedd02.fnal.gov
| Jobsub ID | 297917.171@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:48:25 |
| Site | UK_Oxford |
| Entry | DUNE_UK_SGrid_Oxford_arc01 |
| Last heartbeat | 2026-02-04 00:10:52 |
| From worker node | Hostname | t2wn173.physics.ox.ac.uk |
| cpuinfo | AMD EPYC 9655 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:25:41 |
| Input files | vd-protodune:np02vd_raw_run042421_0006_df-s04-d1_dw_0_20260203T133408.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 00:10:52 |
| Saved logs | justin-logs:297917.171-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
0, 0xa8769510
Caught exception in looking up space point ptr... 0xa7f1e020, 0xa7ba8d70
Caught exception in looking up space point ptr... 0xa7ba8d70, 0xa7f1e020
Caught exception in looking up space point ptr... 0x10763c90, 0x9f30b30
Caught exception in looking up space point ptr... 0x9f30b30, 0x9f313e0
Caught exception in looking up space point ptr... 0xb6c1dc70, 0xb3699680
Caught exception in looking up space point ptr... 0xb3699680, 0xb6c1dc70
Caught exception in looking up space point ptr... 0x9e4ff600, 0x9e6156e0
Caught exception in looking up space point ptr... 0x9e6156e0, 0x9e4ff600
Caught exception in looking up space point ptr... 0x5088a240, 0x46dd5c00
Caught exception in looking up space point ptr... 0x46dd5c00, 0x9dfa730
Caught exception in looking up space point ptr... 0xa873dfb0, 0xa639fe50
Caught exception in looking up space point ptr... 0xa639fe50, 0xa6a7b9d0
Caught exception in looking up space point ptr... 0xe1ca4fe0, 0xe6644bb0
Caught exception in looking up space point ptr... 0xe6644bb0, 0xe62bf770
Caught exception in looking up space point ptr... 0x81fd0a60, 0x81ae2ee0
Caught exception in looking up space point ptr... 0x81ae2ee0, 0x81fd0a60
Caught exception in looking up space point ptr... 0x950a57b0, 0x9745bb20
Caught exception in looking up space point ptr... 0x9745bb20, 0x9745bdf0
Caught exception in looking up space point ptr... 0x97925e70, 0x961aaf00
Caught exception in looking up space point ptr... 0x961aaf00, 0x950a57b0
Caught exception in looking up space point ptr... 0xb828af00, 0xb87b75f0
Caught exception in looking up space point ptr... 0xb87b75f0, 0xc0f88960
Caught exception in looking up space point ptr... 0x7b25a8a0, 0x7bc05860
Caught exception in looking up space point ptr... 0x7bc05860, 0x77348940
Caught exception in looking up space point ptr... 0x77348940, 0x77348ad0
Caught exception in looking up space point ptr... 0x77348ad0, 0x7b25aa30
Caught exception in looking up space point ptr... 0x36162a60, 0x37cfbd50
Caught exception in looking up space point ptr... 0x37cfbd50, 0x3614c4f0
Caught exception in looking up space point ptr... 0xa0c49ce0, 0xa773be50
Caught exception in looking up space point ptr... 0xa773be50, 0xa0c49ce0
Caught exception in looking up space point ptr... 0xa0c49ce0, 0xa4bd47b0
Caught exception in looking up space point ptr... 0xa4bd47b0, 0xa0c49ce0
Caught exception in looking up space point ptr... 0xa0c49ce0, 0xa1bd93b0
Caught exception in looking up space point ptr... 0xa1bd93b0, 0xa0c49ce0
Caught exception in looking up space point ptr... 0xb86034d0, 0xb8888480
Caught exception in looking up space point ptr... 0xb8888480, 0xb86034d0
Caught exception in looking up space point ptr... 0xb8413b10, 0xb8ebbb70
Caught exception in looking up space point ptr... 0xb8ebbb70, 0xb54393e0
Caught exception in looking up space point ptr... 0xdbc08120, 0xdbbfce90
Caught exception in looking up space point ptr... 0xdbbfce90, 0xdbc08120
Caught exception in looking up space point ptr... 0x87ba3c40, 0x8a23ad50
Caught exception in looking up space point ptr... 0x8a23ad50, 0x87ba3c40
Caught exception in looking up space point ptr... 0xe08e13c0, 0xe03807b0
Caught exception in looking up space point ptr... 0xe03807b0, 0xe08e13c0
Caught exception in looking up space point ptr... 0x9e638b50, 0x9f105f30
Caught exception in looking up space point ptr... 0x9f105f30, 0x9e638b50
Caught exception in looking up space point ptr... 0x6c22a130, 0x6cadd760
Caught exception in looking up space point ptr... 0x6cadd760, 0x6cadf960
Caught exception in looking up space point ptr... 0x8fff48e0, 0xa4aabc60
Caught exception in looking up space point ptr... 0xa4aabc60, 0x8340d540
Caught exception in looking up space point ptr... 0xa39179a0, 0xa3920c40
Caught exception in looking up space point ptr... 0xa3920c40, 0xa39179a0
Caught exception in looking up space point ptr... 0xe728c5c0, 0xe6f7b640
Caught exception in looking up space point ptr... 0xe6f7b640, 0xe728c5c0
Caught exception in looking up space point ptr... 0xa9069740, 0x98a78000
Caught exception in looking up space point ptr... 0x98a78000, 0xa9069740
Caught exception in looking up space point ptr... 0x3d4054d0, 0x4a881330
Caught exception in looking up space point ptr... 0x4a881330, 0x3d4054d0
Caught exception in looking up space point ptr... 0x3d4054d0, 0x12253bc0
Caught exception in looking up space point ptr... 0x12253bc0, 0x3d4054d0
Caught exception in looking up space point ptr... 0xca67a620, 0xca67a520
Caught exception in looking up space point ptr... 0xca67a520, 0xca67a620
Caught exception in looking up space point ptr... 0x72362fd0, 0x72bef550
Caught exception in looking up space point ptr... 0x72bef550, 0x72363120
Caught exception in looking up space point ptr... 0x856cb910, 0x856cbaa0
Caught exception in looking up space point ptr... 0x856cbaa0, 0x85cf87c0
Caught exception in looking up space point ptr... 0xc7530570, 0xc7530390
Caught exception in looking up space point ptr... 0xc75302a0, 0xc7530570
Caught exception in looking up space point ptr... 0x9932af60, 0x9f246b80
Caught exception in looking up space point ptr... 0x9f246b80, 0x9932af60
Caught exception in looking up space point ptr... 0xdf7cf0d0, 0xe2dc04a0
Caught exception in looking up space point ptr... 0xe2dc04a0, 0xdf7cf0d0
Caught exception in looking up space point ptr... 0xcc706ec0, 0xcd333580
Caught exception in looking up space point ptr... 0xcd333580, 0xcda4fd80
Caught exception in looking up space point ptr... 0xb0ee3640, 0xb122e110
Caught exception in looking up space point ptr... 0xb122e110, 0xb1800a10
Caught exception in looking up space point ptr... 0x7ae4d240, 0x74b72790
Caught exception in looking up space point ptr... 0x74b72790, 0x7ae4d240
Caught exception in looking up space point ptr... 0x7363ec20, 0x6b8de960
Caught exception in looking up space point ptr... 0x6b8de960, 0x64d4e110
++++>>>> total num hits: 8469377, num free: 8264149
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 157.421 515.007 1732.4 239.913 609.61 5
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 7.8019e-05 0.000104353 0.000189818 8.6401e-05 4.28982e-05 5
produce:tpcrawdecoder:PDVDTPCReader 75.8744 118.936 157.1 125.493 34.4979 5
produce:triggerrawdecoder:PDVDTriggerReader4 0.811482 0.88106 0.999786 0.869082 0.0663263 5
produce:timingrawdecoder:PDHDTimingRawDecoder 0.01955 0.029941 0.0432912 0.0285273 0.00970044 5
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0957217 0.0970663 0.0989537 0.096767 0.00105938 5
produce:beamevent:BeamEvent 0.000114294 0.000331179 0.00113875 0.000129507 0.000403972 5
produce:pdvddaphne:DAPHNEReaderPDVD 12.8675 16.7506 20.3214 16.3487 2.71694 5
produce:ophit:OpHitFinder 0.0291435 0.0333994 0.0369926 0.033785 0.00256259 5
produce:wclsdatavd:WireCellToolkit 50.2128 54.09 57.7508 54.2724 2.389 5
produce:gaushit:GausHitFinder 0.747497 3.11701 6.46313 1.19607 2.55189 5
produce:nhitsfilter:NumberOfHitsFilter 0.000229098 0.000476568 0.000791643 0.000423424 0.0002225 5
produce:reco3d:SpacePointSolver 6.90776 44.6743 152.144 9.82264 62.0594 4
produce:hitpdune:DisambigFromSpacePoints 0.100217 1.66034 6.17234 0.184397 2.60524 4
produce:cluster3d:Cluster3D 1.44811 351.381 1335.96 34.0602 568.601 4
[art]:TriggerResults:TriggerResultInserter 3.3261e-05 6.59344e-05 0.000128836 5.8689e-05 3.28061e-05 5
end_path:out1:RootOutput 5.989e-06 1.29236e-05 3.2409e-05 8.924e-06 9.80591e-06 5
end_path:out1:RootOutput(write) 3.38888 3.58762 3.98611 3.48774 0.233703 4
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 17832.2 MB
Peak resident set size usage (VmHWM): 15136.6 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:10:10 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 5 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 