Jobsub ID 303653.70@dunegpschedd01.fnal.gov
| Jobsub ID | 303653.70@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:52:24 |
| Site | UK_Oxford |
| Entry | DUNE_UK_SGrid_Oxford_arc01 |
| Last heartbeat | 2026-02-04 00:40:29 |
| From worker node | Hostname | t2wn064.physics.ox.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:41:01 |
| Input files | vd-protodune:np02vd_raw_run042421_0050_df-s04-d1_dw_0_20260203T135805.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 00:40:29 |
| Saved logs | justin-logs:303653.70-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ng up space point ptr... 0xc2c1f7a0, 0xc2c1f670
Caught exception in looking up space point ptr... 0xc2c1f670, 0xc2c8e7e0
Caught exception in looking up space point ptr... 0x6c6cb5e0, 0x8d87a9f0
Caught exception in looking up space point ptr... 0x8d87a9f0, 0x8bf27bf0
Caught exception in looking up space point ptr... 0xb1cdc260, 0xb1cbd750
Caught exception in looking up space point ptr... 0xb1cbd750, 0xb1bd8700
Caught exception in looking up space point ptr... 0xb42e6ed0, 0xb4389110
Caught exception in looking up space point ptr... 0xb4389110, 0xb4384040
Caught exception in looking up space point ptr... 0xc307f3e0, 0xc307f340
Caught exception in looking up space point ptr... 0xc307f340, 0xc307f3e0
Caught exception in looking up space point ptr... 0x295772d0, 0x4b8e1d10
Caught exception in looking up space point ptr... 0x41646140, 0x295772d0
Caught exception in looking up space point ptr... 0x53f1a330, 0x548e97e0
Caught exception in looking up space point ptr... 0x548e97e0, 0x548e9970
Caught exception in looking up space point ptr... 0xb659c550, 0xb659bda0
Caught exception in looking up space point ptr... 0xb659bda0, 0xb659c2c0
Caught exception in looking up space point ptr... 0xe5c30d50, 0xe5c30a80
Caught exception in looking up space point ptr... 0xe5c30b70, 0xe5c30d50
Caught exception in looking up space point ptr... 0x90c3f210, 0x7f580380
Caught exception in looking up space point ptr... 0x7f580380, 0x857b28c0
Caught exception in looking up space point ptr... 0x8bae3ca0, 0x8bae41c0
Caught exception in looking up space point ptr... 0x8bae3f30, 0x8bae3ca0
Caught exception in looking up space point ptr... 0x367666d0, 0x33892bc0
Caught exception in looking up space point ptr... 0x33892bc0, 0x367666d0
Caught exception in looking up space point ptr... 0x5e838cd0, 0x5e84d000
Caught exception in looking up space point ptr... 0x5e84d000, 0x5e842540
Caught exception in looking up space point ptr... 0x8fa63730, 0x8fa639c0
Caught exception in looking up space point ptr... 0x8fa639c0, 0x8fa634a0
Caught exception in looking up space point ptr... 0xc219e650, 0xc1f1e940
Caught exception in looking up space point ptr... 0xc1f1e940, 0xc219e650
Caught exception in looking up space point ptr... 0xe545e9f0, 0xe5480190
Caught exception in looking up space point ptr... 0xe54660e0, 0xe545e9f0
Caught exception in looking up space point ptr... 0x71e2bef0, 0x6f95ffd0
Caught exception in looking up space point ptr... 0x6f95ffd0, 0x71993280
Caught exception in looking up space point ptr... 0x71993280, 0x71992ff0
Caught exception in looking up space point ptr... 0x71992ff0, 0x739bfd50
Caught exception in looking up space point ptr... 0xe5756440, 0xca3df8b0
Caught exception in looking up space point ptr... 0xca3df8b0, 0xe5756440
Caught exception in looking up space point ptr... 0x312d4bd0, 0x312d4d60
Caught exception in looking up space point ptr... 0x312d4d60, 0x312d4a40
Caught exception in looking up space point ptr... 0x312d4a40, 0x2b051800
Caught exception in looking up space point ptr... 0x2b051800, 0x2b051900
Caught exception in looking up space point ptr... 0x6aa994f0, 0x6b5d7a10
Caught exception in looking up space point ptr... 0x6b5d7a10, 0x6b3be0b0
Caught exception in looking up space point ptr... 0x8e660e40, 0x8e660cb0
Caught exception in looking up space point ptr... 0x8e660cb0, 0x8e660e40
Caught exception in looking up space point ptr... 0x2d6b2360, 0x298ca500
Caught exception in looking up space point ptr... 0x2d6b2460, 0x2d6b2360
Caught exception in looking up space point ptr... 0x2aed2850, 0x2aed26c0
Caught exception in looking up space point ptr... 0x2aed26c0, 0x316629d0
Caught exception in looking up space point ptr... 0x23766370, 0x23765bc0
Caught exception in looking up space point ptr... 0x23e1baf0, 0x23766370
Caught exception in looking up space point ptr... 0xe4a82280, 0xe4a825a0
Caught exception in looking up space point ptr... 0xe4a825a0, 0xe4a82280
Caught exception in looking up space point ptr... 0xe260c790, 0xdf399cc0
Caught exception in looking up space point ptr... 0xdf399cc0, 0xe260c790
Caught exception in looking up space point ptr... 0xe260c790, 0xe2e289c0
Caught exception in looking up space point ptr... 0xe2e289c0, 0xe260c790
Caught exception in looking up space point ptr... 0x293d6750, 0x49f0a590
Caught exception in looking up space point ptr... 0x49f0a590, 0x293d6750
Caught exception in looking up space point ptr... 0xc2ed2280, 0xc3005c40
Caught exception in looking up space point ptr... 0xc3005c40, 0xc2ffb7e0
Caught exception in looking up space point ptr... 0xc2ffb7e0, 0xc2fe72d0
Caught exception in looking up space point ptr... 0xb644a0e0, 0xb67bc9e0
Caught exception in looking up space point ptr... 0xb67bc9e0, 0xb5d78840
Caught exception in looking up space point ptr... 0xc63af220, 0xcb8664f0
Caught exception in looking up space point ptr... 0xcb8664f0, 0xc63af220
Caught exception in looking up space point ptr... 0xc2eb28b0, 0xc3086950
Caught exception in looking up space point ptr... 0xc3086950, 0xc30869f0
Caught exception in looking up space point ptr... 0xd83d9aa0, 0xd83eb970
Caught exception in looking up space point ptr... 0xd83eb970, 0xd83d9aa0
Caught exception in looking up space point ptr... 0xcb7e5ca0, 0xcd2039d0
Caught exception in looking up space point ptr... 0xcd2039d0, 0xcb7e5ca0
Caught exception in looking up space point ptr... 0xb8271470, 0xb7f62540
Caught exception in looking up space point ptr... 0xb7f62540, 0xb8271470
Caught exception in looking up space point ptr... 0xbef715a0, 0xbef714b0
Caught exception in looking up space point ptr... 0xbef714b0, 0xbef71780
Caught exception in looking up space point ptr... 0x9151f500, 0x8db1d230
Caught exception in looking up space point ptr... 0x920b0920, 0x9151f500
Caught exception in looking up space point ptr... 0xa6238250, 0xa6dd14b0
Caught exception in looking up space point ptr... 0xa6dd14b0, 0xa6238250
++++>>>> total num hits: 9875559, num free: 9617349
04-Feb-2026 00:38:18 GMT Opened output file with pattern "%ifb_reco_stage0_%tc_offline.root"
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 3298.71 3298.71 3298.71 3298.71 0 1
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000348131 0.000348131 0.000348131 0.000348131 0 1
produce:tpcrawdecoder:PDVDTPCReader 156.81 156.81 156.81 156.81 0 1
produce:triggerrawdecoder:PDVDTriggerReader4 0.891641 0.891641 0.891641 0.891641 0 1
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0338883 0.0338883 0.0338883 0.0338883 0 1
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0994184 0.0994184 0.0994184 0.0994184 0 1
produce:beamevent:BeamEvent 0.00150085 0.00150085 0.00150085 0.00150085 0 1
produce:pdvddaphne:DAPHNEReaderPDVD 23.1164 23.1164 23.1164 23.1164 0 1
produce:ophit:OpHitFinder 0.0413541 0.0413541 0.0413541 0.0413541 0 1
produce:wclsdatavd:WireCellToolkit 97.1259 97.1259 97.1259 97.1259 0 1
produce:gaushit:GausHitFinder 15.6813 15.6813 15.6813 15.6813 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.00207654 0.00207654 0.00207654 0.00207654 0 1
produce:reco3d:SpacePointSolver 341.757 341.757 341.757 341.757 0 1
produce:hitpdune:DisambigFromSpacePoints 11.7737 11.7737 11.7737 11.7737 0 1
produce:cluster3d:Cluster3D 2651.29 2651.29 2651.29 2651.29 0 1
[art]:TriggerResults:TriggerResultInserter 0.000202909 0.000202909 0.000202909 0.000202909 0 1
end_path:out1:RootOutput 6.423e-05 6.423e-05 6.423e-05 6.423e-05 0 1
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 20379.5 MB
Peak resident set size usage (VmHWM): 17895 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:38:51 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 1 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 