Jobsub ID 297972.5@dunegpschedd02.fnal.gov
| Jobsub ID | 297972.5@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-04 00:40:28 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce01 |
| Last heartbeat | 2026-02-04 01:46:54 |
| From worker node | Hostname | wn6219301.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 00:40:50 |
| Input files | vd-protodune:np02vd_raw_run042421_0050_df-s03-d0_dw_0_20260203T135805.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:46:54 |
| Saved logs | justin-logs:297972.5-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
up space point ptr... 0xca966f90, 0xca042700
Caught exception in looking up space point ptr... 0xca042700, 0xc9ab6540
Caught exception in looking up space point ptr... 0x356cecd0, 0x35b9d300
Caught exception in looking up space point ptr... 0x374eebb0, 0x356cecd0
Caught exception in looking up space point ptr... 0x7da32160, 0x7da322f0
Caught exception in looking up space point ptr... 0x7da322f0, 0x7da31fd0
Caught exception in looking up space point ptr... 0x43927730, 0x438d91d0
Caught exception in looking up space point ptr... 0x438d91d0, 0x43927730
Caught exception in looking up space point ptr... 0x4e228620, 0x58d54610
Caught exception in looking up space point ptr... 0x58d54610, 0x4e228620
Caught exception in looking up space point ptr... 0x71a082c0, 0x389344a0
Caught exception in looking up space point ptr... 0x72d9e5f0, 0x71a082c0
Caught exception in looking up space point ptr... 0x5e2f87e0, 0x5e1df110
Caught exception in looking up space point ptr... 0x5e1df110, 0x5e2f87e0
Caught exception in looking up space point ptr... 0xd19bca30, 0xd1655860
Caught exception in looking up space point ptr... 0xd179b010, 0xd19bca30
Caught exception in looking up space point ptr... 0x55e93ce0, 0x55e93f70
Caught exception in looking up space point ptr... 0x55e93f70, 0x55e93250
Caught exception in looking up space point ptr... 0x75208290, 0x752084e0
Caught exception in looking up space point ptr... 0x752084e0, 0x75208040
Caught exception in looking up space point ptr... 0x18a95690, 0x1bf0c5f0
Caught exception in looking up space point ptr... 0x1c138960, 0x18a95690
Caught exception in looking up space point ptr... 0xa98f4640, 0xa99799d0
Caught exception in looking up space point ptr... 0xa99799d0, 0xa99791e0
Caught exception in looking up space point ptr... 0x62fb85b0, 0x6289d6e0
Caught exception in looking up space point ptr... 0x6289d6e0, 0x62fb85b0
Caught exception in looking up space point ptr... 0xc6446fe0, 0xc8035680
Caught exception in looking up space point ptr... 0xc8035680, 0xc6446fe0
Caught exception in looking up space point ptr... 0xaaa78930, 0xaaa8ce90
Caught exception in looking up space point ptr... 0xaaa8ce90, 0xaaa6a2b0
Caught exception in looking up space point ptr... 0x82320dc0, 0x82d80590
Caught exception in looking up space point ptr... 0x82d80590, 0x828bef40
Caught exception in looking up space point ptr... 0xae1af340, 0xae1af440
Caught exception in looking up space point ptr... 0xae1af440, 0xae1af340
Caught exception in looking up space point ptr... 0xb0c9ef20, 0xb106bb10
Caught exception in looking up space point ptr... 0xb106bb10, 0xb106bbb0
Caught exception in looking up space point ptr... 0x7b8060e0, 0x7b906c20
Caught exception in looking up space point ptr... 0x7b906c20, 0x7b920bf0
Caught exception in looking up space point ptr... 0x59bf81a0, 0x59bf8100
Caught exception in looking up space point ptr... 0x59bf8100, 0x59bf81a0
Caught exception in looking up space point ptr... 0xb1072010, 0xb1071ee0
Caught exception in looking up space point ptr... 0xb1071ee0, 0xb1072010
Caught exception in looking up space point ptr... 0x7c55a900, 0x7eb17680
Caught exception in looking up space point ptr... 0x7eb17680, 0x7c55a900
Caught exception in looking up space point ptr... 0x4b359430, 0x4acec0c0
Caught exception in looking up space point ptr... 0x4acec0c0, 0x4b3586c0
Caught exception in looking up space point ptr... 0x7ddf5280, 0x7ddf44b0
Caught exception in looking up space point ptr... 0x7ddf44b0, 0x7ddf6980
Caught exception in looking up space point ptr... 0xd1b9fa50, 0xd1b9ff70
Caught exception in looking up space point ptr... 0xd1b9fce0, 0xd1b9fa50
Caught exception in looking up space point ptr... 0x258adbb0, 0x258ade80
Caught exception in looking up space point ptr... 0x258ade80, 0x169a9b60
Caught exception in looking up space point ptr... 0xbb606190, 0xbb2a4ba0
Caught exception in looking up space point ptr... 0xbb2a4ba0, 0xbb606190
Caught exception in looking up space point ptr... 0x5c681cf0, 0x5bb76ea0
Caught exception in looking up space point ptr... 0x5bb76ea0, 0x5bb76d10
Caught exception in looking up space point ptr... 0x377120d0, 0x460e3880
Caught exception in looking up space point ptr... 0x377122b0, 0x377120d0
Caught exception in looking up space point ptr... 0x37767920, 0x377112c0
Caught exception in looking up space point ptr... 0x377112c0, 0x37767920
Caught exception in looking up space point ptr... 0x65a362c0, 0x659526d0
Caught exception in looking up space point ptr... 0x659526d0, 0x72c6b8e0
Caught exception in looking up space point ptr... 0x72c6b8e0, 0x7349e1a0
Caught exception in looking up space point ptr... 0x7349e1a0, 0x65ace270
Caught exception in looking up space point ptr... 0x65ace270, 0x65a362c0
Caught exception in looking up space point ptr... 0x3770c1d0, 0x3770bff0
Caught exception in looking up space point ptr... 0x3770bff0, 0x3770c1d0
Caught exception in looking up space point ptr... 0x46892a70, 0x3772da90
Caught exception in looking up space point ptr... 0x3772da90, 0x3772f0c0
Caught exception in looking up space point ptr... 0xa3dc5b30, 0xa1c705b0
Caught exception in looking up space point ptr... 0xa1c705b0, 0xa3dc5b30
Caught exception in looking up space point ptr... 0xaa0af210, 0xa9d314c0
Caught exception in looking up space point ptr... 0xa9d314c0, 0xaa0af210
Caught exception in looking up space point ptr... 0xaa0af210, 0xa88b47d0
Caught exception in looking up space point ptr... 0xa88b47d0, 0xaa0af210
Caught exception in looking up space point ptr... 0x79cf7820, 0x79cf7ab0
Caught exception in looking up space point ptr... 0x79cf7ab0, 0x79cf7590
Caught exception in looking up space point ptr... 0x18e00de0, 0x18e00ac0
Caught exception in looking up space point ptr... 0x18e00ac0, 0x18e00de0
Caught exception in looking up space point ptr... 0x886c3bd0, 0x968ee3a0
Caught exception in looking up space point ptr... 0x968ee3a0, 0x886c1d70
++++>>>> total num hits: 8670273, num free: 8436904
04-Feb-2026 01:45:03 GMT Opened output file with pattern "%ifb_reco_stage0_%tc_offline.root"
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 3720.05 3720.05 3720.05 3720.05 0 1
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000228501 0.000228501 0.000228501 0.000228501 0 1
produce:tpcrawdecoder:PDVDTPCReader 151.953 151.953 151.953 151.953 0 1
produce:triggerrawdecoder:PDVDTriggerReader4 0.775772 0.775772 0.775772 0.775772 0 1
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0401842 0.0401842 0.0401842 0.0401842 0 1
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.111802 0.111802 0.111802 0.111802 0 1
produce:beamevent:BeamEvent 0.00127077 0.00127077 0.00127077 0.00127077 0 1
produce:pdvddaphne:DAPHNEReaderPDVD 20.1701 20.1701 20.1701 20.1701 0 1
produce:ophit:OpHitFinder 0.0608862 0.0608862 0.0608862 0.0608862 0 1
produce:wclsdatavd:WireCellToolkit 142.327 142.327 142.327 142.327 0 1
produce:gaushit:GausHitFinder 21.3956 21.3956 21.3956 21.3956 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.00288495 0.00288495 0.00288495 0.00288495 0 1
produce:reco3d:SpacePointSolver 353.798 353.798 353.798 353.798 0 1
produce:hitpdune:DisambigFromSpacePoints 13.7411 13.7411 13.7411 13.7411 0 1
produce:cluster3d:Cluster3D 3014.82 3014.82 3014.82 3014.82 0 1
[art]:TriggerResults:TriggerResultInserter 0.000215586 0.000215586 0.000215586 0.000215586 0 1
end_path:out1:RootOutput 7.7717e-05 7.7717e-05 7.7717e-05 7.7717e-05 0 1
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 18859.8 MB
Peak resident set size usage (VmHWM): 16438.8 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 01:45:42 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 1 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 