Jobsub ID 303646.147@dunegpschedd01.fnal.gov
| Jobsub ID | 303646.147@dunegpschedd01.fnal.gov |
| Workflow ID | 12718 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:44:24 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce01 |
| Last heartbeat | 2026-02-04 01:48:47 |
| From worker node | Hostname | wn2208251.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7513 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 22:45:44 |
| Input files | vd-protodune:np02vd_raw_run042422_0115_df-s03-d1_dw_0_20260203T161657.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:48:47 |
| Saved logs | justin-logs:303646.147-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
4199320
Caught exception in looking up space point ptr... 0xa54ef0b0, 0xa4e89d50
Caught exception in looking up space point ptr... 0xa4e89d50, 0xa54ef0b0
Caught exception in looking up space point ptr... 0x5dce8d20, 0x3f31c6c0
Caught exception in looking up space point ptr... 0x3f31c6c0, 0x5dce8d20
Caught exception in looking up space point ptr... 0x88f4cb60, 0x88f4ca60
Caught exception in looking up space point ptr... 0x88f4ca60, 0x88f4cb60
Caught exception in looking up space point ptr... 0xb69fd030, 0xb69fcf90
Caught exception in looking up space point ptr... 0xb69fcf90, 0xb6a75c30
Caught exception in looking up space point ptr... 0xb6a598e0, 0xb6a59800
Caught exception in looking up space point ptr... 0xb6a59800, 0xb6a598e0
Caught exception in looking up space point ptr... 0xb6acf180, 0xb6acf0a0
Caught exception in looking up space point ptr... 0xb6acf0a0, 0xb6acf180
Caught exception in looking up space point ptr... 0xabf57450, 0xabf57770
Caught exception in looking up space point ptr... 0xac015e60, 0xabf57450
Caught exception in looking up space point ptr... 0x93bab590, 0x88b6bbf0
Caught exception in looking up space point ptr... 0x88b6bbf0, 0x93bab590
Caught exception in looking up space point ptr... 0x93bab590, 0x88b8a9f0
Caught exception in looking up space point ptr... 0x88b8a9f0, 0x93bab590
Caught exception in looking up space point ptr... 0x93bab590, 0x94bc6400
Caught exception in looking up space point ptr... 0x94bc6400, 0x93bab590
Caught exception in looking up space point ptr... 0x3a0b99a0, 0x350a5350
Caught exception in looking up space point ptr... 0x350a5350, 0x36da34b0
Caught exception in looking up space point ptr... 0x8f9e89c0, 0x8d151310
Caught exception in looking up space point ptr... 0x8d151310, 0x8e80d880
Caught exception in looking up space point ptr... 0x8e80d880, 0x8f2f85c0
Caught exception in looking up space point ptr... 0x8f2f85c0, 0x8f9e89c0
Caught exception in looking up space point ptr... 0xace1ea10, 0xa4eb8b60
Caught exception in looking up space point ptr... 0xa4eb8b60, 0xace1ea10
Caught exception in looking up space point ptr... 0xace1ea10, 0xa8f19c20
Caught exception in looking up space point ptr... 0xa8f19c20, 0xace1ea10
Caught exception in looking up space point ptr... 0x97dfc960, 0x98805960
Caught exception in looking up space point ptr... 0x98805960, 0x8eaa82f0
Caught exception in looking up space point ptr... 0x8eaa82f0, 0x8d7d2220
Caught exception in looking up space point ptr... 0x8fccd250, 0x97dfc960
Caught exception in looking up space point ptr... 0x3a0c94f0, 0x59fd0470
Caught exception in looking up space point ptr... 0x59fd0470, 0x4539d3b0
Caught exception in looking up space point ptr... 0x9a010230, 0x91d02880
Caught exception in looking up space point ptr... 0x91d02880, 0x8f302330
Caught exception in looking up space point ptr... 0x8f302330, 0x8d0e5b60
Caught exception in looking up space point ptr... 0x8f2ba1e0, 0x9a010230
Caught exception in looking up space point ptr... 0xb660c530, 0xb6643290
Caught exception in looking up space point ptr... 0xb6643290, 0xb660c530
Caught exception in looking up space point ptr... 0xa913a0e0, 0xaa5840c0
Caught exception in looking up space point ptr... 0xaa5840c0, 0xa92f0480
Caught exception in looking up space point ptr... 0x7bb24890, 0x7bb5c480
Caught exception in looking up space point ptr... 0x7bb5c480, 0x7bb24890
Caught exception in looking up space point ptr... 0x85c05f00, 0x7b88d430
Caught exception in looking up space point ptr... 0x7bb65b00, 0x85c05f00
Caught exception in looking up space point ptr... 0xc519cba0, 0xc487fdb0
Caught exception in looking up space point ptr... 0xc487fdb0, 0xc4ec3d50
Caught exception in looking up space point ptr... 0xb6789e00, 0xb6789d10
Caught exception in looking up space point ptr... 0xb6789d10, 0xb6789b30
Caught exception in looking up space point ptr... 0x927d5db0, 0x93f40920
Caught exception in looking up space point ptr... 0x93f40920, 0x93f49a20
Caught exception in looking up space point ptr... 0x747940d0, 0x74794170
Caught exception in looking up space point ptr... 0x74794170, 0x7504b350
Caught exception in looking up space point ptr... 0x59830580, 0x257f3c30
Caught exception in looking up space point ptr... 0x257f3c30, 0x2d2c9460
Caught exception in looking up space point ptr... 0xad88d6a0, 0xafc838f0
Caught exception in looking up space point ptr... 0xafc838f0, 0xaf750290
Caught exception in looking up space point ptr... 0x8bb34ae0, 0x8bb34540
Caught exception in looking up space point ptr... 0x8bb34540, 0x8bb34ae0
Caught exception in looking up space point ptr... 0x8e928500, 0x8f4dc0d0
Caught exception in looking up space point ptr... 0x8f4dc0d0, 0x8e928500
Caught exception in looking up space point ptr... 0x4fd2e7d0, 0x4fd36270
Caught exception in looking up space point ptr... 0x4fd36270, 0x4fd33ae0
Caught exception in looking up space point ptr... 0x3d0e2a90, 0x6438a3a0
Caught exception in looking up space point ptr... 0x6438a3a0, 0x3d0e2900
Caught exception in looking up space point ptr... 0x7e9098d0, 0x7e9095d0
Caught exception in looking up space point ptr... 0x7e9095d0, 0x7e9098d0
Caught exception in looking up space point ptr... 0xbb9d0a40, 0xbdcbe920
Caught exception in looking up space point ptr... 0xbdcbe920, 0xbb9d0a40
Caught exception in looking up space point ptr... 0xbc0d09d0, 0xbc28ba20
Caught exception in looking up space point ptr... 0xbc28ba20, 0xbc0d09d0
Caught exception in looking up space point ptr... 0x5986fff0, 0x59870330
Caught exception in looking up space point ptr... 0x59870330, 0x598704d0
Caught exception in looking up space point ptr... 0xb4f65000, 0xb4ebae90
Caught exception in looking up space point ptr... 0xb4ebae90, 0xb4ebd8d0
Caught exception in looking up space point ptr... 0xb416bd80, 0xb416bc50
Caught exception in looking up space point ptr... 0xb416bc50, 0xb416bd80
++++>>>> total num hits: 7446530, num free: 7220997
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 240.186 400.535 2029.54 337.429 323.816 27
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.2564e-05 0.000138751 0.000282501 0.000120937 4.73771e-05 27
produce:tpcrawdecoder:PDVDTPCReader 116.494 201.928 295.489 191.725 47.0863 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.774684 1.23982 6.02489 1.04502 0.953255 27
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0216303 0.028263 0.0734828 0.0229075 0.010768 27
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.10763 0.123119 0.401515 0.109858 0.0550934 27
produce:beamevent:BeamEvent 0.00010115 0.000254132 0.00118727 0.00019734 0.000197296 27
produce:pdvddaphne:DAPHNEReaderPDVD 16.0836 24.9956 39.8349 23.1619 5.922 27
produce:ophit:OpHitFinder 0.0342344 0.0474892 0.0624553 0.0483385 0.00686135 27
produce:wclsdatavd:WireCellToolkit 70.7547 85.4654 98.6376 85.7589 5.8875 27
produce:gaushit:GausHitFinder 0.895622 1.63672 8.79003 1.37997 1.4302 27
produce:nhitsfilter:NumberOfHitsFilter 0.000202461 0.000426602 0.00189863 0.000379984 0.000303851 27
produce:reco3d:SpacePointSolver 7.23957 22.7653 267.374 13.8155 48.1681 27
produce:hitpdune:DisambigFromSpacePoints 0.086338 0.60159 12.6443 0.124615 2.36215 27
produce:cluster3d:Cluster3D 1.85944 56.5799 1451.16 2.69678 273.5 27
[art]:TriggerResults:TriggerResultInserter 2.8183e-05 4.11975e-05 9.6061e-05 3.8663e-05 1.22716e-05 27
end_path:out1:RootOutput 6.773e-06 1.19891e-05 2.5267e-05 1.071e-05 4.72197e-06 27
end_path:out1:RootOutput(write) 4.20004 5.27397 6.00273 5.36342 0.471321 26
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 16930.1 MB
Peak resident set size usage (VmHWM): 14023.2 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 01:47:49 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 26 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 27 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 