Jobsub ID 297922.15@dunegpschedd02.fnal.gov
| Jobsub ID | 297922.15@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 23:00:24 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce04 |
| Last heartbeat | 2026-02-04 01:41:10 |
| From worker node | Hostname | dune001-1520392.0-lcg2703.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 9654 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:57:20 |
| Input files | vd-protodune:np02vd_raw_run042421_0054_df-s02-d0_dw_0_20260203T140013.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:41:10 |
| Saved logs | justin-logs:297922.15-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
looking up space point ptr... 0x12b6af630, 0x16e55cfa0
Caught exception in looking up space point ptr... 0x12b6edcd0, 0x12b6ee770
Caught exception in looking up space point ptr... 0x12b6ee770, 0x12b743690
Caught exception in looking up space point ptr... 0x12b6fa560, 0x12b6d2b20
Caught exception in looking up space point ptr... 0x12b6d2b20, 0x12b6df170
Caught exception in looking up space point ptr... 0x16001f2b0, 0x16003bf40
Caught exception in looking up space point ptr... 0x160010f10, 0x16001f2b0
Caught exception in looking up space point ptr... 0x16009bc60, 0x16009b7c0
Caught exception in looking up space point ptr... 0x16009b570, 0x16009bc60
Caught exception in looking up space point ptr... 0x14be59220, 0x14be8b0d0
Caught exception in looking up space point ptr... 0x14be59450, 0x14be59220
Caught exception in looking up space point ptr... 0x4cb23990, 0x4a3fa730
Caught exception in looking up space point ptr... 0x4a3fa730, 0x32748e80
Caught exception in looking up space point ptr... 0x154cf4600, 0x154ccfa80
Caught exception in looking up space point ptr... 0x154cfa9a0, 0x154cf4600
Caught exception in looking up space point ptr... 0x15373c660, 0x1533fd690
Caught exception in looking up space point ptr... 0x1534b37f0, 0x15373c660
Caught exception in looking up space point ptr... 0x106fe1f80, 0x106fe1cf0
Caught exception in looking up space point ptr... 0x106fe1cf0, 0x106fe2210
Caught exception in looking up space point ptr... 0x129137bc0, 0x129137e10
Caught exception in looking up space point ptr... 0x129137e10, 0x129137920
Caught exception in looking up space point ptr... 0x12aa90530, 0x12a8c5ac0
Caught exception in looking up space point ptr... 0x12a8c5ac0, 0x18a122c00
Caught exception in looking up space point ptr... 0x12914d940, 0x129343f50
Caught exception in looking up space point ptr... 0x129345f20, 0x12914d940
Caught exception in looking up space point ptr... 0x17d7db590, 0x17d7dba30
Caught exception in looking up space point ptr... 0x17d7db7e0, 0x17d7db590
Caught exception in looking up space point ptr... 0x6d9104a0, 0x6d910730
Caught exception in looking up space point ptr... 0x6d910730, 0x6d9104a0
Caught exception in looking up space point ptr... 0x1e830380, 0x1e8300f0
Caught exception in looking up space point ptr... 0x1e8300f0, 0x1e82fe60
Caught exception in looking up space point ptr... 0x187daa360, 0x18795fd10
Caught exception in looking up space point ptr... 0x1374e2950, 0x187daa360
Caught exception in looking up space point ptr... 0x2d028030, 0x2d027ea0
Caught exception in looking up space point ptr... 0x2d027ea0, 0x322b9930
Caught exception in looking up space point ptr... 0x6c39daf0, 0x39640970
Caught exception in looking up space point ptr... 0x39640970, 0x3d14d410
Caught exception in looking up space point ptr... 0x163d2bbf0, 0x163d2ba60
Caught exception in looking up space point ptr... 0x163d2ba60, 0x163d2bbf0
Caught exception in looking up space point ptr... 0x130ad7f70, 0x130ad7d10
Caught exception in looking up space point ptr... 0x130ad7d10, 0x130ad7be0
Caught exception in looking up space point ptr... 0x1814248d0, 0xdb72b4d0
Caught exception in looking up space point ptr... 0xdb72b4d0, 0x1814248d0
Caught exception in looking up space point ptr... 0xd93ff440, 0x12f42fd30
Caught exception in looking up space point ptr... 0x12f42fd30, 0xd93ff440
Caught exception in looking up space point ptr... 0xff99ef70, 0x112879a40
Caught exception in looking up space point ptr... 0x112879a40, 0xd6257430
Caught exception in looking up space point ptr... 0x316a2190, 0x129f9d680
Caught exception in looking up space point ptr... 0x129f9d680, 0x316a2190
Caught exception in looking up space point ptr... 0x316a2190, 0x129f9d4f0
Caught exception in looking up space point ptr... 0x129f9d4f0, 0x316a2190
Caught exception in looking up space point ptr... 0xe5f711d0, 0x124bcad50
Caught exception in looking up space point ptr... 0xe5f70f40, 0xe5f711d0
Caught exception in looking up space point ptr... 0xee28bed0, 0xee28c010
Caught exception in looking up space point ptr... 0xee28c010, 0xee28c150
Caught exception in looking up space point ptr... 0x100598e50, 0x133378730
Caught exception in looking up space point ptr... 0x133378730, 0x133376520
Caught exception in looking up space point ptr... 0x1a5d8b20, 0x181ef2770
Caught exception in looking up space point ptr... 0x181ef2770, 0x61351510
Caught exception in looking up space point ptr... 0x173513fd0, 0x1e6c0300
Caught exception in looking up space point ptr... 0x1e6c0300, 0x173514440
Caught exception in looking up space point ptr... 0x173514440, 0x173514120
Caught exception in looking up space point ptr... 0x173514120, 0x1e6c0490
Caught exception in looking up space point ptr... 0x6c7a9420, 0x1572a58b0
Caught exception in looking up space point ptr... 0x1572a58b0, 0x1429b1a60
Caught exception in looking up space point ptr... 0x1735153e0, 0x173515570
Caught exception in looking up space point ptr... 0x173515570, 0x1e6c07b0
Caught exception in looking up space point ptr... 0x1611fbf30, 0x17672cc20
Caught exception in looking up space point ptr... 0x17672cc20, 0x17672c940
Caught exception in looking up space point ptr... 0xee27e690, 0xee27e550
Caught exception in looking up space point ptr... 0xee27e550, 0xee27e690
Caught exception in looking up space point ptr... 0x2dc399d0, 0x61dd2db0
Caught exception in looking up space point ptr... 0x61dd2db0, 0x2dc399d0
Caught exception in looking up space point ptr... 0x11d35a4e0, 0xd703fd00
Caught exception in looking up space point ptr... 0xd703fd00, 0x11d35a4e0
Caught exception in looking up space point ptr... 0x170f984e0, 0x170f96920
Caught exception in looking up space point ptr... 0x170f96920, 0x1710013c0
Caught exception in looking up space point ptr... 0x119dc5f20, 0x1feb6f10
Caught exception in looking up space point ptr... 0x1feb6f10, 0x1feb31a0
++++>>>> total num hits: 7363267, num free: 7173734
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 84.2628 216.736 1284.21 99.8915 311.972 28
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 7.6376e-05 0.00013831 0.000276408 0.000113627 5.29888e-05 28
produce:tpcrawdecoder:PDVDTPCReader 13.7547 15.3049 18.5514 14.5169 1.6105 28
produce:triggerrawdecoder:PDVDTriggerReader4 0.679176 0.702621 0.796155 0.703809 0.0216728 28
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0227518 0.0241697 0.0400187 0.0230817 0.00387394 28
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.112937 0.122926 0.138137 0.114947 0.0110956 28
produce:beamevent:BeamEvent 0.000111499 0.000218982 0.00110133 0.000167058 0.000207397 28
produce:pdvddaphne:DAPHNEReaderPDVD 3.2226 3.65923 3.97717 3.68613 0.179114 28
produce:ophit:OpHitFinder 0.0276646 0.0408667 0.0502656 0.041574 0.00464478 28
produce:wclsdatavd:WireCellToolkit 49.3814 65.866 80.7498 66.0554 6.1446 28
produce:gaushit:GausHitFinder 0.527405 1.76187 6.8582 0.944369 1.84045 28
produce:nhitsfilter:NumberOfHitsFilter 0.00013245 0.000420274 0.00129834 0.000242862 0.000372879 28
produce:reco3d:SpacePointSolver 3.68526 23.7792 174.381 6.90591 41.7622 27
produce:hitpdune:DisambigFromSpacePoints 0.0559335 1.26709 11.1805 0.0900419 2.94705 27
produce:cluster3d:Cluster3D 0.588029 103.436 1010.62 1.46399 267.631 27
[art]:TriggerResults:TriggerResultInserter 2.61e-05 4.55405e-05 0.000155145 3.59445e-05 2.64171e-05 28
end_path:out1:RootOutput 6.64e-06 1.0351e-05 3.5594e-05 8.2975e-06 6.02949e-06 28
end_path:out1:RootOutput(write) 2.83388 5.51748 25.061 4.00819 4.9958 27
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 18510.3 MB
Peak resident set size usage (VmHWM): 15547.9 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 01:40:07 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 27 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 28 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 