Jobsub ID 297920.106@dunegpschedd02.fnal.gov
| Jobsub ID | 297920.106@dunegpschedd02.fnal.gov |
| Workflow ID | 12719 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:54:27 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce04 |
| Last heartbeat | 2026-02-04 01:50:12 |
| From worker node | Hostname | dune001-1520101.0-lcg2568.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:42:19 |
| Input files | vd-protodune:np02vd_raw_run042433_0048_df-s02-d1_dw_0_20260203T190020.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:50:12 |
| Saved logs | justin-logs:297920.106-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
x11c987bb0
Caught exception in looking up space point ptr... 0x11c987bb0, 0x11c988050
Caught exception in looking up space point ptr... 0x6b44bb80, 0x6b44b380
Caught exception in looking up space point ptr... 0x6b44b380, 0x6b44b8f0
Caught exception in looking up space point ptr... 0x11e7494d0, 0x11e6520a0
Caught exception in looking up space point ptr... 0x11e651e10, 0x11e7494d0
Caught exception in looking up space point ptr... 0xfbb5b340, 0xfbb5b590
Caught exception in looking up space point ptr... 0xfbb5b590, 0xfbb5c1c0
Caught exception in looking up space point ptr... 0x1750857f0, 0x183dbec40
Caught exception in looking up space point ptr... 0x183dbec40, 0x184890b20
Caught exception in looking up space point ptr... 0x12ce5e490, 0x12ce5e930
Caught exception in looking up space point ptr... 0x12ce5e6e0, 0x12ce5e490
Caught exception in looking up space point ptr... 0x11b681cf0, 0x61cfb700
Caught exception in looking up space point ptr... 0x61cfb700, 0x103127060
Caught exception in looking up space point ptr... 0x103127060, 0x61cfb700
Caught exception in looking up space point ptr... 0x61cfb700, 0x9c85d970
Caught exception in looking up space point ptr... 0xee8e9c20, 0xee8f6d10
Caught exception in looking up space point ptr... 0xee8ebb40, 0xee8e9c20
Caught exception in looking up space point ptr... 0xe75614c0, 0xe7cf6d40
Caught exception in looking up space point ptr... 0xe7cf6d40, 0x11ceb8760
Caught exception in looking up space point ptr... 0x11ceb8760, 0xe79df1f0
Caught exception in looking up space point ptr... 0xe79df1f0, 0xe75614c0
Caught exception in looking up space point ptr... 0xb7bf3800, 0xb7b5c880
Caught exception in looking up space point ptr... 0xb7b5c880, 0xb7bac0c0
Caught exception in looking up space point ptr... 0x1279d4ea0, 0x127839cd0
Caught exception in looking up space point ptr... 0x127839cd0, 0x1279d4d50
Caught exception in looking up space point ptr... 0x162243700, 0xa97d63b0
Caught exception in looking up space point ptr... 0xa97d63b0, 0x162243700
Caught exception in looking up space point ptr... 0xf83478d0, 0xf83e8b80
Caught exception in looking up space point ptr... 0xf83e8b80, 0xf85fe210
Caught exception in looking up space point ptr... 0xf85fe210, 0xf8b6e3b0
Caught exception in looking up space point ptr... 0xf8b6e3b0, 0xf83478d0
Caught exception in looking up space point ptr... 0x1894b1bd0, 0x1894c7d10
Caught exception in looking up space point ptr... 0x1894c7d10, 0x1894b1d60
Caught exception in looking up space point ptr... 0x15984b240, 0x9f014470
Caught exception in looking up space point ptr... 0x9f014470, 0x9f00eee0
Caught exception in looking up space point ptr... 0x100b5e100, 0x100b5e390
Caught exception in looking up space point ptr... 0x100b5de70, 0x100b5e100
Caught exception in looking up space point ptr... 0x18404c3c0, 0xb7388b50
Caught exception in looking up space point ptr... 0xb7388b50, 0x18404c3c0
Caught exception in looking up space point ptr... 0x120b97bc0, 0x1207e4830
Caught exception in looking up space point ptr... 0x1207e4830, 0x120b97bc0
Caught exception in looking up space point ptr... 0x1846b3c90, 0x1846b3b90
Caught exception in looking up space point ptr... 0x1846b3b90, 0x1846b3c90
Caught exception in looking up space point ptr... 0x9ff92670, 0x15eb80590
Caught exception in looking up space point ptr... 0x15eb80590, 0x15eb800e0
Caught exception in looking up space point ptr... 0x15eb80400, 0x9ff92570
Caught exception in looking up space point ptr... 0x9ff92570, 0x15eb80270
Caught exception in looking up space point ptr... 0x15eb80270, 0x9ff92670
Caught exception in looking up space point ptr... 0x11ac73060, 0x16d1013a0
Caught exception in looking up space point ptr... 0x16d1013a0, 0x13ca25260
Caught exception in looking up space point ptr... 0xe0e12ec0, 0xe0872ab0
Caught exception in looking up space point ptr... 0xe0872ab0, 0xe0872980
Caught exception in looking up space point ptr... 0xe0872850, 0xe0e12d90
Caught exception in looking up space point ptr... 0xe0e12d90, 0xe0e12ec0
Caught exception in looking up space point ptr... 0x11a272fb0, 0x16d3cedb0
Caught exception in looking up space point ptr... 0x16d3cedb0, 0x11a272fb0
Caught exception in looking up space point ptr... 0xce1b9da0, 0xaa6685e0
Caught exception in looking up space point ptr... 0xaa6685e0, 0xce1b9da0
Caught exception in looking up space point ptr... 0xce1b9da0, 0x8bd2f880
Caught exception in looking up space point ptr... 0x8bd2f880, 0xce1b9da0
Caught exception in looking up space point ptr... 0x87aea860, 0x4d8752c0
Caught exception in looking up space point ptr... 0x4d8752c0, 0x4d875450
Caught exception in looking up space point ptr... 0x12eb4eaa0, 0x158ff9b60
Caught exception in looking up space point ptr... 0x158ff9b60, 0x158ff99d0
Caught exception in looking up space point ptr... 0x158ff99d0, 0x158ff9cf0
Caught exception in looking up space point ptr... 0x1404ac7a0, 0x176ba74e0
Caught exception in looking up space point ptr... 0x176ba74e0, 0x4daaf720
Caught exception in looking up space point ptr... 0xb897a5d0, 0x178b81720
Caught exception in looking up space point ptr... 0x154d18340, 0xb897a5d0
Caught exception in looking up space point ptr... 0xfa078210, 0xfa078530
Caught exception in looking up space point ptr... 0xfa078530, 0xfa078210
Caught exception in looking up space point ptr... 0x92e776a0, 0x92e77550
Caught exception in looking up space point ptr... 0x92e77550, 0x92e776a0
Caught exception in looking up space point ptr... 0x9b3aa470, 0x9b0cb0e0
Caught exception in looking up space point ptr... 0x9b0cb0e0, 0x17863fda0
Caught exception in looking up space point ptr... 0x63a476a0, 0x63a47ba0
Caught exception in looking up space point ptr... 0x63a47ba0, 0x63a476a0
Caught exception in looking up space point ptr... 0x63a476a0, 0x63a47a10
Caught exception in looking up space point ptr... 0x63a47a10, 0x63a476a0
++++>>>> total num hits: 7875340, num free: 7688631
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 210.952 497.105 2284.2 238.188 580.806 15
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000101272 0.000154358 0.000342517 0.000120852 6.82517e-05 15
produce:tpcrawdecoder:PDVDTPCReader 16.7594 17.4955 18.8661 17.5506 0.51853 15
produce:triggerrawdecoder:PDVDTriggerReader4 0.688134 0.702721 0.734164 0.698388 0.0122842 15
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0232339 0.0236416 0.024496 0.0235771 0.000326184 15
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0230663 0.0238409 0.0270071 0.0234771 0.000916774 15
produce:beamevent:BeamEvent 0.000142983 0.000341915 0.00125099 0.000257346 0.000276419 15
produce:pdvddaphne:DAPHNEReaderPDVD 4.01215 4.41524 4.96687 4.31617 0.264362 15
produce:ophit:OpHitFinder 0.0304893 0.0502115 0.0669618 0.0511837 0.00774874 15
produce:wclsdatavd:WireCellToolkit 149.089 175.384 209.081 171.101 14.8833 15
produce:gaushit:GausHitFinder 1.49691 6.53643 28.4148 2.45604 9.13991 15
produce:nhitsfilter:NumberOfHitsFilter 0.000359418 0.000863938 0.00196575 0.000646253 0.000475545 15
produce:reco3d:SpacePointSolver 12.9342 59.6313 279.113 29.9585 72.2349 15
produce:hitpdune:DisambigFromSpacePoints 0.179381 2.12406 12.9141 0.357569 4.00908 15
produce:cluster3d:Cluster3D 1.52881 220.464 1737.23 4.53639 483.776 15
[art]:TriggerResults:TriggerResultInserter 3.1381e-05 5.39016e-05 0.000126272 3.577e-05 3.3129e-05 15
end_path:out1:RootOutput 7.31e-06 1.24969e-05 3.2901e-05 8.26e-06 7.85266e-06 15
end_path:out1:RootOutput(write) 7.76706 10.8969 32.5927 8.49603 6.55654 14
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 20021.8 MB
Peak resident set size usage (VmHWM): 17214.9 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 01:48:50 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 14 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 15 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 