Jobsub ID 297919.119@dunegpschedd02.fnal.gov
| Jobsub ID | 297919.119@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:54:25 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce01 |
| Last heartbeat | 2026-02-04 00:32:12 |
| From worker node | Hostname | wn1208241.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:31:29 |
| Input files | vd-protodune:np02vd_raw_run042421_0006_df-s03-d1_dw_0_20260203T133408.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 00:32:12 |
| Saved logs | justin-logs:297919.119-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
x8a4abb80
Caught exception in looking up space point ptr... 0x348f0700, 0x348f0570
Caught exception in looking up space point ptr... 0x348f0570, 0x16c4fff0
Caught exception in looking up space point ptr... 0xb7ea6cf0, 0xb7ea8060
Caught exception in looking up space point ptr... 0xb7ea8060, 0xb7ea7dd0
Caught exception in looking up space point ptr... 0x2be258f0, 0x79791280
Caught exception in looking up space point ptr... 0x79791280, 0xa29965f0
Caught exception in looking up space point ptr... 0xd1116290, 0xcdd4f820
Caught exception in looking up space point ptr... 0xcdd4f820, 0xcd5c4c50
Caught exception in looking up space point ptr... 0x4c5a6390, 0x4c58d860
Caught exception in looking up space point ptr... 0x4c58d860, 0x4c86b330
Caught exception in looking up space point ptr... 0x4c86b330, 0x4c86b4c0
Caught exception in looking up space point ptr... 0x4c86b4c0, 0x4c5a6390
Caught exception in looking up space point ptr... 0xd9e16740, 0xd9e165f0
Caught exception in looking up space point ptr... 0xd9e165f0, 0xd9e16740
Caught exception in looking up space point ptr... 0xd9e16740, 0xd9e198f0
Caught exception in looking up space point ptr... 0xd9e198f0, 0xd9e16740
Caught exception in looking up space point ptr... 0x722be430, 0x722c3e70
Caught exception in looking up space point ptr... 0x722c3e70, 0x722c30e0
Caught exception in looking up space point ptr... 0x8be8a6b0, 0x8b969f30
Caught exception in looking up space point ptr... 0x9e67f360, 0x8be8a6b0
Caught exception in looking up space point ptr... 0x63228500, 0x631c7ee0
Caught exception in looking up space point ptr... 0x631c7ee0, 0x632372b0
Caught exception in looking up space point ptr... 0xca722470, 0xca272510
Caught exception in looking up space point ptr... 0xca272510, 0xca722470
Caught exception in looking up space point ptr... 0x65d5e750, 0x667b2cb0
Caught exception in looking up space point ptr... 0xa7ce2fd0, 0x65d5e750
Caught exception in looking up space point ptr... 0x43ee0c60, 0x43db1ae0
Caught exception in looking up space point ptr... 0x43db1ae0, 0x7a647460
Caught exception in looking up space point ptr... 0xd4ec8410, 0xc8581c10
Caught exception in looking up space point ptr... 0xc8581c10, 0xcbac9ab0
Caught exception in looking up space point ptr... 0xb50a9360, 0xb50a8fe0
Caught exception in looking up space point ptr... 0xb50a8e20, 0xb50a9360
Caught exception in looking up space point ptr... 0xa447fe40, 0xa281f2a0
Caught exception in looking up space point ptr... 0xa281f2a0, 0x1a6dfac0
Caught exception in looking up space point ptr... 0x7bef73c0, 0x7c289520
Caught exception in looking up space point ptr... 0x7c2897b0, 0x7bef73c0
Caught exception in looking up space point ptr... 0xb748c680, 0xb7490ed0
Caught exception in looking up space point ptr... 0xb7490ed0, 0xb748c680
Caught exception in looking up space point ptr... 0x78d5f970, 0x885cd740
Caught exception in looking up space point ptr... 0x885cd740, 0x9473c290
Caught exception in looking up space point ptr... 0xccd15d60, 0xd3256280
Caught exception in looking up space point ptr... 0xd3256280, 0xccd15d60
Caught exception in looking up space point ptr... 0x3fc1b7f0, 0x3fc1b660
Caught exception in looking up space point ptr... 0x3fc1b660, 0x6dc5a060
Caught exception in looking up space point ptr... 0xb4a2c7c0, 0xb487adb0
Caught exception in looking up space point ptr... 0xb487adb0, 0xb487a750
Caught exception in looking up space point ptr... 0x367e87d0, 0x36730e30
Caught exception in looking up space point ptr... 0x36730e30, 0x80c4d090
Caught exception in looking up space point ptr... 0xc2666050, 0xc2665e80
Caught exception in looking up space point ptr... 0xc2666220, 0xc2666050
Caught exception in looking up space point ptr... 0x8833a720, 0x88403a20
Caught exception in looking up space point ptr... 0x88403a20, 0x5702f000
Caught exception in looking up space point ptr... 0x63d8b450, 0x64672f00
Caught exception in looking up space point ptr... 0x64672f00, 0x64671e20
Caught exception in looking up space point ptr... 0x9cbdbaa0, 0x9cd7d6d0
Caught exception in looking up space point ptr... 0x9cd8aca0, 0x9cbdbaa0
Caught exception in looking up space point ptr... 0xbfc38e60, 0xbfaeeeb0
Caught exception in looking up space point ptr... 0xbfaeeeb0, 0xbfaeefe0
Caught exception in looking up space point ptr... 0xd465a2c0, 0xcd1dbef0
Caught exception in looking up space point ptr... 0xcd1dbef0, 0xd465a2c0
Caught exception in looking up space point ptr... 0xb9aa5ac0, 0xb8c9ff20
Caught exception in looking up space point ptr... 0xb8c9ff20, 0xb9aa5ac0
Caught exception in looking up space point ptr... 0x727ce050, 0x5d27ae30
Caught exception in looking up space point ptr... 0x5d27ae30, 0x727ce050
Caught exception in looking up space point ptr... 0xbfad8710, 0xbfc7d2f0
Caught exception in looking up space point ptr... 0xbfc7d2f0, 0xbf9d1de0
Caught exception in looking up space point ptr... 0x5716ca20, 0x5716cbb0
Caught exception in looking up space point ptr... 0x5716cbb0, 0x5716cd40
Caught exception in looking up space point ptr... 0x59f08440, 0x59f08340
Caught exception in looking up space point ptr... 0x59f08340, 0x59f08440
Caught exception in looking up space point ptr... 0x3da887d0, 0x7bf2a610
Caught exception in looking up space point ptr... 0x7bf2a610, 0x3da887d0
Caught exception in looking up space point ptr... 0xb1042ed0, 0xaf36d830
Caught exception in looking up space point ptr... 0xaf36d830, 0xb6074e00
Caught exception in looking up space point ptr... 0xb0dad5b0, 0xb0d10470
Caught exception in looking up space point ptr... 0xb0d10470, 0xb0e02200
Caught exception in looking up space point ptr... 0xbfae17e0, 0xbfd0f930
Caught exception in looking up space point ptr... 0xbfd0f930, 0xbfc38a00
Caught exception in looking up space point ptr... 0xb3114eb0, 0xb3114d20
Caught exception in looking up space point ptr... 0xb3114d20, 0xb3114eb0
++++>>>> total num hits: 7513540, num free: 7344430
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 166.575 571.913 2115 184.645 707.905 6
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.8907e-05 0.000134567 0.000302259 0.000101556 7.50105e-05 6
produce:tpcrawdecoder:PDVDTPCReader 15.7402 17.1043 19.8575 16.5709 1.39164 6
produce:triggerrawdecoder:PDVDTriggerReader4 0.694638 0.751384 0.785534 0.758726 0.0276242 6
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0222964 0.0225428 0.0230348 0.0224561 0.000246403 6
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.110741 0.118133 0.132781 0.111607 0.00985517 6
produce:beamevent:BeamEvent 0.000100459 0.000369362 0.00144565 0.000146411 0.000483108 6
produce:pdvddaphne:DAPHNEReaderPDVD 4.94875 5.05777 5.19515 5.06571 0.0797396 6
produce:ophit:OpHitFinder 0.0504869 0.0567452 0.0642068 0.0562459 0.00424921 6
produce:wclsdatavd:WireCellToolkit 100.387 122.385 138.171 125.952 11.8787 6
produce:gaushit:GausHitFinder 1.33281 5.35256 9.60087 5.06679 3.6794 6
produce:nhitsfilter:NumberOfHitsFilter 0.000440099 0.00116463 0.00223023 0.00113705 0.000626549 6
produce:reco3d:SpacePointSolver 17.2052 70.0022 227.356 25.1598 80.2407 5
produce:hitpdune:DisambigFromSpacePoints 0.181776 2.48316 7.62581 0.197072 3.00697 5
produce:cluster3d:Cluster3D 3.41215 423.106 1746.57 4.98424 675.628 5
[art]:TriggerResults:TriggerResultInserter 3.4355e-05 5.56282e-05 0.000128722 4.1228e-05 3.32219e-05 6
end_path:out1:RootOutput 5.029e-06 1.45942e-05 6.1305e-05 5.2e-06 2.08906e-05 6
end_path:out1:RootOutput(write) 6.83962 9.64466 19.1667 7.18128 4.7761 5
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 16687.7 MB
Peak resident set size usage (VmHWM): 14425.1 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:31:17 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 5 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 6 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 