Jobsub ID 303656.2@dunegpschedd01.fnal.gov
| Jobsub ID | 303656.2@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:56:25 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce02 |
| Last heartbeat | 2026-02-04 02:34:56 |
| From worker node | Hostname | wn1209250.tier2.hep.manchester.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 00:11:00 |
| Input files | vd-protodune:np02vd_raw_run042421_0011_df-s02-d1_dw_0_20260203T133653.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 02:34:56 |
| Saved logs | justin-logs:303656.2-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
66bce950
Caught exception in looking up space point ptr... 0x66bce950, 0x24353770
Caught exception in looking up space point ptr... 0x10f7d6020, 0x1c3678550
Caught exception in looking up space point ptr... 0x1c3678550, 0x10f7d6020
Caught exception in looking up space point ptr... 0x11ea64240, 0x11ea63f20
Caught exception in looking up space point ptr... 0x11ea63f20, 0x11ea64240
Caught exception in looking up space point ptr... 0x15181110, 0x6819b5b0
Caught exception in looking up space point ptr... 0x6819b5b0, 0x52a97c50
Caught exception in looking up space point ptr... 0x578d3180, 0x2e898ff0
Caught exception in looking up space point ptr... 0x2e898ff0, 0x578d3180
Caught exception in looking up space point ptr... 0x13c81ef0, 0x13c82080
Caught exception in looking up space point ptr... 0x13c82080, 0x13c82210
Caught exception in looking up space point ptr... 0x3074ae70, 0x4ecf6160
Caught exception in looking up space point ptr... 0x4ecf6160, 0xf2834890
Caught exception in looking up space point ptr... 0x1c668f680, 0x16774de10
Caught exception in looking up space point ptr... 0x16774de10, 0x1c6944590
Caught exception in looking up space point ptr... 0x1c6944590, 0x1c66e3890
Caught exception in looking up space point ptr... 0x1c66e3890, 0x1c668f680
Caught exception in looking up space point ptr... 0x16fe709f0, 0x67a27a30
Caught exception in looking up space point ptr... 0x67a27a30, 0x16fe709f0
Caught exception in looking up space point ptr... 0x2ecb75e0, 0x13c930c0
Caught exception in looking up space point ptr... 0x13c930c0, 0x2ecb75e0
Caught exception in looking up space point ptr... 0x17a4757a0, 0x17a3f1b30
Caught exception in looking up space point ptr... 0x17a3f1b30, 0x17a4757a0
Caught exception in looking up space point ptr... 0x19f709ca0, 0x19fd0f1e0
Caught exception in looking up space point ptr... 0x19fd0f1e0, 0x381c88d0
Caught exception in looking up space point ptr... 0x1a97ac130, 0x1a97abe10
Caught exception in looking up space point ptr... 0x1a97abe10, 0x1a944fa40
Caught exception in looking up space point ptr... 0x130eec650, 0x130eec970
Caught exception in looking up space point ptr... 0x130eec970, 0x130eec650
Caught exception in looking up space point ptr... 0x10f4881d0, 0x19c931130
Caught exception in looking up space point ptr... 0x1b6327d90, 0x10f4881d0
Caught exception in looking up space point ptr... 0x498441a0, 0x49844010
Caught exception in looking up space point ptr... 0x49844010, 0x498441a0
Caught exception in looking up space point ptr... 0x4983fb50, 0x4983fce0
Caught exception in looking up space point ptr... 0x4983fce0, 0x4983f9c0
Caught exception in looking up space point ptr... 0x4a7a4d50, 0x3e104ca0
Caught exception in looking up space point ptr... 0x3e104ca0, 0x4c5ea7a0
Caught exception in looking up space point ptr... 0x23c8f770, 0x30f5e4d0
Caught exception in looking up space point ptr... 0x30f5e4d0, 0x23c8f770
Caught exception in looking up space point ptr... 0x1c5602e10, 0x1c5602fa0
Caught exception in looking up space point ptr... 0x1c5602fa0, 0x31258820
Caught exception in looking up space point ptr... 0x1033cf290, 0x19a2b63e0
Caught exception in looking up space point ptr... 0x19a2b63e0, 0x41739ee0
Caught exception in looking up space point ptr... 0x23f714c0, 0x1acb16aa0
Caught exception in looking up space point ptr... 0x1acb16aa0, 0x1302b5290
Caught exception in looking up space point ptr... 0x21da11e0, 0x2fa17840
Caught exception in looking up space point ptr... 0x2fa17840, 0x559e9f20
Caught exception in looking up space point ptr... 0x559e9f20, 0x23f4a230
Caught exception in looking up space point ptr... 0x23f4a230, 0x23f714c0
Caught exception in looking up space point ptr... 0xae40df70, 0xf0ac4c20
Caught exception in looking up space point ptr... 0xf0ac4c20, 0xae851230
Caught exception in looking up space point ptr... 0x1a655fef0, 0x1a5edcaa0
Caught exception in looking up space point ptr... 0x1a5edcaa0, 0x1a655fef0
Caught exception in looking up space point ptr... 0x12f1ca9d0, 0x12f1cacf0
Caught exception in looking up space point ptr... 0x12f1cacf0, 0x12f1ca9d0
Caught exception in looking up space point ptr... 0x12f1ca9d0, 0x12cf02cf0
Caught exception in looking up space point ptr... 0x12cf02cf0, 0x12f1ca9d0
Caught exception in looking up space point ptr... 0x12d262420, 0xfd9deef0
Caught exception in looking up space point ptr... 0xfd9deef0, 0x12cd371b0
Caught exception in looking up space point ptr... 0x56da2de0, 0x56c2c3e0
Caught exception in looking up space point ptr... 0x56c2c3e0, 0x56da2de0
Caught exception in looking up space point ptr... 0xea02610, 0x1aca53860
Caught exception in looking up space point ptr... 0x1aca53860, 0xea02610
Caught exception in looking up space point ptr... 0x11ea483b0, 0x11ea47f00
Caught exception in looking up space point ptr... 0x11ea47f00, 0x11ea483b0
Caught exception in looking up space point ptr... 0x10025bec0, 0x313728a0
Caught exception in looking up space point ptr... 0x24f00c40, 0x10025bec0
Caught exception in looking up space point ptr... 0x155eb2130, 0xfe49c10
Caught exception in looking up space point ptr... 0xfe49c10, 0x15e8d0a80
Caught exception in looking up space point ptr... 0x18a76d330, 0x18a7c5000
Caught exception in looking up space point ptr... 0x18a7c5000, 0x18a5ebb80
Caught exception in looking up space point ptr... 0x52e397b0, 0x52e3bf00
Caught exception in looking up space point ptr... 0x52e3bf00, 0x52e397b0
Caught exception in looking up space point ptr... 0xec62f60, 0x12fbea50
Caught exception in looking up space point ptr... 0x12fbea50, 0xec62f60
Caught exception in looking up space point ptr... 0x11c7b5a80, 0x1b334d1a0
Caught exception in looking up space point ptr... 0x1b334d1a0, 0x1b36331a0
Caught exception in looking up space point ptr... 0x189de00f0, 0x189de0230
Caught exception in looking up space point ptr... 0x189de0230, 0x189de00f0
++++>>>> total num hits: 8139619, num free: 7926332
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 345.838 925.373 3000.77 389.049 1032.95 9
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000109172 0.000204659 0.000421225 0.000208395 9.39368e-05 9
produce:tpcrawdecoder:PDVDTPCReader 155.37 179.545 236.075 163.305 27.8146 9
produce:triggerrawdecoder:PDVDTriggerReader4 0.812716 0.999313 1.54159 0.896074 0.223862 9
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0222945 0.0289182 0.0405461 0.0284109 0.00543572 9
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.109445 0.116863 0.150916 0.111087 0.0128288 9
produce:beamevent:BeamEvent 0.00016136 0.00041178 0.00171726 0.000261236 0.000464089 9
produce:pdvddaphne:DAPHNEReaderPDVD 20.7276 24.7219 31.5646 23.9078 3.73989 9
produce:ophit:OpHitFinder 0.0743955 0.0789458 0.087549 0.0773321 0.00394861 9
produce:wclsdatavd:WireCellToolkit 122.129 135.135 154.329 128.888 12.6048 9
produce:gaushit:GausHitFinder 1.31694 6.12606 15.187 2.28201 5.96047 9
produce:nhitsfilter:NumberOfHitsFilter 0.000525994 0.00154461 0.00358351 0.000719729 0.00123256 9
produce:reco3d:SpacePointSolver 14.4396 83.1661 308.387 22.9508 110.529 8
produce:hitpdune:DisambigFromSpacePoints 0.154962 4.6553 19.5141 0.266489 7.69045 8
produce:cluster3d:Cluster3D 2.61242 550.673 2319.07 3.87755 949.611 8
[art]:TriggerResults:TriggerResultInserter 3.5546e-05 6.69024e-05 0.000135767 4.8318e-05 3.30911e-05 9
end_path:out1:RootOutput 1.3223e-05 2.02334e-05 3.8466e-05 1.7982e-05 8.13345e-06 9
end_path:out1:RootOutput(write) 6.13868 12.387 48.9021 6.90912 13.8557 8
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 20854.3 MB
Peak resident set size usage (VmHWM): 17965 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 02:33:30 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 8 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 9 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 