Jobsub ID 303646.9@dunegpschedd01.fnal.gov
| Jobsub ID | 303646.9@dunegpschedd01.fnal.gov |
| Workflow ID | 12718 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:44:24 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce01 |
| Last heartbeat | 2026-02-03 23:36:55 |
| From worker node | Hostname | wn2209250.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7513 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 22:45:49 |
| Input files | vd-protodune:np02vd_raw_run042422_0116_df-s04-d1_dw_0_20260203T161728.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-03 23:36:55 |
| Saved logs | justin-logs:303646.9-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
x93522730
Caught exception in looking up space point ptr... 0x86383540, 0x7f1c08a0
Caught exception in looking up space point ptr... 0x7f1c08a0, 0x86383540
Caught exception in looking up space point ptr... 0x86e96b90, 0x86e96670
Caught exception in looking up space point ptr... 0x86e96670, 0x86e96900
Caught exception in looking up space point ptr... 0x8bcda5c0, 0x89f44510
Caught exception in looking up space point ptr... 0x89f44510, 0x8bcda5c0
Caught exception in looking up space point ptr... 0x90128a70, 0x93bf1c30
Caught exception in looking up space point ptr... 0x93bf1c30, 0x90128a70
Caught exception in looking up space point ptr... 0x86b30cc0, 0x86019120
Caught exception in looking up space point ptr... 0x86019120, 0x86b30b30
Caught exception in looking up space point ptr... 0x33d6dc50, 0x488fc2b0
Caught exception in looking up space point ptr... 0x488fc2b0, 0x488fc440
Caught exception in looking up space point ptr... 0x4d410d30, 0x18ba7120
Caught exception in looking up space point ptr... 0x18ba7120, 0x4d410d30
Caught exception in looking up space point ptr... 0x7790ed80, 0x779a4e40
Caught exception in looking up space point ptr... 0x779a4150, 0x77b02e30
Caught exception in looking up space point ptr... 0x77b02e30, 0x77b02ca0
Caught exception in looking up space point ptr... 0x77b02ca0, 0x7790ed80
Caught exception in looking up space point ptr... 0x5f77f4f0, 0x48ecf8d0
Caught exception in looking up space point ptr... 0x48ecfbf0, 0x48ecfd80
Caught exception in looking up space point ptr... 0x48ecfd80, 0x48ecfa60
Caught exception in looking up space point ptr... 0x48ecfa60, 0x5f77f4f0
Caught exception in looking up space point ptr... 0x58780ab0, 0x58784270
Caught exception in looking up space point ptr... 0x58784270, 0x587856f0
Caught exception in looking up space point ptr... 0x698040b0, 0x69802b30
Caught exception in looking up space point ptr... 0x69802b30, 0x698040b0
Caught exception in looking up space point ptr... 0x76149600, 0x76522e80
Caught exception in looking up space point ptr... 0x76522e80, 0x76149600
Caught exception in looking up space point ptr... 0x76149370, 0x76522bf0
Caught exception in looking up space point ptr... 0x76522bf0, 0x76149370
Caught exception in looking up space point ptr... 0xb4b67a60, 0xba5293a0
Caught exception in looking up space point ptr... 0xba5293a0, 0xb4b67a60
Caught exception in looking up space point ptr... 0x9d459f60, 0x9d1feb80
Caught exception in looking up space point ptr... 0x9d1feb80, 0x9d459f60
Caught exception in looking up space point ptr... 0x68c4e7c0, 0x75f3d480
Caught exception in looking up space point ptr... 0x75f3d480, 0x68c4e7c0
Caught exception in looking up space point ptr... 0x68c4e7c0, 0x75d9fec0
Caught exception in looking up space point ptr... 0x75d9fec0, 0x68c4e7c0
Caught exception in looking up space point ptr... 0xb12a30f0, 0xad551fa0
Caught exception in looking up space point ptr... 0xad551fa0, 0xb05df710
Caught exception in looking up space point ptr... 0x75e1bb90, 0x747bb110
Caught exception in looking up space point ptr... 0x747bb110, 0x7f476400
Caught exception in looking up space point ptr... 0xb6fab3b0, 0xb6f3f4a0
Caught exception in looking up space point ptr... 0xb6f3f4a0, 0xb6faa1b0
Caught exception in looking up space point ptr... 0x93ad7c40, 0x9d0e23d0
Caught exception in looking up space point ptr... 0x9d0e23d0, 0x93ad7c40
Caught exception in looking up space point ptr... 0x68e80cf0, 0x68dc4770
Caught exception in looking up space point ptr... 0x68dc4770, 0x733668a0
Caught exception in looking up space point ptr... 0x9cc43b50, 0x9ca18ed0
Caught exception in looking up space point ptr... 0x9ca18ed0, 0x9b3a7300
Caught exception in looking up space point ptr... 0x58714580, 0x58714430
Caught exception in looking up space point ptr... 0x58714430, 0x58714580
Caught exception in looking up space point ptr... 0x4d411280, 0x4d5316e0
Caught exception in looking up space point ptr... 0x4d5316e0, 0x4d411280
Caught exception in looking up space point ptr... 0x3f0b29d0, 0x558ef770
Caught exception in looking up space point ptr... 0x558ef770, 0x3f0b2b60
Caught exception in looking up space point ptr... 0x7c52f6f0, 0x7c52ee00
Caught exception in looking up space point ptr... 0x7c52ee00, 0x7b8b2cc0
Caught exception in looking up space point ptr... 0xa5d52570, 0x97b7b8a0
Caught exception in looking up space point ptr... 0x97b7b8a0, 0xa5d52570
Caught exception in looking up space point ptr... 0x587a7790, 0x5879f7d0
Caught exception in looking up space point ptr... 0x5879f7d0, 0x5879f960
Caught exception in looking up space point ptr... 0x55a3d6d0, 0x55a4ee50
Caught exception in looking up space point ptr... 0x55a4ee50, 0x55a465c0
Caught exception in looking up space point ptr... 0xd4bbde80, 0xdb7fbde0
Caught exception in looking up space point ptr... 0xdb7fbde0, 0xc7850a40
Caught exception in looking up space point ptr... 0x7f48d640, 0x83c27b20
Caught exception in looking up space point ptr... 0x83c27b20, 0x7f48d640
Caught exception in looking up space point ptr... 0xa4af1460, 0xb96475b0
Caught exception in looking up space point ptr... 0xb96475b0, 0xa4af1460
Caught exception in looking up space point ptr... 0xe2062d70, 0xe25c1050
Caught exception in looking up space point ptr... 0xe25c1050, 0xe2062d70
Caught exception in looking up space point ptr... 0x81cafaf0, 0x816946f0
Caught exception in looking up space point ptr... 0x816946f0, 0x81cafaf0
Caught exception in looking up space point ptr... 0xbdf79440, 0xbe4fea60
Caught exception in looking up space point ptr... 0xbe4fea60, 0xbe4febf0
Caught exception in looking up space point ptr... 0xc44cf9a0, 0xcd02aec0
Caught exception in looking up space point ptr... 0xcd02aec0, 0xc44cf9a0
Caught exception in looking up space point ptr... 0xc44cf9a0, 0xd4e933a0
Caught exception in looking up space point ptr... 0xd4e933a0, 0xc44cf9a0
++++>>>> total num hits: 8801582, num free: 8564182
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 341.762 1433.14 2524.52 1433.14 1091.38 2
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.4058e-05 0.000320957 0.000547855 0.000320957 0.000226899 2
produce:tpcrawdecoder:PDVDTPCReader 206.562 207.91 209.258 207.91 1.34785 2
produce:triggerrawdecoder:PDVDTriggerReader4 1.21 1.21978 1.22957 1.21978 0.00978855 2
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0220556 0.0247958 0.027536 0.0247958 0.0027402 2
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.109002 0.109919 0.110837 0.109919 0.00091755 2
produce:beamevent:BeamEvent 0.000133301 0.000980968 0.00182863 0.000980968 0.000847667 2
produce:pdvddaphne:DAPHNEReaderPDVD 23.311 25.6403 27.9696 25.6403 2.32929 2
produce:ophit:OpHitFinder 0.0448981 0.0451944 0.0454907 0.0451944 0.000296304 2
produce:wclsdatavd:WireCellToolkit 80.5646 85.8062 91.0479 85.8062 5.24166 2
produce:gaushit:GausHitFinder 8.33015 10.0935 11.8569 10.0935 1.76336 2
produce:nhitsfilter:NumberOfHitsFilter 0.00146162 0.00184689 0.00223217 0.00184689 0.000385273 2
[art]:TriggerResults:TriggerResultInserter 0.000104869 0.000133698 0.000162527 0.000133698 2.8829e-05 2
end_path:out1:RootOutput 2.0879e-05 3.2752e-05 4.4625e-05 3.2752e-05 1.1873e-05 2
end_path:out1:RootOutput(write) 7.5277 7.5277 7.5277 7.5277 0 1
produce:reco3d:SpacePointSolver 299.221 299.221 299.221 299.221 0 1
produce:hitpdune:DisambigFromSpacePoints 15.1492 15.1492 15.1492 15.1492 0 1
produce:cluster3d:Cluster3D 1881.81 1881.81 1881.81 1881.81 0 1
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 19498.3 MB
Peak resident set size usage (VmHWM): 17399.4 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 03-Feb-2026 23:35:34 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 2 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 