Jobsub ID 297837.1@dunegpschedd02.fnal.gov
| Jobsub ID | 297837.1@dunegpschedd02.fnal.gov |
| Workflow ID | 12689 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 18:50:12 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce01 |
| Last heartbeat | 2026-02-03 20:42:22 |
| From worker node | Hostname | wn1205080.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 18:50:33 |
| Input files | vd-protodune:np02vd_raw_run042421_0011_df-s02-d0_dw_0_20260203T133654.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-03 20:42:22 |
| Saved logs | justin-logs:297837.1-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
exception in looking up space point ptr... 0xc2f24180, 0xc2f24030
Caught exception in looking up space point ptr... 0xc2f24030, 0xc2f24180
Caught exception in looking up space point ptr... 0x2fd376b0, 0xc4e2ddf0
Caught exception in looking up space point ptr... 0xc4e2ddf0, 0xf459ef00
Caught exception in looking up space point ptr... 0xe6744620, 0x100be38c0
Caught exception in looking up space point ptr... 0x100be38c0, 0x101317020
Caught exception in looking up space point ptr... 0x58959d90, 0xd4801980
Caught exception in looking up space point ptr... 0xd4a01030, 0x58959d90
Caught exception in looking up space point ptr... 0x110f5daf0, 0x110f5ad10
Caught exception in looking up space point ptr... 0x110f5ad10, 0x110f5daf0
Caught exception in looking up space point ptr... 0x110f5daf0, 0xc519f7d0
Caught exception in looking up space point ptr... 0xc519f7d0, 0x110f5daf0
Caught exception in looking up space point ptr... 0x110f5daf0, 0x296ed200
Caught exception in looking up space point ptr... 0x296ed200, 0x110f5daf0
Caught exception in looking up space point ptr... 0xfbcd6d10, 0xfbbeae40
Caught exception in looking up space point ptr... 0xfbbeae40, 0xfbcd6d10
Caught exception in looking up space point ptr... 0xa7a14e0, 0xa7a17b0
Caught exception in looking up space point ptr... 0xa7a17b0, 0xa7a14e0
Caught exception in looking up space point ptr... 0x50a63c10, 0xd3602cb0
Caught exception in looking up space point ptr... 0xd35bfaf0, 0x4db787a0
Caught exception in looking up space point ptr... 0x4db787a0, 0x46662ff0
Caught exception in looking up space point ptr... 0x46662ff0, 0x50a63c10
Caught exception in looking up space point ptr... 0x121015380, 0x12004d5c0
Caught exception in looking up space point ptr... 0x12004d5c0, 0x12004d480
Caught exception in looking up space point ptr... 0x36bce410, 0x21f239e0
Caught exception in looking up space point ptr... 0x21f239e0, 0x36bce410
Caught exception in looking up space point ptr... 0x36bce410, 0x36d943b0
Caught exception in looking up space point ptr... 0x36d943b0, 0x36bce410
Caught exception in looking up space point ptr... 0x56ba2da0, 0x56b82a40
Caught exception in looking up space point ptr... 0x56b82a40, 0x56ba2da0
Caught exception in looking up space point ptr... 0x56ba2da0, 0x56ba18d0
Caught exception in looking up space point ptr... 0x56ba18d0, 0x56ba2da0
Caught exception in looking up space point ptr... 0x553fa470, 0xbd0a2b90
Caught exception in looking up space point ptr... 0xbd0a2b90, 0x553fa470
Caught exception in looking up space point ptr... 0x553fa470, 0x5540b600
Caught exception in looking up space point ptr... 0x5540b600, 0x553fa470
Caught exception in looking up space point ptr... 0x10d4b8cf0, 0xf3e56800
Caught exception in looking up space point ptr... 0xf3e56800, 0x110d0c710
Caught exception in looking up space point ptr... 0xbdc2aaa0, 0xbdc2abe0
Caught exception in looking up space point ptr... 0xe9cd6260, 0xbdc2aaa0
Caught exception in looking up space point ptr... 0xdac3c40, 0x9ffc1c0
Caught exception in looking up space point ptr... 0x9ffc1c0, 0xdac3c40
Caught exception in looking up space point ptr... 0x110d17180, 0xfe8b2710
Caught exception in looking up space point ptr... 0xfe8b2710, 0x110d17180
Caught exception in looking up space point ptr... 0xf3bea60, 0x56fa3a30
Caught exception in looking up space point ptr... 0x56fa3a30, 0x42c15c10
Caught exception in looking up space point ptr... 0xbda50ec0, 0xbda51140
Caught exception in looking up space point ptr... 0xbda51140, 0xbda51000
Caught exception in looking up space point ptr... 0xc4396160, 0x1041a65c0
Caught exception in looking up space point ptr... 0x1041a65c0, 0xc4396160
Caught exception in looking up space point ptr... 0xc4396160, 0xf3470f80
Caught exception in looking up space point ptr... 0xf3470f80, 0xc4396160
Caught exception in looking up space point ptr... 0x12126f4b0, 0x12126f640
Caught exception in looking up space point ptr... 0x12126f640, 0x12126f960
Caught exception in looking up space point ptr... 0xde31620, 0xde31940
Caught exception in looking up space point ptr... 0xde31940, 0xde317b0
Caught exception in looking up space point ptr... 0x3da9b900, 0x5e50b030
Caught exception in looking up space point ptr... 0x5e50b030, 0x45139390
Caught exception in looking up space point ptr... 0x10d557900, 0x5ee52de0
Caught exception in looking up space point ptr... 0x5ee52de0, 0x10d557900
Caught exception in looking up space point ptr... 0x80f81260, 0xc4138f40
Caught exception in looking up space point ptr... 0xc4138f40, 0x80f81260
Caught exception in looking up space point ptr... 0x4ea49d10, 0x370b85d0
Caught exception in looking up space point ptr... 0x370b85d0, 0x4ea49d10
Caught exception in looking up space point ptr... 0xd4e27e20, 0xd4e27c90
Caught exception in looking up space point ptr... 0xd4e27c90, 0xd4e27e20
Caught exception in looking up space point ptr... 0x458e0630, 0x10c822c00
Caught exception in looking up space point ptr... 0x10c822c00, 0xc32fdcb0
Caught exception in looking up space point ptr... 0xc38db230, 0x5964a650
Caught exception in looking up space point ptr... 0x5964a650, 0xc38db230
Caught exception in looking up space point ptr... 0x4565bae0, 0x456588a0
Caught exception in looking up space point ptr... 0x456588a0, 0x4565bae0
Caught exception in looking up space point ptr... 0xfef660b0, 0x807781a0
Caught exception in looking up space point ptr... 0x807781a0, 0xde1f83c0
Caught exception in looking up space point ptr... 0x125c7e000, 0x56b65570
Caught exception in looking up space point ptr... 0x56b65570, 0x56ab6780
Caught exception in looking up space point ptr... 0x10af1c0e0, 0x10af1bc30
Caught exception in looking up space point ptr... 0x10af1bc30, 0x10af1c0e0
Caught exception in looking up space point ptr... 0xde31df0, 0xde31c60
Caught exception in looking up space point ptr... 0xde31c60, 0xde31df0
++++>>>> total num hits: 7781150, num free: 7550558
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 263.293 804.161 2886.3 291.745 935.229 8
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.6041e-05 0.000281922 0.00126679 0.000107122 0.000381734 8
produce:tpcrawdecoder:PDVDTPCReader 99.3506 125.484 161.158 124.936 18.9346 8
produce:triggerrawdecoder:PDVDTriggerReader4 0.725307 0.877427 1.15765 0.830404 0.136989 8
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0226576 0.0265281 0.033482 0.0255329 0.0038401 8
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.110089 0.12204 0.1544 0.112295 0.0152701 8
produce:beamevent:BeamEvent 0.000109146 0.000326841 0.00167972 0.000140094 0.000511644 8
produce:pdvddaphne:DAPHNEReaderPDVD 11.6856 17.1855 22.4618 17.986 3.348 8
produce:ophit:OpHitFinder 0.0462135 0.0583922 0.0743989 0.0579675 0.00750741 8
produce:wclsdatavd:WireCellToolkit 105.522 112.976 131.149 109.473 8.49015 8
produce:gaushit:GausHitFinder 1.69611 8.04026 29.1261 2.36062 9.26884 8
produce:nhitsfilter:NumberOfHitsFilter 0.000416635 0.00119945 0.00245824 0.000791666 0.000809887 8
produce:reco3d:SpacePointSolver 21.285 95.0395 370.337 25.9433 123.897 7
produce:hitpdune:DisambigFromSpacePoints 0.179245 4.52838 16.8195 0.249958 6.86011 7
produce:cluster3d:Cluster3D 2.89122 505.522 2172.68 5.95482 822.047 7
[art]:TriggerResults:TriggerResultInserter 3.164e-05 5.9497e-05 0.000136106 5.31255e-05 3.13841e-05 8
end_path:out1:RootOutput 4.428e-06 8.45975e-06 3.1259e-05 5.1695e-06 8.62893e-06 8
end_path:out1:RootOutput(write) 6.2276 11.3184 32.1459 7.11757 8.76426 7
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 19424.5 MB
Peak resident set size usage (VmHWM): 16821.1 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 03-Feb-2026 20:40:33 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 7 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 8 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 