Jobsub ID 303728.29@dunegpschedd01.fnal.gov
| Jobsub ID | 303728.29@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-04 01:20:30 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce02 |
| Last heartbeat | 2026-02-04 02:15:08 |
| From worker node | Hostname | wn1205091.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 01:21:08 |
| Input files | vd-protodune:np02vd_raw_run042421_0010_df-s03-d1_dw_0_20260203T133622.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 02:15:08 |
| Saved logs | justin-logs:303728.29-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
up space point ptr... 0xb94f5490, 0xb94f5390
Caught exception in looking up space point ptr... 0xb94f5390, 0xb94f5490
Caught exception in looking up space point ptr... 0xb94f5490, 0xb826d200
Caught exception in looking up space point ptr... 0xb826d200, 0xb94f5490
Caught exception in looking up space point ptr... 0xb94f5490, 0xb826d520
Caught exception in looking up space point ptr... 0xb826d520, 0xb94f5490
Caught exception in looking up space point ptr... 0x8845ec40, 0x8845eba0
Caught exception in looking up space point ptr... 0x8845eba0, 0x8845ec40
Caught exception in looking up space point ptr... 0x8845ec40, 0x87b96ec0
Caught exception in looking up space point ptr... 0x87b96ec0, 0x8845ec40
Caught exception in looking up space point ptr... 0x8845ec40, 0x87b96ba0
Caught exception in looking up space point ptr... 0x87b96ba0, 0x8845ec40
Caught exception in looking up space point ptr... 0x669c0fe0, 0x669c12b0
Caught exception in looking up space point ptr... 0x669c12b0, 0x66c4c6e0
Caught exception in looking up space point ptr... 0x1c6033b0, 0x5599f870
Caught exception in looking up space point ptr... 0x5599f870, 0x1c6033b0
Caught exception in looking up space point ptr... 0x2f865c40, 0x2f880db0
Caught exception in looking up space point ptr... 0x2f880db0, 0x2f878710
Caught exception in looking up space point ptr... 0x2fb7cf80, 0x5b22a8c0
Caught exception in looking up space point ptr... 0x5b22a8c0, 0x289cad30
Caught exception in looking up space point ptr... 0x1d08a260, 0x71fb4d30
Caught exception in looking up space point ptr... 0x5496eb10, 0x1d08a260
Caught exception in looking up space point ptr... 0xac147570, 0xac147a90
Caught exception in looking up space point ptr... 0xac147800, 0xac147570
Caught exception in looking up space point ptr... 0x5396d5c0, 0x3ba302f0
Caught exception in looking up space point ptr... 0x3ba302f0, 0x5396d5c0
Caught exception in looking up space point ptr... 0x624a71e0, 0x623fb000
Caught exception in looking up space point ptr... 0x623fb000, 0x624b6fc0
Caught exception in looking up space point ptr... 0x505566d0, 0x50556540
Caught exception in looking up space point ptr... 0x50556540, 0x505566d0
Caught exception in looking up space point ptr... 0x600c61d0, 0x39b8d8d0
Caught exception in looking up space point ptr... 0x39b8d8d0, 0x1b0b9750
Caught exception in looking up space point ptr... 0x627b6310, 0x5c4ffd90
Caught exception in looking up space point ptr... 0x5c4ffd90, 0x627b6310
Caught exception in looking up space point ptr... 0xc8de2af0, 0xc8ddf920
Caught exception in looking up space point ptr... 0xc8ddf920, 0xc8de2af0
Caught exception in looking up space point ptr... 0x7e90e180, 0x7cf26370
Caught exception in looking up space point ptr... 0x7cf26370, 0x7cf25b80
Caught exception in looking up space point ptr... 0x7d39fe80, 0x7d7acfe0
Caught exception in looking up space point ptr... 0x7d7acfe0, 0x7da16960
Caught exception in looking up space point ptr... 0x7da16960, 0x7e90e180
Caught exception in looking up space point ptr... 0xaeaeca20, 0xaf567480
Caught exception in looking up space point ptr... 0xaf567480, 0xaf5675b0
Caught exception in looking up space point ptr... 0x573543f0, 0x39129d70
Caught exception in looking up space point ptr... 0x55d8b880, 0x573543f0
Caught exception in looking up space point ptr... 0x5b44b110, 0x5b1d7a90
Caught exception in looking up space point ptr... 0x5b1d7a90, 0x5b44ae80
Caught exception in looking up space point ptr... 0x7072abd0, 0x56bda610
Caught exception in looking up space point ptr... 0x56bda610, 0x757779f0
Caught exception in looking up space point ptr... 0x2fb01bf0, 0x2fb01af0
Caught exception in looking up space point ptr... 0x2fb01af0, 0x2fb01bf0
Caught exception in looking up space point ptr... 0x2fb08b60, 0x2fb08ff0
Caught exception in looking up space point ptr... 0x2fb08ff0, 0x2fb08b60
Caught exception in looking up space point ptr... 0x68e4ea50, 0x6a7a3ed0
Caught exception in looking up space point ptr... 0x6994bbd0, 0x68e4ea50
Caught exception in looking up space point ptr... 0x7ac4f490, 0x7acdeac0
Caught exception in looking up space point ptr... 0x7ef7bbd0, 0x7ac4f490
Caught exception in looking up space point ptr... 0x69c7be30, 0x69c7bba0
Caught exception in looking up space point ptr... 0x69c7bba0, 0x69c7c0c0
Caught exception in looking up space point ptr... 0x72b29860, 0x7507b7c0
Caught exception in looking up space point ptr... 0x7507b7c0, 0x72b29860
Caught exception in looking up space point ptr... 0x55ab6a50, 0x53ece020
Caught exception in looking up space point ptr... 0x53ece020, 0x55ab6a50
Caught exception in looking up space point ptr... 0x93a5fc70, 0x88383260
Caught exception in looking up space point ptr... 0x88383260, 0x9ae2c190
Caught exception in looking up space point ptr... 0x5caae900, 0x1de1d820
Caught exception in looking up space point ptr... 0x1de1d820, 0x5caae900
Caught exception in looking up space point ptr... 0x5caae900, 0x27cd1140
Caught exception in looking up space point ptr... 0x27cd1140, 0x5caae900
Caught exception in looking up space point ptr... 0xdb6377c0, 0xdb6376c0
Caught exception in looking up space point ptr... 0xdb6376c0, 0xdb6377c0
Caught exception in looking up space point ptr... 0x6aed2550, 0x68a136d0
Caught exception in looking up space point ptr... 0x68a136d0, 0x699ec9a0
Caught exception in looking up space point ptr... 0xa5156d70, 0xa6c43620
Caught exception in looking up space point ptr... 0xa6c43620, 0x88f1ac90
Caught exception in looking up space point ptr... 0x74a302e0, 0x74ee0020
Caught exception in looking up space point ptr... 0x74ee0020, 0x74a302e0
Caught exception in looking up space point ptr... 0x8d234d90, 0x8e3cf6b0
Caught exception in looking up space point ptr... 0x8e3cf6b0, 0x8de13040
Caught exception in looking up space point ptr... 0xa1d4e230, 0xa19e4950
Caught exception in looking up space point ptr... 0xa19e4950, 0xa4c7dd90
++++>>>> total num hits: 9303978, num free: 9069581
04-Feb-2026 02:13:17 GMT Opened output file with pattern "%ifb_reco_stage0_%tc_offline.root"
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 3021.38 3021.38 3021.38 3021.38 0 1
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000382049 0.000382049 0.000382049 0.000382049 0 1
produce:tpcrawdecoder:PDVDTPCReader 17.5859 17.5859 17.5859 17.5859 0 1
produce:triggerrawdecoder:PDVDTriggerReader4 0.701387 0.701387 0.701387 0.701387 0 1
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0340193 0.0340193 0.0340193 0.0340193 0 1
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.111393 0.111393 0.111393 0.111393 0 1
produce:beamevent:BeamEvent 0.00135812 0.00135812 0.00135812 0.00135812 0 1
produce:pdvddaphne:DAPHNEReaderPDVD 5.37499 5.37499 5.37499 5.37499 0 1
produce:ophit:OpHitFinder 0.0585771 0.0585771 0.0585771 0.0585771 0 1
produce:wclsdatavd:WireCellToolkit 113.651 113.651 113.651 113.651 0 1
produce:gaushit:GausHitFinder 12.3653 12.3653 12.3653 12.3653 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.00168832 0.00168832 0.00168832 0.00168832 0 1
produce:reco3d:SpacePointSolver 326.819 326.819 326.819 326.819 0 1
produce:hitpdune:DisambigFromSpacePoints 10.3728 10.3728 10.3728 10.3728 0 1
produce:cluster3d:Cluster3D 2534.27 2534.27 2534.27 2534.27 0 1
[art]:TriggerResults:TriggerResultInserter 0.000109576 0.000109576 0.000109576 0.000109576 0 1
end_path:out1:RootOutput 2.7912e-05 2.7912e-05 2.7912e-05 2.7912e-05 0 1
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 18155.7 MB
Peak resident set size usage (VmHWM): 15640.9 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 02:13:58 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 1 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 