Jobsub ID 303653.98@dunegpschedd01.fnal.gov
| Jobsub ID | 303653.98@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:52:24 |
| Site | UK_RAL-PPD |
| Entry | CMSHTPC_T2_UK_SGrid_RALPP_hep208 |
| Last heartbeat | 2026-02-04 00:35:34 |
| From worker node | Hostname | heplnc145.pp.rl.ac.uk |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:41:39 |
| Input files | vd-protodune:np02vd_raw_run042421_0050_df-s03-d0_dw_0_20260203T135805.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 00:35:34 |
| Saved logs | justin-logs:303653.98-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
up space point ptr... 0x9840e7c0, 0x983d9d90
Caught exception in looking up space point ptr... 0x983d9d90, 0x9844e280
Caught exception in looking up space point ptr... 0x9844e280, 0x98636ba0
Caught exception in looking up space point ptr... 0x98636ba0, 0x9840e7c0
Caught exception in looking up space point ptr... 0x76be2f20, 0x76d07880
Caught exception in looking up space point ptr... 0x76d07880, 0x76be2f20
Caught exception in looking up space point ptr... 0x4a3a8950, 0x4a3e38b0
Caught exception in looking up space point ptr... 0x4a3e38b0, 0x4a3a8950
Caught exception in looking up space point ptr... 0x98634630, 0x983205e0
Caught exception in looking up space point ptr... 0x983205e0, 0x983229e0
Caught exception in looking up space point ptr... 0x983229e0, 0x983126b0
Caught exception in looking up space point ptr... 0x983126b0, 0x985ac940
Caught exception in looking up space point ptr... 0x985ac940, 0x98590b60
Caught exception in looking up space point ptr... 0x98590b60, 0x98634630
Caught exception in looking up space point ptr... 0x8aeace20, 0x8aea9710
Caught exception in looking up space point ptr... 0x8aea9710, 0x8aeace20
Caught exception in looking up space point ptr... 0x7df09190, 0x918eb4d0
Caught exception in looking up space point ptr... 0x918eb4d0, 0x7df09190
Caught exception in looking up space point ptr... 0x98585ee0, 0x981c5920
Caught exception in looking up space point ptr... 0x981c5920, 0x981c5a50
Caught exception in looking up space point ptr... 0x981c5a50, 0x98187c50
Caught exception in looking up space point ptr... 0x98187c50, 0x9861d2a0
Caught exception in looking up space point ptr... 0x9861d2a0, 0x986264a0
Caught exception in looking up space point ptr... 0x986264a0, 0x98585ee0
Caught exception in looking up space point ptr... 0x8af2e020, 0x8a1f4200
Caught exception in looking up space point ptr... 0x8a1f4200, 0x8af2e020
Caught exception in looking up space point ptr... 0xace3f980, 0xaca3dd10
Caught exception in looking up space point ptr... 0xaca3dd10, 0xaca3db80
Caught exception in looking up space point ptr... 0x9819c5e0, 0x986281e0
Caught exception in looking up space point ptr... 0x986281e0, 0x9819c5e0
Caught exception in looking up space point ptr... 0x9819c5e0, 0x981cf020
Caught exception in looking up space point ptr... 0x981cf020, 0x9819c5e0
Caught exception in looking up space point ptr... 0x9819c5e0, 0x98628440
Caught exception in looking up space point ptr... 0x98628440, 0x9819c5e0
Caught exception in looking up space point ptr... 0x1daf1e80, 0x1daf1b60
Caught exception in looking up space point ptr... 0x1daf1b60, 0x1daf1e80
Caught exception in looking up space point ptr... 0x21865330, 0x282189d0
Caught exception in looking up space point ptr... 0x282189d0, 0x21865330
Caught exception in looking up space point ptr... 0xa82ad710, 0xa6fdb480
Caught exception in looking up space point ptr... 0xa6fdb480, 0xa82ad710
Caught exception in looking up space point ptr... 0x4c873ff0, 0x4c875220
Caught exception in looking up space point ptr... 0x4c875220, 0x4c873ff0
Caught exception in looking up space point ptr... 0x27865ae0, 0x27865e00
Caught exception in looking up space point ptr... 0x27865e00, 0x27865f90
Caught exception in looking up space point ptr... 0x66d5d680, 0x66d5d9a0
Caught exception in looking up space point ptr... 0x66d5d9a0, 0x66d5d680
Caught exception in looking up space point ptr... 0xa41741c0, 0xa4daa090
Caught exception in looking up space point ptr... 0xa4daa090, 0xa41741c0
Caught exception in looking up space point ptr... 0xa4bea1b0, 0xa6ec0970
Caught exception in looking up space point ptr... 0xa6ec0970, 0xa4bea1b0
Caught exception in looking up space point ptr... 0x1db0d270, 0x1db0cc30
Caught exception in looking up space point ptr... 0x1db0cc30, 0x1db0cdc0
Caught exception in looking up space point ptr... 0x46b81540, 0x2c575000
Caught exception in looking up space point ptr... 0x2c575000, 0x46b81540
Caught exception in looking up space point ptr... 0x7998b520, 0x7998b390
Caught exception in looking up space point ptr... 0x7998b390, 0x7998b520
Caught exception in looking up space point ptr... 0x7933a050, 0x7933a570
Caught exception in looking up space point ptr... 0x7933a570, 0x7933a050
Caught exception in looking up space point ptr... 0x48bf4f20, 0x48bf51c0
Caught exception in looking up space point ptr... 0x48bf51c0, 0x48bf4dd0
Caught exception in looking up space point ptr... 0x98166320, 0x986374f0
Caught exception in looking up space point ptr... 0x986374f0, 0x98166320
Caught exception in looking up space point ptr... 0x98166320, 0x98637750
Caught exception in looking up space point ptr... 0x98637750, 0x98166320
Caught exception in looking up space point ptr... 0x232a15d0, 0x232a1120
Caught exception in looking up space point ptr... 0x232a1120, 0x232a15d0
Caught exception in looking up space point ptr... 0xb02d1b00, 0xb5b40380
Caught exception in looking up space point ptr... 0xb5b40380, 0xaa82a9c0
Caught exception in looking up space point ptr... 0xa4d28220, 0xb02d1b00
Caught exception in looking up space point ptr... 0x78e6a660, 0x79aa8790
Caught exception in looking up space point ptr... 0x79aa8790, 0x799574a0
Caught exception in looking up space point ptr... 0x209e9790, 0x502a32f0
Caught exception in looking up space point ptr... 0x502a32f0, 0x209e9790
Caught exception in looking up space point ptr... 0x3b0232e0, 0x1e0f9730
Caught exception in looking up space point ptr... 0x1e0f9730, 0x3b0232e0
Caught exception in looking up space point ptr... 0x1db06060, 0x1db05ed0
Caught exception in looking up space point ptr... 0x1db05ed0, 0x1db061f0
Caught exception in looking up space point ptr... 0xb712b780, 0xb712b690
Caught exception in looking up space point ptr... 0xb712b690, 0xb712b780
Caught exception in looking up space point ptr... 0x48e87430, 0x48e7cd70
Caught exception in looking up space point ptr... 0x48e7cd70, 0x48e7d230
++++>>>> total num hits: 7415594, num free: 7205991
04-Feb-2026 00:28:35 GMT Opened output file with pattern "%ifb_reco_stage0_%tc_offline.root"
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 2739.35 2739.35 2739.35 2739.35 0 1
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000341975 0.000341975 0.000341975 0.000341975 0 1
produce:tpcrawdecoder:PDVDTPCReader 12.4285 12.4285 12.4285 12.4285 0 1
produce:triggerrawdecoder:PDVDTriggerReader4 0.544023 0.544023 0.544023 0.544023 0 1
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0183365 0.0183365 0.0183365 0.0183365 0 1
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.088678 0.088678 0.088678 0.088678 0 1
produce:beamevent:BeamEvent 0.00134075 0.00134075 0.00134075 0.00134075 0 1
produce:pdvddaphne:DAPHNEReaderPDVD 3.63415 3.63415 3.63415 3.63415 0 1
produce:ophit:OpHitFinder 0.0490355 0.0490355 0.0490355 0.0490355 0 1
produce:wclsdatavd:WireCellToolkit 83.0474 83.0474 83.0474 83.0474 0 1
produce:gaushit:GausHitFinder 12.5245 12.5245 12.5245 12.5245 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.00214608 0.00214608 0.00214608 0.00214608 0 1
produce:reco3d:SpacePointSolver 204.095 204.095 204.095 204.095 0 1
produce:hitpdune:DisambigFromSpacePoints 7.52237 7.52237 7.52237 7.52237 0 1
produce:cluster3d:Cluster3D 2415.36 2415.36 2415.36 2415.36 0 1
[art]:TriggerResults:TriggerResultInserter 0.00012183 0.00012183 0.00012183 0.00012183 0 1
end_path:out1:RootOutput 3.0518e-05 3.0518e-05 3.0518e-05 3.0518e-05 0 1
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 17038.9 MB
Peak resident set size usage (VmHWM): 14814.8 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:28:57 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 1 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 