Jobsub ID 297972.3@dunegpschedd02.fnal.gov
| Jobsub ID | 297972.3@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-04 00:40:28 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce02 |
| Last heartbeat | 2026-02-04 01:20:28 |
| From worker node | Hostname | wn2209251.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7513 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 00:41:39 |
| Input files | vd-protodune:np02vd_raw_run042421_0010_df-s03-d1_dw_0_20260203T133622.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:20:28 |
| Saved logs | justin-logs:297972.3-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
up space point ptr... 0x78b3c9f0, 0x7358be90
Caught exception in looking up space point ptr... 0xaefca420, 0xaf258d50
Caught exception in looking up space point ptr... 0xaf258d50, 0xaefca420
Caught exception in looking up space point ptr... 0xaefca420, 0xaea62bc0
Caught exception in looking up space point ptr... 0xaea62bc0, 0xaefca420
Caught exception in looking up space point ptr... 0xaefca420, 0xaf553e40
Caught exception in looking up space point ptr... 0xaf553e40, 0xaefca420
Caught exception in looking up space point ptr... 0x2073fd80, 0x2072c9d0
Caught exception in looking up space point ptr... 0x2072c9d0, 0x2073fd80
Caught exception in looking up space point ptr... 0xa6c036b0, 0xa6a67900
Caught exception in looking up space point ptr... 0xa6a67900, 0xa6c036b0
Caught exception in looking up space point ptr... 0x953bb800, 0x951b7ac0
Caught exception in looking up space point ptr... 0x951b7ac0, 0x953bb800
Caught exception in looking up space point ptr... 0x6418ef40, 0x636358b0
Caught exception in looking up space point ptr... 0x636358b0, 0x636359b0
Caught exception in looking up space point ptr... 0x9542cdd0, 0x953e2fa0
Caught exception in looking up space point ptr... 0x953e2fa0, 0x9542cdd0
Caught exception in looking up space point ptr... 0x9542cdd0, 0x94f93e30
Caught exception in looking up space point ptr... 0x94f93e30, 0x9542cdd0
Caught exception in looking up space point ptr... 0x9542cdd0, 0x94f965c0
Caught exception in looking up space point ptr... 0x94f965c0, 0x9542cdd0
Caught exception in looking up space point ptr... 0x76bae730, 0x72e72a10
Caught exception in looking up space point ptr... 0x72e72a10, 0x72e70770
Caught exception in looking up space point ptr... 0x7682ab20, 0x72878ba0
Caught exception in looking up space point ptr... 0x72878ba0, 0x7682ab20
Caught exception in looking up space point ptr... 0xb1b05090, 0xa2416390
Caught exception in looking up space point ptr... 0xa2416390, 0x9f53b730
Caught exception in looking up space point ptr... 0x7bbedfc0, 0x763969c0
Caught exception in looking up space point ptr... 0x763969c0, 0x7abdb7d0
Caught exception in looking up space point ptr... 0x7abdb7d0, 0x7abd9db0
Caught exception in looking up space point ptr... 0x7abd9db0, 0x7bbedfc0
Caught exception in looking up space point ptr... 0xa72d2900, 0xa587fe40
Caught exception in looking up space point ptr... 0xa587f230, 0xa72d2900
Caught exception in looking up space point ptr... 0x318cf640, 0x32c2ec00
Caught exception in looking up space point ptr... 0x32c2ec00, 0x318cf7d0
Caught exception in looking up space point ptr... 0x992c3500, 0x9eacb7f0
Caught exception in looking up space point ptr... 0x9eacb7f0, 0xa3b224e0
Caught exception in looking up space point ptr... 0xad104fa0, 0xaf4acf50
Caught exception in looking up space point ptr... 0xaf0c5650, 0xad104fa0
Caught exception in looking up space point ptr... 0xa39adbc0, 0xa2a10b60
Caught exception in looking up space point ptr... 0xa2a10b60, 0xa39adbc0
Caught exception in looking up space point ptr... 0xa39adbc0, 0xa1814230
Caught exception in looking up space point ptr... 0xa1814230, 0xa39adbc0
Caught exception in looking up space point ptr... 0x38aff6b0, 0x38ad15b0
Caught exception in looking up space point ptr... 0x3f79a6f0, 0x38aff6b0
Caught exception in looking up space point ptr... 0x4159a640, 0x4159a4b0
Caught exception in looking up space point ptr... 0x4159a4b0, 0x4159a640
Caught exception in looking up space point ptr... 0x7680c1b0, 0x72e8c730
Caught exception in looking up space point ptr... 0x72e8c730, 0x7680c1b0
Caught exception in looking up space point ptr... 0x76ea5e80, 0x76bfcfe0
Caught exception in looking up space point ptr... 0x76bfcfe0, 0x78161340
Caught exception in looking up space point ptr... 0x94fb0e20, 0x94fb0ec0
Caught exception in looking up space point ptr... 0x94fb0ec0, 0x94fb0ce0
Caught exception in looking up space point ptr... 0x283794f0, 0x2834fb20
Caught exception in looking up space point ptr... 0x2834fb20, 0x1f8a96d0
Caught exception in looking up space point ptr... 0x1f8a96d0, 0x4aa7d920
Caught exception in looking up space point ptr... 0x4aa7d920, 0x3d788c70
Caught exception in looking up space point ptr... 0x76ae4170, 0x76ae3e50
Caught exception in looking up space point ptr... 0x76ae3e50, 0x76ae3cc0
Caught exception in looking up space point ptr... 0xa0f3fbb0, 0xa0f3fa60
Caught exception in looking up space point ptr... 0xa0f3fa60, 0xa1dc9e70
Caught exception in looking up space point ptr... 0x3d9b5320, 0x4edf10e0
Caught exception in looking up space point ptr... 0x3920db30, 0x3d9b5320
Caught exception in looking up space point ptr... 0x60361860, 0x5c1e8020
Caught exception in looking up space point ptr... 0x5c1e8020, 0x60361860
Caught exception in looking up space point ptr... 0x14ea5770, 0x14d8eef0
Caught exception in looking up space point ptr... 0x14d8eef0, 0x14d82ac0
Caught exception in looking up space point ptr... 0xb3d4f170, 0xb3d4f020
Caught exception in looking up space point ptr... 0xb3d4f020, 0xb3d4ef20
Caught exception in looking up space point ptr... 0x96a01200, 0x96a01110
Caught exception in looking up space point ptr... 0x96a01110, 0x96a01200
Caught exception in looking up space point ptr... 0x5e8a38b0, 0x5e678ba0
Caught exception in looking up space point ptr... 0x5e678ba0, 0x5f5ea530
Caught exception in looking up space point ptr... 0xa3af1ff0, 0xa3af2310
Caught exception in looking up space point ptr... 0xa3af2310, 0xa3af1ff0
Caught exception in looking up space point ptr... 0x89d4d0f0, 0x8beb9eb0
Caught exception in looking up space point ptr... 0x8beb9eb0, 0x89d4d0f0
Caught exception in looking up space point ptr... 0xa61f67b0, 0xa61e8960
Caught exception in looking up space point ptr... 0xa61e8960, 0xa563d020
Caught exception in looking up space point ptr... 0x1a8beae0, 0x28737830
Caught exception in looking up space point ptr... 0x28737830, 0x1a8beae0
++++>>>> total num hits: 7145573, num free: 6953780
04-Feb-2026 01:19:09 GMT Opened output file with pattern "%ifb_reco_stage0_%tc_offline.root"
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 2169.52 2169.52 2169.52 2169.52 0 1
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000292641 0.000292641 0.000292641 0.000292641 0 1
produce:tpcrawdecoder:PDVDTPCReader 192.905 192.905 192.905 192.905 0 1
produce:triggerrawdecoder:PDVDTriggerReader4 1.31725 1.31725 1.31725 1.31725 0 1
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0234717 0.0234717 0.0234717 0.0234717 0 1
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.11206 0.11206 0.11206 0.11206 0 1
produce:beamevent:BeamEvent 0.00159356 0.00159356 0.00159356 0.00159356 0 1
produce:pdvddaphne:DAPHNEReaderPDVD 22.4846 22.4846 22.4846 22.4846 0 1
produce:ophit:OpHitFinder 0.0555234 0.0555234 0.0555234 0.0555234 0 1
produce:wclsdatavd:WireCellToolkit 96.1125 96.1125 96.1125 96.1125 0 1
produce:gaushit:GausHitFinder 14.9881 14.9881 14.9881 14.9881 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.00137397 0.00137397 0.00137397 0.00137397 0 1
produce:reco3d:SpacePointSolver 214.656 214.656 214.656 214.656 0 1
produce:hitpdune:DisambigFromSpacePoints 9.1462 9.1462 9.1462 9.1462 0 1
produce:cluster3d:Cluster3D 1617.67 1617.67 1617.67 1617.67 0 1
[art]:TriggerResults:TriggerResultInserter 0.000171603 0.000171603 0.000171603 0.000171603 0 1
end_path:out1:RootOutput 5.5785e-05 5.5785e-05 5.5785e-05 5.5785e-05 0 1
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 16224.4 MB
Peak resident set size usage (VmHWM): 13664.3 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 01:19:39 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 1 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 