Jobsub ID 303656.52@dunegpschedd01.fnal.gov
| Jobsub ID | 303656.52@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:56:25 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce512 |
| Last heartbeat | 2026-02-04 01:02:51 |
| From worker node | Hostname | b9p10p5730.cern.ch |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 00:11:48 |
| Input files | vd-protodune:np02vd_raw_run042421_0009_df-s03-d1_dw_0_20260203T133547.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:02:51 |
| Saved logs | justin-logs:303656.52-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ac0, 0x803ca310
Caught exception in looking up space point ptr... 0x175de170, 0x175de920
Caught exception in looking up space point ptr... 0x175de920, 0x175de400
Caught exception in looking up space point ptr... 0x96bbc190, 0x96781070
Caught exception in looking up space point ptr... 0x96781070, 0x97a8b290
Caught exception in looking up space point ptr... 0x464c9750, 0x464cbe40
Caught exception in looking up space point ptr... 0x464cbe40, 0x464cb920
Caught exception in looking up space point ptr... 0x64923c30, 0x6634fef0
Caught exception in looking up space point ptr... 0x6634fef0, 0x661ecd70
Caught exception in looking up space point ptr... 0x87438df0, 0x7ba5a710
Caught exception in looking up space point ptr... 0x7ba5a710, 0x8946ba00
Caught exception in looking up space point ptr... 0x71f139b0, 0x72a9bfc0
Caught exception in looking up space point ptr... 0x72a9bfc0, 0x798170f0
Caught exception in looking up space point ptr... 0x25e1a550, 0x24637ba0
Caught exception in looking up space point ptr... 0x24637ba0, 0x288c1890
Caught exception in looking up space point ptr... 0x41211470, 0x412111e0
Caught exception in looking up space point ptr... 0x412111e0, 0x41211a10
Caught exception in looking up space point ptr... 0x294c4cc0, 0x5c0041a0
Caught exception in looking up space point ptr... 0x5c0041a0, 0x26500cf0
Caught exception in looking up space point ptr... 0x645b9d50, 0x645ba500
Caught exception in looking up space point ptr... 0x645ba500, 0x645b9fe0
Caught exception in looking up space point ptr... 0x3c32fbc0, 0x2f87bec0
Caught exception in looking up space point ptr... 0x2f87bec0, 0x10d20cb0
Caught exception in looking up space point ptr... 0x6cec8040, 0x6cec7eb0
Caught exception in looking up space point ptr... 0x6cec7eb0, 0x6cec8040
Caught exception in looking up space point ptr... 0xe238c3c0, 0xde4907b0
Caught exception in looking up space point ptr... 0xde4907b0, 0xd79ba180
Caught exception in looking up space point ptr... 0x4cb1ffb0, 0x12acc020
Caught exception in looking up space point ptr... 0x12acc020, 0x5c364180
Caught exception in looking up space point ptr... 0x8d305f00, 0x8d9c1690
Caught exception in looking up space point ptr... 0x780bfc70, 0x8d305f00
Caught exception in looking up space point ptr... 0x87919e30, 0x8687f2f0
Caught exception in looking up space point ptr... 0x6f3e6570, 0x87919e30
Caught exception in looking up space point ptr... 0x6ebc3250, 0x8b6959c0
Caught exception in looking up space point ptr... 0x8b6959c0, 0x7234bc40
Caught exception in looking up space point ptr... 0xd86d5330, 0xc9124ad0
Caught exception in looking up space point ptr... 0xc9124ad0, 0xd86d5330
Caught exception in looking up space point ptr... 0x617b9c20, 0x654ca230
Caught exception in looking up space point ptr... 0x654ca230, 0x617b9c20
Caught exception in looking up space point ptr... 0xbd27d360, 0xbd27d180
Caught exception in looking up space point ptr... 0xbd27d180, 0xbd27d360
Caught exception in looking up space point ptr... 0x6e2cefd0, 0x6e2cf260
Caught exception in looking up space point ptr... 0x6e2cf260, 0x6e2d0820
Caught exception in looking up space point ptr... 0x670c6420, 0x2cc7f900
Caught exception in looking up space point ptr... 0x2cc7f900, 0x670c6420
Caught exception in looking up space point ptr... 0x8291e140, 0x80ba86c0
Caught exception in looking up space point ptr... 0x81142280, 0x8291e140
Caught exception in looking up space point ptr... 0xbf2c4bb0, 0xbf2df7a0
Caught exception in looking up space point ptr... 0xbf2b56f0, 0xbf2c4bb0
Caught exception in looking up space point ptr... 0x642075c0, 0x4c530380
Caught exception in looking up space point ptr... 0x4c530380, 0x642075c0
Caught exception in looking up space point ptr... 0xd86d0d60, 0xd6152cb0
Caught exception in looking up space point ptr... 0xd6152cb0, 0xd86d0d60
Caught exception in looking up space point ptr... 0xd86d0d60, 0xd35c1d50
Caught exception in looking up space point ptr... 0xd35c1d50, 0xd86d0d60
Caught exception in looking up space point ptr... 0x175d35e0, 0x175d3870
Caught exception in looking up space point ptr... 0x175d3b00, 0x175d35e0
Caught exception in looking up space point ptr... 0xdbb8be70, 0xdb324590
Caught exception in looking up space point ptr... 0xdb324590, 0xdbb8be70
Caught exception in looking up space point ptr... 0x2bbd6b50, 0x2bbd69c0
Caught exception in looking up space point ptr... 0x2bbd69c0, 0x568a6f90
Caught exception in looking up space point ptr... 0x50385770, 0xea475a0
Caught exception in looking up space point ptr... 0xea475a0, 0x714ca710
Caught exception in looking up space point ptr... 0xdeb27810, 0xe0fa2b40
Caught exception in looking up space point ptr... 0xe0fa2b40, 0xdeb27810
Caught exception in looking up space point ptr... 0x9ba6ef70, 0x9a405560
Caught exception in looking up space point ptr... 0x9a6bcdc0, 0x9ba6ef70
Caught exception in looking up space point ptr... 0xbbd54850, 0xbbd54710
Caught exception in looking up space point ptr... 0xbbd54710, 0xbbd54850
Caught exception in looking up space point ptr... 0x52ee8bf0, 0x52eeb680
Caught exception in looking up space point ptr... 0x52eeb680, 0x52ee8bf0
Caught exception in looking up space point ptr... 0xdb3024a0, 0xdb2d05c0
Caught exception in looking up space point ptr... 0xdb2d05c0, 0xdb3024a0
Caught exception in looking up space point ptr... 0xba964db0, 0xba911df0
Caught exception in looking up space point ptr... 0xba911df0, 0xba964db0
Caught exception in looking up space point ptr... 0xcf108f0, 0xcf109f0
Caught exception in looking up space point ptr... 0xcf109f0, 0xcf108f0
Caught exception in looking up space point ptr... 0x899101e0, 0x8a033730
Caught exception in looking up space point ptr... 0x8a033730, 0x899101e0
Caught exception in looking up space point ptr... 0x7658f000, 0x72d85420
Caught exception in looking up space point ptr... 0x72d85420, 0x7658f000
++++>>>> total num hits: 8274907, num free: 8059532
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 65.3018 118.583 1145.11 74.5521 209.642 25
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 4.446e-05 0.000107888 0.000650828 8.2501e-05 0.000115353 25
produce:tpcrawdecoder:PDVDTPCReader 3.30336 3.76706 4.51345 3.80494 0.30308 25
produce:triggerrawdecoder:PDVDTriggerReader4 0.0102429 0.0147936 0.031806 0.012591 0.00565143 25
produce:timingrawdecoder:PDHDTimingRawDecoder 0.000385705 0.00078381 0.00717637 0.000499576 0.00130825 25
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.00152336 0.00201261 0.00326473 0.00186664 0.000377748 25
produce:beamevent:BeamEvent 5.762e-05 0.000138287 0.000916221 0.000109592 0.000161182 25
produce:pdvddaphne:DAPHNEReaderPDVD 2.00874 2.53433 3.1392 2.53304 0.300083 25
produce:ophit:OpHitFinder 0.0267455 0.0355559 0.0446355 0.0348576 0.00390941 25
produce:wclsdatavd:WireCellToolkit 48.5568 53.9686 61.4024 53.6737 3.31961 25
produce:gaushit:GausHitFinder 0.417772 1.06571 6.76804 0.800094 1.1879 25
produce:nhitsfilter:NumberOfHitsFilter 0.000150052 0.000307199 0.000989322 0.000240043 0.000192066 25
produce:reco3d:SpacePointSolver 4.63062 16.739 180.593 9.68776 33.6288 25
produce:hitpdune:DisambigFromSpacePoints 0.0389689 0.315316 5.75732 0.0824746 1.11137 25
produce:cluster3d:Cluster3D 0.635292 36.9386 884.171 1.57034 172.942 25
[art]:TriggerResults:TriggerResultInserter 2.206e-05 3.2531e-05 8.9721e-05 2.666e-05 1.53605e-05 25
end_path:out1:RootOutput 5.27e-06 1.00524e-05 2.5091e-05 8.84e-06 4.26353e-06 25
end_path:out1:RootOutput(write) 2.74587 3.31454 4.33175 3.12954 0.413883 24
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 16655.5 MB
Peak resident set size usage (VmHWM): 13907 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 02:02:13 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 24 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 25 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 