Jobsub ID 297921.65@dunegpschedd02.fnal.gov
| Jobsub ID | 297921.65@dunegpschedd02.fnal.gov |
| Workflow ID | 12719 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:58:26 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce507 |
| Last heartbeat | 2026-02-04 00:41:05 |
| From worker node | Hostname | b9p35p0005.cern.ch |
| cpuinfo | AMD EPYC 9655P 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:51:40 |
| Input files | vd-protodune:np02vd_raw_run042433_0047_df-s04-d0_dw_0_20260203T185959.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 00:41:05 |
| Saved logs | justin-logs:297921.65-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ught exception in looking up space point ptr... 0xf58d8460, 0xf51f5080
Caught exception in looking up space point ptr... 0xf51f5080, 0xf58d8460
Caught exception in looking up space point ptr... 0xcc2c8bc0, 0xbd599280
Caught exception in looking up space point ptr... 0xbd599280, 0x222dd920
Caught exception in looking up space point ptr... 0xe3ca4bc0, 0xe3ca4a30
Caught exception in looking up space point ptr... 0xe3ca4a30, 0xe3ca4bc0
Caught exception in looking up space point ptr... 0xe3ca4bc0, 0xe3ca48a0
Caught exception in looking up space point ptr... 0xe3ca48a0, 0xe3ca4bc0
Caught exception in looking up space point ptr... 0xe3525fe0, 0xe35382e0
Caught exception in looking up space point ptr... 0xe35382e0, 0xe3525fe0
Caught exception in looking up space point ptr... 0x33033c00, 0x10d8b29e0
Caught exception in looking up space point ptr... 0x10d8b29e0, 0x3284d980
Caught exception in looking up space point ptr... 0x3284d980, 0x3284dca0
Caught exception in looking up space point ptr... 0x3284dca0, 0x10d8b29e0
Caught exception in looking up space point ptr... 0x10d8b29e0, 0x32787d70
Caught exception in looking up space point ptr... 0x32787d70, 0x33033a70
Caught exception in looking up space point ptr... 0x33033a70, 0x3284db10
Caught exception in looking up space point ptr... 0x3284db10, 0x33033c00
Caught exception in looking up space point ptr... 0xca9dd0b0, 0xc0fc5310
Caught exception in looking up space point ptr... 0xca9dcfb0, 0xca9dd0b0
Caught exception in looking up space point ptr... 0xa4818850, 0xa4818980
Caught exception in looking up space point ptr... 0xa4818720, 0xa6b5e620
Caught exception in looking up space point ptr... 0xa6b5e620, 0xa6b5e6c0
Caught exception in looking up space point ptr... 0xa6b5e6c0, 0xa4818850
Caught exception in looking up space point ptr... 0xe7b8ceb0, 0xe7b8cc60
Caught exception in looking up space point ptr... 0xe7b8cc60, 0xf6e3a320
Caught exception in looking up space point ptr... 0x537d67c0, 0x3329d4b0
Caught exception in looking up space point ptr... 0x3329d4b0, 0x537d67c0
Caught exception in looking up space point ptr... 0x537d67c0, 0x537d6630
Caught exception in looking up space point ptr... 0x537d6630, 0x537d67c0
Caught exception in looking up space point ptr... 0xf1af3af0, 0xf1be29b0
Caught exception in looking up space point ptr... 0xf1be29b0, 0xf1af3af0
Caught exception in looking up space point ptr... 0xabce94d0, 0xf707f700
Caught exception in looking up space point ptr... 0xf707f700, 0xabce94d0
Caught exception in looking up space point ptr... 0x101a1c800, 0xf9b92970
Caught exception in looking up space point ptr... 0xf2887e70, 0x101a1c800
Caught exception in looking up space point ptr... 0xefbde200, 0xefbdbef0
Caught exception in looking up space point ptr... 0xefbdbef0, 0xefbde0d0
Caught exception in looking up space point ptr... 0xf98aa970, 0xf9971d30
Caught exception in looking up space point ptr... 0xc861bf40, 0xf98aa970
Caught exception in looking up space point ptr... 0xd56f5120, 0xc9024570
Caught exception in looking up space point ptr... 0xc9024570, 0x26e65a30
Caught exception in looking up space point ptr... 0xf5a7a250, 0xf5a978f0
Caught exception in looking up space point ptr... 0xf5a978f0, 0xf5a7a250
Caught exception in looking up space point ptr... 0xf5440cd0, 0xf59508e0
Caught exception in looking up space point ptr... 0xf59508e0, 0xf5440cd0
Caught exception in looking up space point ptr... 0xf1bc1140, 0xf5b6be10
Caught exception in looking up space point ptr... 0xf5b6be10, 0xf1bc1140
Caught exception in looking up space point ptr... 0x105d90c30, 0x5acdacf0
Caught exception in looking up space point ptr... 0x5acdacf0, 0x105d90c30
Caught exception in looking up space point ptr... 0xcd4e6d80, 0xcd4e6bf0
Caught exception in looking up space point ptr... 0xcd4e6bf0, 0xc7e9e910
Caught exception in looking up space point ptr... 0xc7e9e910, 0xf08f4f80
Caught exception in looking up space point ptr... 0xf08f4f80, 0xcd4e6d80
Caught exception in looking up space point ptr... 0xf1b8cb50, 0xf1e8de20
Caught exception in looking up space point ptr... 0xf1e8de20, 0xf1b8cb50
Caught exception in looking up space point ptr... 0xf1b8cb50, 0xf16ffa70
Caught exception in looking up space point ptr... 0xf16ffa70, 0xf1b8cb50
Caught exception in looking up space point ptr... 0xedd83a40, 0xedd83720
Caught exception in looking up space point ptr... 0xedd83720, 0xedd838b0
Caught exception in looking up space point ptr... 0x186a4c50, 0x18796130
Caught exception in looking up space point ptr... 0x18796130, 0x186a4c50
Caught exception in looking up space point ptr... 0x3233e700, 0x10d8ba6e0
Caught exception in looking up space point ptr... 0x10d8ba6e0, 0x3233e700
Caught exception in looking up space point ptr... 0x103ced0c0, 0xa84196a0
Caught exception in looking up space point ptr... 0xa84196a0, 0xf93b8710
Caught exception in looking up space point ptr... 0x15fc2070, 0x15fc2dc0
Caught exception in looking up space point ptr... 0x15fc2dc0, 0x11190e6c0
Caught exception in looking up space point ptr... 0x9a2b8260, 0xfbea5820
Caught exception in looking up space point ptr... 0xfbea5820, 0xfc2c35f0
Caught exception in looking up space point ptr... 0xefbd8560, 0xefbd58e0
Caught exception in looking up space point ptr... 0xefbd58e0, 0xefbd8560
Caught exception in looking up space point ptr... 0x9cbe8ab0, 0x998b5860
Caught exception in looking up space point ptr... 0x998b5860, 0x9cbe8ab0
Caught exception in looking up space point ptr... 0x9cbe8ab0, 0xc0db6b50
Caught exception in looking up space point ptr... 0xc0db6b50, 0x9cbe8ab0
Caught exception in looking up space point ptr... 0x1df8b420, 0x1d9800a0
Caught exception in looking up space point ptr... 0x1d9800a0, 0x1d97ff70
Caught exception in looking up space point ptr... 0x45d35da0, 0x3f6ef500
Caught exception in looking up space point ptr... 0x3f6ef500, 0x45d35da0
++++>>>> total num hits: 8218980, num free: 8015767
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 107.959 476.48 1272.55 257.396 419.754 6
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.283e-05 8.89702e-05 0.0001581 7.39205e-05 3.21225e-05 6
produce:tpcrawdecoder:PDVDTPCReader 6.5075 6.73872 7.05789 6.67708 0.219548 6
produce:triggerrawdecoder:PDVDTriggerReader4 0.00942582 0.0231321 0.0497354 0.0156019 0.0144904 6
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0004397 0.000574765 0.00075042 0.0005811 9.88302e-05 6
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.00041734 0.00071918 0.00156802 0.000573821 0.000387191 6
produce:beamevent:BeamEvent 7.73e-05 0.00020884 0.00076315 0.00010293 0.00024825 6
produce:pdvddaphne:DAPHNEReaderPDVD 2.03449 2.24281 2.52066 2.21553 0.153658 6
produce:ophit:OpHitFinder 0.0210189 0.0342767 0.0414334 0.0352561 0.00646571 6
produce:wclsdatavd:WireCellToolkit 79.7416 86.8852 92.3013 87.1016 4.51967 6
produce:gaushit:GausHitFinder 1.03268 5.64625 14.1556 3.93776 4.67102 6
produce:nhitsfilter:NumberOfHitsFilter 0.00023297 0.000631024 0.000903361 0.000665081 0.00025738 6
produce:reco3d:SpacePointSolver 11.5114 54.5539 163.675 31.5816 52.5913 6
produce:hitpdune:DisambigFromSpacePoints 0.119732 2.14466 4.78272 2.04908 1.5893 6
produce:cluster3d:Cluster3D 1.71855 310.189 997.605 112.071 363.006 6
[art]:TriggerResults:TriggerResultInserter 2.826e-05 0.00027939 0.00142965 4.334e-05 0.000515042 6
end_path:out1:RootOutput 5.02e-06 1.01133e-05 2.102e-05 7.985e-06 5.67042e-06 6
end_path:out1:RootOutput(write) 5.09485 9.47152 18.3362 8.03444 4.64854 5
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 19000.8 MB
Peak resident set size usage (VmHWM): 16533.4 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 01:40:24 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 5 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 6 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 