Jobsub ID 297917.114@dunegpschedd02.fnal.gov
| Jobsub ID | 297917.114@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:48:25 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce505 |
| Last heartbeat | 2026-02-03 23:52:53 |
| From worker node | Hostname | b9p35p0031.cern.ch |
| cpuinfo | AMD EPYC 9655P 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:22:36 |
| Input files | vd-protodune:np02vd_raw_run042421_0143_df-s04-d0_dw_0_20260203T144820.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-03 23:52:53 |
| Saved logs | justin-logs:297917.114-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
0x30b1d220
Caught exception in looking up space point ptr... 0xa2943540, 0xa4d12090
Caught exception in looking up space point ptr... 0xa44a44a0, 0xa2943540
Caught exception in looking up space point ptr... 0xc0ea9d50, 0xc09b0b40
Caught exception in looking up space point ptr... 0xc09b0b40, 0xc0ea9d50
Caught exception in looking up space point ptr... 0x71a4d9d0, 0x578b2900
Caught exception in looking up space point ptr... 0x578b2900, 0x71a4d9d0
Caught exception in looking up space point ptr... 0xb7c69bc0, 0xb7c5fb20
Caught exception in looking up space point ptr... 0xb7c5fb20, 0xb7c69bc0
Caught exception in looking up space point ptr... 0x24565c80, 0x245661a0
Caught exception in looking up space point ptr... 0x245661a0, 0x24566430
Caught exception in looking up space point ptr... 0xc5ddc990, 0xc5ddd0b0
Caught exception in looking up space point ptr... 0xc5ddd0b0, 0xc5dddef0
Caught exception in looking up space point ptr... 0xc5d8a870, 0xc5d771f0
Caught exception in looking up space point ptr... 0xc5d754a0, 0xc5d8a870
Caught exception in looking up space point ptr... 0x8484fa40, 0x8484f8b0
Caught exception in looking up space point ptr... 0x8484f8b0, 0x84851350
Caught exception in looking up space point ptr... 0xb6019480, 0xb60193e0
Caught exception in looking up space point ptr... 0xb60193e0, 0xb6019480
Caught exception in looking up space point ptr... 0x4a4c3da0, 0x4275b1e0
Caught exception in looking up space point ptr... 0x4275b1e0, 0x4a4c3da0
Caught exception in looking up space point ptr... 0x4a4c3da0, 0x2225a090
Caught exception in looking up space point ptr... 0x2225a090, 0x4a4c3da0
Caught exception in looking up space point ptr... 0x4a4c3da0, 0x3d44a040
Caught exception in looking up space point ptr... 0x3d44a040, 0x4a4c3da0
Caught exception in looking up space point ptr... 0xb6209940, 0xb7ae5610
Caught exception in looking up space point ptr... 0xb7ae5610, 0xb6209a30
Caught exception in looking up space point ptr... 0xbc9aef00, 0xc44e1ee0
Caught exception in looking up space point ptr... 0xc44e1ee0, 0xbc9aef00
Caught exception in looking up space point ptr... 0x5b2dab60, 0x5b2da8d0
Caught exception in looking up space point ptr... 0x5b2da8d0, 0x5b2dab60
Caught exception in looking up space point ptr... 0xbe720010, 0xbe639a90
Caught exception in looking up space point ptr... 0xbe71ad20, 0xbe720010
Caught exception in looking up space point ptr... 0x11484670, 0x1132d120
Caught exception in looking up space point ptr... 0x1132d120, 0x114d45a0
Caught exception in looking up space point ptr... 0xbbde8b00, 0xbbde8d30
Caught exception in looking up space point ptr... 0xbbde8d30, 0xbbde8e20
Caught exception in looking up space point ptr... 0xb9c23550, 0xb4aa6be0
Caught exception in looking up space point ptr... 0xb4aa6be0, 0xb8eefb60
Caught exception in looking up space point ptr... 0x33ac17a0, 0x33b01420
Caught exception in looking up space point ptr... 0x33ac4070, 0x33ac17a0
Caught exception in looking up space point ptr... 0x9aa48ee0, 0x9ba79720
Caught exception in looking up space point ptr... 0x9b956590, 0x9aa48ee0
Caught exception in looking up space point ptr... 0xbac80a00, 0xbab99240
Caught exception in looking up space point ptr... 0xbac80af0, 0xbac80a00
Caught exception in looking up space point ptr... 0xab5f37c0, 0xab5f3630
Caught exception in looking up space point ptr... 0xab5f3630, 0xab5f37c0
Caught exception in looking up space point ptr... 0xcad423b0, 0xcad425b0
Caught exception in looking up space point ptr... 0xcad425b0, 0xcad423b0
Caught exception in looking up space point ptr... 0xc45dcbd0, 0xc45dd0f0
Caught exception in looking up space point ptr... 0xc45dd0f0, 0xc4e9a8d0
Caught exception in looking up space point ptr... 0x8dabe050, 0x8e9ec800
Caught exception in looking up space point ptr... 0x8e9ec800, 0x8e9ec990
Caught exception in looking up space point ptr... 0x7dab5480, 0x7d9643a0
Caught exception in looking up space point ptr... 0x7d9643a0, 0x7da91720
Caught exception in looking up space point ptr... 0xc08787d0, 0xc0878aa0
Caught exception in looking up space point ptr... 0xc0878aa0, 0xc08788c0
Caught exception in looking up space point ptr... 0xbf741220, 0xbf415ee0
Caught exception in looking up space point ptr... 0xbf4160c0, 0xbf415fd0
Caught exception in looking up space point ptr... 0xbf415fd0, 0xbf7412c0
Caught exception in looking up space point ptr... 0xbf7412c0, 0xbf741220
Caught exception in looking up space point ptr... 0xc8059ac0, 0xc8059d90
Caught exception in looking up space point ptr... 0xc8059d90, 0xc846afb0
Caught exception in looking up space point ptr... 0xc846afb0, 0xc8059bb0
Caught exception in looking up space point ptr... 0xc8059bb0, 0xc846b0b0
Caught exception in looking up space point ptr... 0x82f00580, 0x7f8df090
Caught exception in looking up space point ptr... 0x7f8df090, 0x82f00880
Caught exception in looking up space point ptr... 0x14d8de80, 0x4a4cb480
Caught exception in looking up space point ptr... 0x4a4cb480, 0x14d8de80
Caught exception in looking up space point ptr... 0xc93f3330, 0xc817a320
Caught exception in looking up space point ptr... 0xc817a320, 0xc93f3650
Caught exception in looking up space point ptr... 0xa3df3230, 0xa5351fc0
Caught exception in looking up space point ptr... 0xa5351fc0, 0xa5352150
Caught exception in looking up space point ptr... 0xc0aac410, 0xc0b75930
Caught exception in looking up space point ptr... 0xc0b75930, 0xc3a15460
Caught exception in looking up space point ptr... 0xc3a15460, 0xc0b71870
Caught exception in looking up space point ptr... 0xc0b71870, 0xc0aac410
Caught exception in looking up space point ptr... 0xb68de150, 0xb603f260
Caught exception in looking up space point ptr... 0xb603f260, 0xb68de150
Caught exception in looking up space point ptr... 0xb54e5e00, 0xb738d8a0
Caught exception in looking up space point ptr... 0xb738d8a0, 0xb5675320
++++>>>> total num hits: 7225213, num free: 7039011
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 64.5022 344.683 1366.47 104.712 511.231 5
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.367e-05 8.5684e-05 0.00015789 6.909e-05 3.62285e-05 5
produce:tpcrawdecoder:PDVDTPCReader 3.55151 3.77187 4.0176 3.68129 0.190863 5
produce:triggerrawdecoder:PDVDTriggerReader4 0.00954009 0.0119833 0.0156087 0.0116304 0.00216096 5
produce:timingrawdecoder:PDHDTimingRawDecoder 0.00042647 0.000662314 0.00093977 0.00066809 0.00019783 5
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.00139995 0.00184971 0.00252935 0.00170519 0.000435522 5
produce:beamevent:BeamEvent 9.104e-05 0.000262372 0.00090615 0.00010315 0.000321981 5
produce:pdvddaphne:DAPHNEReaderPDVD 1.78411 2.07186 2.27398 2.08726 0.163422 5
produce:ophit:OpHitFinder 0.0299343 0.0333483 0.0370669 0.0331091 0.00291568 5
produce:wclsdatavd:WireCellToolkit 48.3089 52.1097 58.1874 50.8516 3.51102 5
produce:gaushit:GausHitFinder 0.465635 1.56828 4.08695 1.04846 1.28742 5
produce:nhitsfilter:NumberOfHitsFilter 0.00013888 0.000302972 0.00059095 0.00024603 0.00016346 5
produce:reco3d:SpacePointSolver 4.90492 31.4559 131.199 5.85779 49.9006 5
produce:hitpdune:DisambigFromSpacePoints 0.0420864 1.10096 5.01307 0.120535 1.95686 5
produce:cluster3d:Cluster3D 0.696845 249.555 1166.71 38.8109 458.883 5
[art]:TriggerResults:TriggerResultInserter 2.296e-05 5.0654e-05 9.327e-05 2.933e-05 2.92641e-05 5
end_path:out1:RootOutput 4.77e-06 1.0158e-05 2.211e-05 5.03e-06 6.95288e-06 5
end_path:out1:RootOutput(write) 3.09628 3.70597 4.14509 3.79126 0.399074 4
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 15024 MB
Peak resident set size usage (VmHWM): 12692.2 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:52:15 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 5 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 