Jobsub ID 303650.135@dunegpschedd01.fnal.gov
| Jobsub ID | 303650.135@dunegpschedd01.fnal.gov |
| Workflow ID | 12719 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:48:28 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2026-02-04 01:08:35 |
| From worker node | Hostname | wn-s04-06-01-02.cr.cnaf.infn.it |
| cpuinfo | AMD EPYC 7351 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:16:55 |
| Input files | vd-protodune:np02vd_raw_run042433_0586_df-s03-d1_dw_0_20260203T222451.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:08:35 |
| Saved logs | justin-logs:303650.135-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
, 0x7a18bb50
Caught exception in looking up space point ptr... 0x7a18bb50, 0x7a226280
Caught exception in looking up space point ptr... 0x7a226280, 0x7a225f60
Caught exception in looking up space point ptr... 0x7a225f60, 0x7a226280
Caught exception in looking up space point ptr... 0xa3010450, 0xa2bba6f0
Caught exception in looking up space point ptr... 0xa2bba6f0, 0xa3010450
Caught exception in looking up space point ptr... 0x5f74fd50, 0x8d0984f0
Caught exception in looking up space point ptr... 0x8d0984f0, 0x5f74fd50
Caught exception in looking up space point ptr... 0xabe1a1e0, 0xabd41d20
Caught exception in looking up space point ptr... 0xabd41d20, 0xabe1a1e0
Caught exception in looking up space point ptr... 0xb05665c0, 0xb9b5f440
Caught exception in looking up space point ptr... 0xb9b5f440, 0xb21c1810
Caught exception in looking up space point ptr... 0xbe54df20, 0xbef79990
Caught exception in looking up space point ptr... 0xbef79990, 0xbe54df20
Caught exception in looking up space point ptr... 0x7ce73810, 0x7bf6d340
Caught exception in looking up space point ptr... 0x7bf6d340, 0x7ce73810
Caught exception in looking up space point ptr... 0xa9bf3d90, 0xac3fa150
Caught exception in looking up space point ptr... 0xa7448640, 0xa9bf3d90
Caught exception in looking up space point ptr... 0x110ecc00, 0xd576c10
Caught exception in looking up space point ptr... 0xd576c10, 0x110ecc00
Caught exception in looking up space point ptr... 0xd58df00, 0xd58dd70
Caught exception in looking up space point ptr... 0xd58dd70, 0x13df78d0
Caught exception in looking up space point ptr... 0xe4946c30, 0xe4034070
Caught exception in looking up space point ptr... 0xe4034070, 0xe4946c30
Caught exception in looking up space point ptr... 0x16143250, 0x16143150
Caught exception in looking up space point ptr... 0x732a7660, 0x16143250
Caught exception in looking up space point ptr... 0x494e0bc0, 0x33371c70
Caught exception in looking up space point ptr... 0x33371c70, 0x4950af60
Caught exception in looking up space point ptr... 0x4950af60, 0x40335900
Caught exception in looking up space point ptr... 0x40335900, 0x49519910
Caught exception in looking up space point ptr... 0x49519910, 0x494e0bc0
Caught exception in looking up space point ptr... 0x6ad28750, 0x6ad27fa0
Caught exception in looking up space point ptr... 0x6ad27fa0, 0x6ad284c0
Caught exception in looking up space point ptr... 0x6ae46740, 0x567e9910
Caught exception in looking up space point ptr... 0x567e9910, 0x6ae46740
Caught exception in looking up space point ptr... 0xc1f16b10, 0xc2de2110
Caught exception in looking up space point ptr... 0xc2de2110, 0xc2de22a0
Caught exception in looking up space point ptr... 0x6d3fe5d0, 0x5abb13d0
Caught exception in looking up space point ptr... 0x5abb13d0, 0x6d3fe5d0
Caught exception in looking up space point ptr... 0x110ec700, 0x5183f7a0
Caught exception in looking up space point ptr... 0x5183f7a0, 0x110ec700
Caught exception in looking up space point ptr... 0xeb563760, 0xeb7a9e80
Caught exception in looking up space point ptr... 0xeb7a9e80, 0xeb563800
Caught exception in looking up space point ptr... 0xe1401680, 0xd4c8cb80
Caught exception in looking up space point ptr... 0xd4c8cb80, 0xe1401680
Caught exception in looking up space point ptr... 0xe1401680, 0xe1c1e510
Caught exception in looking up space point ptr... 0xe1c1e510, 0xe1401680
Caught exception in looking up space point ptr... 0xe1401680, 0xe35bec70
Caught exception in looking up space point ptr... 0xe35bec70, 0xe1401680
Caught exception in looking up space point ptr... 0x6aa97160, 0x6aa96ed0
Caught exception in looking up space point ptr... 0x6aa969b0, 0x6aa97160
Caught exception in looking up space point ptr... 0xbdac45d0, 0xbdb225e0
Caught exception in looking up space point ptr... 0xbdad7060, 0xbdac45d0
Caught exception in looking up space point ptr... 0x83983a60, 0x7bede1d0
Caught exception in looking up space point ptr... 0x7bede1d0, 0x59cc3170
Caught exception in looking up space point ptr... 0xcc8ad680, 0xcc916bb0
Caught exception in looking up space point ptr... 0xcc916bb0, 0xcc8ad680
Caught exception in looking up space point ptr... 0x3886f480, 0x388742d0
Caught exception in looking up space point ptr... 0x388742d0, 0x387f2010
Caught exception in looking up space point ptr... 0x85578c00, 0x544fd460
Caught exception in looking up space point ptr... 0x544fd460, 0x85578c00
Caught exception in looking up space point ptr... 0xea129f00, 0xea129e60
Caught exception in looking up space point ptr... 0xea129e60, 0xea129f00
Caught exception in looking up space point ptr... 0xa06c36e0, 0xa39f3480
Caught exception in looking up space point ptr... 0xa7d229a0, 0xa06c36e0
Caught exception in looking up space point ptr... 0xade85fe0, 0xadfd4540
Caught exception in looking up space point ptr... 0xadfd4540, 0xade85fe0
Caught exception in looking up space point ptr... 0x82a4f4c0, 0x82a4e640
Caught exception in looking up space point ptr... 0x82a4e640, 0x82a4e810
Caught exception in looking up space point ptr... 0xdfd534c0, 0xe41ce0a0
Caught exception in looking up space point ptr... 0xe41ce0a0, 0xdfd534c0
Caught exception in looking up space point ptr... 0x90981e80, 0x7c55e6d0
Caught exception in looking up space point ptr... 0x7c55e6d0, 0x90981e80
Caught exception in looking up space point ptr... 0x90981e80, 0x707adfb0
Caught exception in looking up space point ptr... 0x707adfb0, 0x90981e80
Caught exception in looking up space point ptr... 0x90981e80, 0x90674f20
Caught exception in looking up space point ptr... 0x90674f20, 0x90981e80
Caught exception in looking up space point ptr... 0xdc121ca0, 0xdc121980
Caught exception in looking up space point ptr... 0xdc121980, 0xdc121ca0
Caught exception in looking up space point ptr... 0xb60e4400, 0xb3dad6d0
Caught exception in looking up space point ptr... 0xb3dad6d0, 0xb60e4400
++++>>>> total num hits: 8195784, num free: 8000444
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 195.968 393.68 2486.37 261.257 540.958 16
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 8.8091e-05 0.000157155 0.000288213 0.000152377 4.94541e-05 16
produce:tpcrawdecoder:PDVDTPCReader 23.8086 31.2678 41.0058 31.2041 5.13051 16
produce:triggerrawdecoder:PDVDTriggerReader4 0.321172 0.390219 0.609503 0.362214 0.0712037 16
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0104448 0.0171977 0.0273049 0.0169273 0.00624187 16
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.010105 0.0106769 0.0118641 0.0106458 0.000392474 16
produce:beamevent:BeamEvent 9.2721e-05 0.000244573 0.00131388 0.000172802 0.000280909 16
produce:pdvddaphne:DAPHNEReaderPDVD 4.30786 5.57372 7.78366 5.40154 0.848975 16
produce:ophit:OpHitFinder 0.0446006 0.0559283 0.0721649 0.0548169 0.00585012 16
produce:wclsdatavd:WireCellToolkit 138.483 176.056 214.195 178.231 17.5137 16
produce:gaushit:GausHitFinder 1.14691 3.0551 14.9639 2.35382 3.14004 16
produce:nhitsfilter:NumberOfHitsFilter 0.000388995 0.000625489 0.00151009 0.000559962 0.000265488 16
produce:reco3d:SpacePointSolver 11.1759 39.4315 236.33 25.6176 51.5314 16
produce:hitpdune:DisambigFromSpacePoints 0.120416 0.784603 7.66415 0.330151 1.77955 16
produce:cluster3d:Cluster3D 2.37577 129.269 2000.43 4.74508 483.136 16
[art]:TriggerResults:TriggerResultInserter 2.593e-05 4.80851e-05 0.000117351 4.04355e-05 2.39002e-05 16
end_path:out1:RootOutput 6.68e-06 1.19409e-05 2.9941e-05 1.0145e-05 6.20921e-06 16
end_path:out1:RootOutput(write) 5.84224 8.23829 10.2313 8.5548 1.24728 15
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 17925.4 MB
Peak resident set size usage (VmHWM): 15543.3 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 02:06:55 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 15 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 16 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 