Jobsub ID 303656.8@dunegpschedd01.fnal.gov
| Jobsub ID | 303656.8@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:56:25 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2026-02-04 00:50:13 |
| From worker node | Hostname | wn-s04-05-32-01.cr.cnaf.infn.it |
| cpuinfo | AMD EPYC 7313 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 00:11:53 |
| Input files | vd-protodune:np02vd_raw_run042421_0143_df-s04-d0_dw_0_20260203T144820.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 00:50:13 |
| Saved logs | justin-logs:303656.8-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
a670, 0x9b079660
Caught exception in looking up space point ptr... 0x9b079660, 0xb1f4a670
Caught exception in looking up space point ptr... 0x108546ea0, 0x10815bad0
Caught exception in looking up space point ptr... 0x10815bad0, 0x108546ea0
Caught exception in looking up space point ptr... 0x54547550, 0x412eb690
Caught exception in looking up space point ptr... 0x412eb690, 0x54547550
Caught exception in looking up space point ptr... 0x54547550, 0x412eb370
Caught exception in looking up space point ptr... 0x412eb370, 0x54547550
Caught exception in looking up space point ptr... 0x54547550, 0x545473c0
Caught exception in looking up space point ptr... 0x545473c0, 0x54547550
Caught exception in looking up space point ptr... 0x54544fd0, 0x412e9a70
Caught exception in looking up space point ptr... 0x412e9a70, 0x54544fd0
Caught exception in looking up space point ptr... 0x54544fd0, 0x412e9750
Caught exception in looking up space point ptr... 0x412e9750, 0x54544fd0
Caught exception in looking up space point ptr... 0x54544fd0, 0x54544e40
Caught exception in looking up space point ptr... 0x54544e40, 0x54544fd0
Caught exception in looking up space point ptr... 0xe71a2a90, 0xe803cf40
Caught exception in looking up space point ptr... 0xe803cf40, 0xe71a2a90
Caught exception in looking up space point ptr... 0x5d9314a0, 0x41886a50
Caught exception in looking up space point ptr... 0x41886a50, 0x5d9314a0
Caught exception in looking up space point ptr... 0x924d5f00, 0x9d018290
Caught exception in looking up space point ptr... 0x9d018290, 0x9e11dca0
Caught exception in looking up space point ptr... 0xfc12e7d0, 0xfe935610
Caught exception in looking up space point ptr... 0xfe935610, 0xfc12e7d0
Caught exception in looking up space point ptr... 0xce044d00, 0xce044b70
Caught exception in looking up space point ptr... 0xce044b70, 0xce044d00
Caught exception in looking up space point ptr... 0x106d9a260, 0x106d9a0d0
Caught exception in looking up space point ptr... 0x106d9a0d0, 0x106d9a260
Caught exception in looking up space point ptr... 0x10fe4cde0, 0x10fe29d40
Caught exception in looking up space point ptr... 0x10fe29d40, 0x10fe4cde0
Caught exception in looking up space point ptr... 0x10fe19c30, 0x10fe57c30
Caught exception in looking up space point ptr... 0x10fe57c30, 0x10fe19c30
Caught exception in looking up space point ptr... 0x5451ece0, 0x2e276600
Caught exception in looking up space point ptr... 0x2e276600, 0x2e276500
Caught exception in looking up space point ptr... 0x10fe30db0, 0x10fdf6f40
Caught exception in looking up space point ptr... 0x10fdf6f40, 0x10fe30db0
Caught exception in looking up space point ptr... 0xdaf78970, 0xe29d4a30
Caught exception in looking up space point ptr... 0xe29d4a30, 0xdaf78970
Caught exception in looking up space point ptr... 0x109c10c20, 0x109c10900
Caught exception in looking up space point ptr... 0x109c10900, 0x109c10c20
Caught exception in looking up space point ptr... 0x10fe1a4e0, 0x10fe15310
Caught exception in looking up space point ptr... 0x10fe15310, 0x10fe1a4e0
Caught exception in looking up space point ptr... 0xe97d4850, 0xe90b77c0
Caught exception in looking up space point ptr... 0xe90b77c0, 0xe97d4850
Caught exception in looking up space point ptr... 0x10fe40bb0, 0x10fe222f0
Caught exception in looking up space point ptr... 0x10fe222f0, 0x10fe40bb0
Caught exception in looking up space point ptr... 0x10fe42260, 0x10fe57d90
Caught exception in looking up space point ptr... 0x10fe57d90, 0x10fe42260
Caught exception in looking up space point ptr... 0x10fe25a80, 0x10fe33630
Caught exception in looking up space point ptr... 0x10fe33630, 0x10fe30590
Caught exception in looking up space point ptr... 0x10e870860, 0x10d2f5010
Caught exception in looking up space point ptr... 0x10d2f5010, 0x10e870860
Caught exception in looking up space point ptr... 0x10e870860, 0x10d2f4f20
Caught exception in looking up space point ptr... 0x10d2f4f20, 0x10e870860
Caught exception in looking up space point ptr... 0x10e870860, 0x10cacb670
Caught exception in looking up space point ptr... 0x10cacb670, 0x10e870860
Caught exception in looking up space point ptr... 0x10e870860, 0x10c296a30
Caught exception in looking up space point ptr... 0x10c296a30, 0x10e870860
Caught exception in looking up space point ptr... 0x10e870860, 0x10dabdd40
Caught exception in looking up space point ptr... 0x10dabdd40, 0x10e870860
Caught exception in looking up space point ptr... 0x10e870860, 0x10d5b8eb0
Caught exception in looking up space point ptr... 0x10d5b8eb0, 0x10e870860
Caught exception in looking up space point ptr... 0xf567450, 0x968f800
Caught exception in looking up space point ptr... 0x968f800, 0xf567450
Caught exception in looking up space point ptr... 0x7c291940, 0x149e62f0
Caught exception in looking up space point ptr... 0x149e62f0, 0x7c291940
Caught exception in looking up space point ptr... 0x10e57f710, 0x10de2e410
Caught exception in looking up space point ptr... 0x10de2e410, 0x10e57f710
Caught exception in looking up space point ptr... 0x7ed1bc50, 0x7ed1bac0
Caught exception in looking up space point ptr... 0x7ed1bac0, 0x7ed1bde0
Caught exception in looking up space point ptr... 0x7ed1bde0, 0x7ed1b930
Caught exception in looking up space point ptr... 0x7ed1b930, 0x7ed1bf70
Caught exception in looking up space point ptr... 0x31765270, 0x3989f500
Caught exception in looking up space point ptr... 0x3989f820, 0x31765270
Caught exception in looking up space point ptr... 0x3eb0bdf0, 0x3eb0bad0
Caught exception in looking up space point ptr... 0x3eb0bad0, 0x3eb0bc60
Caught exception in looking up space point ptr... 0x3eb01210, 0x3eb01080
Caught exception in looking up space point ptr... 0x3eb01080, 0x3eb01210
Caught exception in looking up space point ptr... 0x3e0864c0, 0x404c8640
Caught exception in looking up space point ptr... 0x404c8640, 0x3e0864c0
++++>>>> total num hits: 10695676, num free: 10443582
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 77.8373 360.994 1412.69 97.1123 526.111 5
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 7.1194e-05 0.000125414 0.000252063 9.4567e-05 6.49978e-05 5
produce:tpcrawdecoder:PDVDTPCReader 7.34013 7.50943 7.6735 7.51086 0.108849 5
produce:triggerrawdecoder:PDVDTriggerReader4 0.301521 0.30853 0.324249 0.302914 0.00861177 5
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0100579 0.0102205 0.010393 0.0102136 0.000126168 5
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0498896 0.0502623 0.0507634 0.0500091 0.000403858 5
produce:beamevent:BeamEvent 7.3919e-05 0.000268833 0.000962097 9.3625e-05 0.000347045 5
produce:pdvddaphne:DAPHNEReaderPDVD 2.58657 2.68551 2.7993 2.69244 0.0732556 5
produce:ophit:OpHitFinder 0.0297985 0.0326062 0.034487 0.033276 0.00188531 5
produce:wclsdatavd:WireCellToolkit 51.0372 57.0981 64.6599 57.2568 4.45739 5
produce:gaushit:GausHitFinder 0.536311 2.38721 7.63084 1.32094 2.6415 5
produce:nhitsfilter:NumberOfHitsFilter 0.000195898 0.000469596 0.000903246 0.000297459 0.000281526 5
produce:reco3d:SpacePointSolver 4.35009 39.9851 171.501 6.60265 65.8032 5
produce:hitpdune:DisambigFromSpacePoints 0.044184 1.046 4.65328 0.185076 1.80454 5
produce:cluster3d:Cluster3D 0.886296 246.791 1163.5 26.5171 458.583 5
[art]:TriggerResults:TriggerResultInserter 2.1711e-05 3.6188e-05 7.9089e-05 2.7952e-05 2.16078e-05 5
end_path:out1:RootOutput 5.29e-06 9.4138e-06 2.2633e-05 6.753e-06 6.63999e-06 5
end_path:out1:RootOutput(write) 2.60707 3.84138 5.04185 3.85829 0.920142 4
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 20648.7 MB
Peak resident set size usage (VmHWM): 18225.5 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 01:47:30 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 5 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 