Jobsub ID 303647.91@dunegpschedd01.fnal.gov
| Jobsub ID | 303647.91@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:46:24 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce515 |
| Last heartbeat | 2026-02-03 23:20:37 |
| From worker node | Hostname | b9p35p0043.cern.ch |
| cpuinfo | AMD EPYC 9655P 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 22:47:44 |
| Input files | vd-protodune:np02vd_raw_run042421_0050_df-s04-d1_dw_0_20260203T135805.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-03 23:20:37 |
| Saved logs | justin-logs:303647.91-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
up space point ptr... 0xde5d9800, 0xde5d9710
Caught exception in looking up space point ptr... 0x989c8aa0, 0x88a87a70
Caught exception in looking up space point ptr... 0x88a87a70, 0x87c77bb0
Caught exception in looking up space point ptr... 0x4f30a3b0, 0x4e88e580
Caught exception in looking up space point ptr... 0x4e88e580, 0x4e88e2b0
Caught exception in looking up space point ptr... 0xd6eed700, 0xd6eedc20
Caught exception in looking up space point ptr... 0xd6eed990, 0xd6eed700
Caught exception in looking up space point ptr... 0x324ddd40, 0x487a5eb0
Caught exception in looking up space point ptr... 0x487a5eb0, 0x324ddd40
Caught exception in looking up space point ptr... 0x2be6dc90, 0x2be3cf90
Caught exception in looking up space point ptr... 0x2be3cf90, 0x2be71f70
Caught exception in looking up space point ptr... 0xb0894f10, 0xaa8992a0
Caught exception in looking up space point ptr... 0xaa8992a0, 0xb0894f10
Caught exception in looking up space point ptr... 0x2bfb6bf0, 0x488eee40
Caught exception in looking up space point ptr... 0x488eee40, 0x48a583c0
Caught exception in looking up space point ptr... 0x2feb98a0, 0x489bac60
Caught exception in looking up space point ptr... 0x489bac60, 0x2feb98a0
Caught exception in looking up space point ptr... 0xc1340c80, 0xc0f1c5b0
Caught exception in looking up space point ptr... 0xc0f1c5b0, 0xc1423fc0
Caught exception in looking up space point ptr... 0xd1337830, 0xd027e200
Caught exception in looking up space point ptr... 0xd027e200, 0xd1337830
Caught exception in looking up space point ptr... 0x70c47690, 0x77dc1530
Caught exception in looking up space point ptr... 0x77dc1530, 0x712a7340
Caught exception in looking up space point ptr... 0xbdaa9990, 0xbdaa98a0
Caught exception in looking up space point ptr... 0xbdaa98a0, 0xbdaa9990
Caught exception in looking up space point ptr... 0x88d7fde0, 0x8c559b90
Caught exception in looking up space point ptr... 0x8c559b90, 0x88d7fde0
Caught exception in looking up space point ptr... 0x88d7fde0, 0x8e3b4390
Caught exception in looking up space point ptr... 0x8e3b4390, 0x88d7fde0
Caught exception in looking up space point ptr... 0x88d7fde0, 0x8809ba70
Caught exception in looking up space point ptr... 0x8809ba70, 0x88d7fde0
Caught exception in looking up space point ptr... 0x215828d0, 0x54dac630
Caught exception in looking up space point ptr... 0x54dac630, 0x2a285280
Caught exception in looking up space point ptr... 0x2a285280, 0x2a53b1e0
Caught exception in looking up space point ptr... 0x2a53b1e0, 0x215828d0
Caught exception in looking up space point ptr... 0x77799000, 0x67691eb0
Caught exception in looking up space point ptr... 0x67691eb0, 0x70827230
Caught exception in looking up space point ptr... 0x43e1b6e0, 0x542959f0
Caught exception in looking up space point ptr... 0x542959f0, 0x2a45c870
Caught exception in looking up space point ptr... 0x6a3bad80, 0x69dd9400
Caught exception in looking up space point ptr... 0x69dd9400, 0x6a3bad80
Caught exception in looking up space point ptr... 0x75dc8310, 0x735d94b0
Caught exception in looking up space point ptr... 0x735d94b0, 0x73d562d0
Caught exception in looking up space point ptr... 0x70b010d0, 0x71436d10
Caught exception in looking up space point ptr... 0x71436d10, 0x70b010d0
Caught exception in looking up space point ptr... 0x35512270, 0x4ca45400
Caught exception in looking up space point ptr... 0x4ca45400, 0x35512270
Caught exception in looking up space point ptr... 0x88705be0, 0x8a4fc6e0
Caught exception in looking up space point ptr... 0x8a4fc6e0, 0x883e9be0
Caught exception in looking up space point ptr... 0xd0d41450, 0xcda39ba0
Caught exception in looking up space point ptr... 0xcda39ba0, 0xc597fa30
Caught exception in looking up space point ptr... 0xc0eb1940, 0xc1340fe0
Caught exception in looking up space point ptr... 0xc1340fe0, 0xc13365e0
Caught exception in looking up space point ptr... 0xc1441080, 0xc0e2f5a0
Caught exception in looking up space point ptr... 0xc0e2f5a0, 0xc0eb1940
Caught exception in looking up space point ptr... 0xcc1b0b70, 0xcc1bd0c0
Caught exception in looking up space point ptr... 0xcc1bd0c0, 0xc7ed31b0
Caught exception in looking up space point ptr... 0xc1463ad0, 0xc134e6d0
Caught exception in looking up space point ptr... 0xc134e6d0, 0xc1463ad0
Caught exception in looking up space point ptr... 0xc1463ad0, 0xc0bfe4a0
Caught exception in looking up space point ptr... 0xc0bfe4a0, 0xc1463ad0
Caught exception in looking up space point ptr... 0x9ffe97f0, 0xa0098800
Caught exception in looking up space point ptr... 0xa0098800, 0xb7a2a850
Caught exception in looking up space point ptr... 0xc14933d0, 0xc1351520
Caught exception in looking up space point ptr... 0xc1351520, 0xc14933d0
Caught exception in looking up space point ptr... 0xc14933d0, 0xc03f8790
Caught exception in looking up space point ptr... 0xc03f8790, 0xc14933d0
Caught exception in looking up space point ptr... 0xce005180, 0xce00f760
Caught exception in looking up space point ptr... 0xce00f760, 0xce005180
Caught exception in looking up space point ptr... 0xce005180, 0xce0072b0
Caught exception in looking up space point ptr... 0xce0072b0, 0xce005180
Caught exception in looking up space point ptr... 0xce005180, 0xce00f4d0
Caught exception in looking up space point ptr... 0xce00f4d0, 0xce005180
Caught exception in looking up space point ptr... 0xb36386c0, 0xadf924b0
Caught exception in looking up space point ptr... 0xadf924b0, 0xb36386c0
Caught exception in looking up space point ptr... 0x8cfba2f0, 0x8b027680
Caught exception in looking up space point ptr... 0x8b027680, 0x8c53bf00
Caught exception in looking up space point ptr... 0xb8249a80, 0xb8f6b9f0
Caught exception in looking up space point ptr... 0xb8f6b9f0, 0xb8249a80
Caught exception in looking up space point ptr... 0x551475c0, 0x360b9600
Caught exception in looking up space point ptr... 0x360b9600, 0x54273f70
++++>>>> total num hits: 9513890, num free: 9279615
04-Feb-2026 00:19:11 CET Opened output file with pattern "%ifb_reco_stage0_%tc_offline.root"
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 1844.23 1844.23 1844.23 1844.23 0 1
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.00018436 0.00018436 0.00018436 0.00018436 0 1
produce:tpcrawdecoder:PDVDTPCReader 5.2129 5.2129 5.2129 5.2129 0 1
produce:triggerrawdecoder:PDVDTriggerReader4 0.0108603 0.0108603 0.0108603 0.0108603 0 1
produce:timingrawdecoder:PDHDTimingRawDecoder 0.00078377 0.00078377 0.00078377 0.00078377 0 1
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.00170956 0.00170956 0.00170956 0.00170956 0 1
produce:beamevent:BeamEvent 0.00083041 0.00083041 0.00083041 0.00083041 0 1
produce:pdvddaphne:DAPHNEReaderPDVD 3.59436 3.59436 3.59436 3.59436 0 1
produce:ophit:OpHitFinder 0.0364044 0.0364044 0.0364044 0.0364044 0 1
produce:wclsdatavd:WireCellToolkit 56.9721 56.9721 56.9721 56.9721 0 1
produce:gaushit:GausHitFinder 6.7531 6.7531 6.7531 6.7531 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.00099561 0.00099561 0.00099561 0.00099561 0 1
produce:reco3d:SpacePointSolver 220.599 220.599 220.599 220.599 0 1
produce:hitpdune:DisambigFromSpacePoints 6.54107 6.54107 6.54107 6.54107 0 1
produce:cluster3d:Cluster3D 1544.47 1544.47 1544.47 1544.47 0 1
[art]:TriggerResults:TriggerResultInserter 9.879e-05 9.879e-05 9.879e-05 9.879e-05 0 1
end_path:out1:RootOutput 2.385e-05 2.385e-05 2.385e-05 2.385e-05 0 1
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 19882.1 MB
Peak resident set size usage (VmHWM): 17708.9 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:19:29 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 1 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 