Jobsub ID 303651.73@dunegpschedd01.fnal.gov
| Jobsub ID | 303651.73@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:50:24 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce507 |
| Last heartbeat | 2026-02-03 23:40:27 |
| From worker node | Hostname | b9p35p0020.cern.ch |
| cpuinfo | AMD EPYC 9655P 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:19:46 |
| Input files | vd-protodune:np02vd_raw_run042421_0050_df-s04-d1_dw_0_20260203T135805.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-03 23:40:27 |
| Saved logs | justin-logs:303651.73-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
up space point ptr... 0xb491bab0, 0xb47a9fa0
Caught exception in looking up space point ptr... 0x9b8f08c0, 0x9b6a36a0
Caught exception in looking up space point ptr... 0x9b6a36a0, 0x9b8eed20
Caught exception in looking up space point ptr... 0x29bab4e0, 0x4a59e140
Caught exception in looking up space point ptr... 0x3b2417d0, 0x29bab4e0
Caught exception in looking up space point ptr... 0xa2fd2f90, 0xa29c3660
Caught exception in looking up space point ptr... 0xa29646e0, 0xa29d2460
Caught exception in looking up space point ptr... 0xa29d2460, 0xa2fd2f90
Caught exception in looking up space point ptr... 0x8a2ed420, 0x8a2ecf00
Caught exception in looking up space point ptr... 0x8a2ecc70, 0x8a2ed420
Caught exception in looking up space point ptr... 0xa2c366d0, 0xa2c2d740
Caught exception in looking up space point ptr... 0xa2be5950, 0xa2c366d0
Caught exception in looking up space point ptr... 0x976a3210, 0x8d537980
Caught exception in looking up space point ptr... 0x8d537980, 0x976a3210
Caught exception in looking up space point ptr... 0xa9bbda20, 0xa9bbf6b0
Caught exception in looking up space point ptr... 0xa9bbf6b0, 0xa9ba3dd0
Caught exception in looking up space point ptr... 0x626e2e40, 0x626e30d0
Caught exception in looking up space point ptr... 0x626e30d0, 0x626e2bb0
Caught exception in looking up space point ptr... 0x5fe85340, 0x5e261fb0
Caught exception in looking up space point ptr... 0x664b1cd0, 0x5fe85340
Caught exception in looking up space point ptr... 0x6854d5f0, 0x6854d460
Caught exception in looking up space point ptr... 0x6854d460, 0x6854d5f0
Caught exception in looking up space point ptr... 0x3e3ad690, 0x3e3ad9b0
Caught exception in looking up space point ptr... 0x3e3ad9b0, 0x3e3ad820
Caught exception in looking up space point ptr... 0x696e8930, 0x696e87a0
Caught exception in looking up space point ptr... 0x696e87a0, 0x696e8930
Caught exception in looking up space point ptr... 0x7a6558b0, 0x7a655270
Caught exception in looking up space point ptr... 0x7a655270, 0x7a6558b0
Caught exception in looking up space point ptr... 0x7a6558b0, 0x7a655720
Caught exception in looking up space point ptr... 0x7a655720, 0x7a6558b0
Caught exception in looking up space point ptr... 0x83f9d3b0, 0x83f9d220
Caught exception in looking up space point ptr... 0x83f9d220, 0x83f9d540
Caught exception in looking up space point ptr... 0x297e8af0, 0x28817b10
Caught exception in looking up space point ptr... 0x28817b10, 0x1d924b80
Caught exception in looking up space point ptr... 0x4e74a170, 0x4e74b760
Caught exception in looking up space point ptr... 0x4e74b760, 0x4e74a170
Caught exception in looking up space point ptr... 0x5fdc2600, 0x5fb72bc0
Caught exception in looking up space point ptr... 0x5fb72bc0, 0x5fdc2600
Caught exception in looking up space point ptr... 0x5fdc2600, 0x5faf3a30
Caught exception in looking up space point ptr... 0x5faf3a30, 0x5fdc2600
Caught exception in looking up space point ptr... 0x5fdc2600, 0x5fdc2890
Caught exception in looking up space point ptr... 0x5fdc2890, 0x5fdc2600
Caught exception in looking up space point ptr... 0xa519f140, 0xa519f230
Caught exception in looking up space point ptr... 0xa519f230, 0xa519f140
Caught exception in looking up space point ptr... 0x8ca97d80, 0x85c6d770
Caught exception in looking up space point ptr... 0x85c6d770, 0x85c6d8b0
Caught exception in looking up space point ptr... 0x85c6d8b0, 0x8ca97f10
Caught exception in looking up space point ptr... 0x8ca97f10, 0x8ca97d80
Caught exception in looking up space point ptr... 0xb3ce1ee0, 0xb411ac20
Caught exception in looking up space point ptr... 0xb411ac20, 0xb411ae70
Caught exception in looking up space point ptr... 0x393f4ab0, 0x4c1fc560
Caught exception in looking up space point ptr... 0x499d3340, 0x4a336fd0
Caught exception in looking up space point ptr... 0x4a336fd0, 0x4faca220
Caught exception in looking up space point ptr... 0x4faca220, 0x393f4ab0
Caught exception in looking up space point ptr... 0xa1aa0c10, 0xa1a4ed10
Caught exception in looking up space point ptr... 0xa1a4ed10, 0xa1a589b0
Caught exception in looking up space point ptr... 0x42371d00, 0x494db190
Caught exception in looking up space point ptr... 0x494db190, 0x494db7d0
Caught exception in looking up space point ptr... 0x494db7d0, 0x494db640
Caught exception in looking up space point ptr... 0x494db640, 0x494db4b0
Caught exception in looking up space point ptr... 0x494db4b0, 0x494db7d0
Caught exception in looking up space point ptr... 0x423544e0, 0x2d736690
Caught exception in looking up space point ptr... 0x2d736690, 0x494de390
Caught exception in looking up space point ptr... 0xb6a0cd80, 0xb6a0cba0
Caught exception in looking up space point ptr... 0xb6a0cba0, 0xb6a0cd80
Caught exception in looking up space point ptr... 0xad2e7df0, 0xae377520
Caught exception in looking up space point ptr... 0xae377520, 0xad2e7c60
Caught exception in looking up space point ptr... 0x62dc36d0, 0x5ab8f520
Caught exception in looking up space point ptr... 0x5ab8f520, 0x62dc36d0
Caught exception in looking up space point ptr... 0x62dc36d0, 0x5e9f9050
Caught exception in looking up space point ptr... 0x5e9f9050, 0x62dc36d0
Caught exception in looking up space point ptr... 0x62dc36d0, 0x5e1a9b70
Caught exception in looking up space point ptr... 0x5e1a9b70, 0x62dc36d0
Caught exception in looking up space point ptr... 0xb03ff950, 0xb0651060
Caught exception in looking up space point ptr... 0xb0651060, 0xb03ff950
Caught exception in looking up space point ptr... 0x40660070, 0x406a6d70
Caught exception in looking up space point ptr... 0x406a6d70, 0x406da880
Caught exception in looking up space point ptr... 0x404f9790, 0x4048ab90
Caught exception in looking up space point ptr... 0x4048ab90, 0x404f9790
Caught exception in looking up space point ptr... 0xa09820e0, 0xa0981fa0
Caught exception in looking up space point ptr... 0xa0981fa0, 0xa09820e0
++++>>>> total num hits: 7250104, num free: 7037074
04-Feb-2026 00:39:43 CET Opened output file with pattern "%ifb_reco_stage0_%tc_offline.root"
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 1160.37 1160.37 1160.37 1160.37 0 1
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000146541 0.000146541 0.000146541 0.000146541 0 1
produce:tpcrawdecoder:PDVDTPCReader 4.49545 4.49545 4.49545 4.49545 0 1
produce:triggerrawdecoder:PDVDTriggerReader4 0.0117627 0.0117627 0.0117627 0.0117627 0 1
produce:timingrawdecoder:PDHDTimingRawDecoder 0.000838546 0.000838546 0.000838546 0.000838546 0 1
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.00227383 0.00227383 0.00227383 0.00227383 0 1
produce:beamevent:BeamEvent 0.000959867 0.000959867 0.000959867 0.000959867 0 1
produce:pdvddaphne:DAPHNEReaderPDVD 1.88518 1.88518 1.88518 1.88518 0 1
produce:ophit:OpHitFinder 0.0292155 0.0292155 0.0292155 0.0292155 0 1
produce:wclsdatavd:WireCellToolkit 44.5509 44.5509 44.5509 44.5509 0 1
produce:gaushit:GausHitFinder 6.12868 6.12868 6.12868 6.12868 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.000890426 0.000890426 0.000890426 0.000890426 0 1
produce:reco3d:SpacePointSolver 126.258 126.258 126.258 126.258 0 1
produce:hitpdune:DisambigFromSpacePoints 5.41144 5.41144 5.41144 5.41144 0 1
produce:cluster3d:Cluster3D 971.565 971.565 971.565 971.565 0 1
[art]:TriggerResults:TriggerResultInserter 8.157e-05 8.157e-05 8.157e-05 8.157e-05 0 1
end_path:out1:RootOutput 1.803e-05 1.803e-05 1.803e-05 1.803e-05 0 1
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 17329.7 MB
Peak resident set size usage (VmHWM): 14598.3 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:39:55 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 1 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 