Jobsub ID 297980.1@dunegpschedd02.fnal.gov
| Jobsub ID | 297980.1@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-04 00:48:30 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce514 |
| Last heartbeat | 2026-02-04 01:18:50 |
| From worker node | Hostname | b9p08p6650.cern.ch |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 00:49:10 |
| Input files | vd-protodune:np02vd_raw_run042421_0010_df-s02-d0_dw_0_20260203T133622.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:18:50 |
| Saved logs | justin-logs:297980.1-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
up space point ptr... 0x193e7730, 0x19185720
Caught exception in looking up space point ptr... 0x41f70110, 0x56d5f7b0
Caught exception in looking up space point ptr... 0x56d5f7b0, 0x19ecd100
Caught exception in looking up space point ptr... 0x6bf91140, 0x6bbee230
Caught exception in looking up space point ptr... 0x6bbee230, 0x6bf91140
Caught exception in looking up space point ptr... 0x43221a20, 0x432218d0
Caught exception in looking up space point ptr... 0x432218d0, 0x43221a20
Caught exception in looking up space point ptr... 0x5e5f1dd0, 0x5d6958f0
Caught exception in looking up space point ptr... 0x5d6958f0, 0x5d695760
Caught exception in looking up space point ptr... 0xa27a2c20, 0xa27a2d50
Caught exception in looking up space point ptr... 0xa27a2d50, 0xa2900060
Caught exception in looking up space point ptr... 0x6cbcee90, 0x6ca8e580
Caught exception in looking up space point ptr... 0x6cb9e510, 0x6cbcee90
Caught exception in looking up space point ptr... 0x46c3ab90, 0x28a90cf0
Caught exception in looking up space point ptr... 0x17f015f0, 0x46c3ab90
Caught exception in looking up space point ptr... 0x9ed867a0, 0x9ed866b0
Caught exception in looking up space point ptr... 0x9ed866b0, 0x9ed867a0
Caught exception in looking up space point ptr... 0x58ea1eb0, 0x6c74e090
Caught exception in looking up space point ptr... 0x6c74e090, 0x6b9a3dc0
Caught exception in looking up space point ptr... 0x34a627b0, 0x34a62940
Caught exception in looking up space point ptr... 0x34a62940, 0x34a62ad0
Caught exception in looking up space point ptr... 0x6c607a20, 0x60da5f30
Caught exception in looking up space point ptr... 0x60da5f30, 0x6c607a20
Caught exception in looking up space point ptr... 0x333ce060, 0x274aa450
Caught exception in looking up space point ptr... 0x274aa450, 0x367a1760
Caught exception in looking up space point ptr... 0x367a1760, 0x18749ea0
Caught exception in looking up space point ptr... 0xa56521d0, 0xa56520e0
Caught exception in looking up space point ptr... 0xa56520e0, 0xa15dc1a0
Caught exception in looking up space point ptr... 0x6bbc6b40, 0x626d67f0
Caught exception in looking up space point ptr... 0x626d67f0, 0x6bbc6b40
Caught exception in looking up space point ptr... 0x6bcf6050, 0x21202d40
Caught exception in looking up space point ptr... 0x21202d40, 0x6bcf6050
Caught exception in looking up space point ptr... 0x95adc300, 0x96849c10
Caught exception in looking up space point ptr... 0x9424cfb0, 0x95adc300
Caught exception in looking up space point ptr... 0x6ddd6af0, 0x6dd7af60
Caught exception in looking up space point ptr... 0x6dd7af60, 0x6ddd6af0
Caught exception in looking up space point ptr... 0x97fba8b0, 0x97848310
Caught exception in looking up space point ptr... 0x97848310, 0x983bdd50
Caught exception in looking up space point ptr... 0x983bdd50, 0x96cec880
Caught exception in looking up space point ptr... 0x96cec880, 0x969787b0
Caught exception in looking up space point ptr... 0x34a60ff0, 0x34a60e60
Caught exception in looking up space point ptr... 0x34a60e60, 0x34a60af0
Caught exception in looking up space point ptr... 0xa0150b20, 0x9fa89710
Caught exception in looking up space point ptr... 0x9fa89710, 0xa0150b20
Caught exception in looking up space point ptr... 0xa0150b20, 0xa0150990
Caught exception in looking up space point ptr... 0xa0150990, 0xa0150b20
Caught exception in looking up space point ptr... 0xa0150b20, 0x9fd33ff0
Caught exception in looking up space point ptr... 0x9fd33ff0, 0xa0150b20
Caught exception in looking up space point ptr... 0x34a61fe0, 0x34a62300
Caught exception in looking up space point ptr... 0x34a62300, 0x34a61fe0
Caught exception in looking up space point ptr... 0x96bd56f0, 0x96dca730
Caught exception in looking up space point ptr... 0x96dca730, 0x96bd56f0
Caught exception in looking up space point ptr... 0x96bd56f0, 0x97850290
Caught exception in looking up space point ptr... 0x97850290, 0x96bd56f0
Caught exception in looking up space point ptr... 0x96bd56f0, 0x97684c40
Caught exception in looking up space point ptr... 0x97684c40, 0x96bd56f0
Caught exception in looking up space point ptr... 0x6c9d2540, 0x6c8cee90
Caught exception in looking up space point ptr... 0x6c8cee90, 0x60f482f0
Caught exception in looking up space point ptr... 0xc5823720, 0xc5f4f120
Caught exception in looking up space point ptr... 0xc5f4f120, 0xc5f4f270
Caught exception in looking up space point ptr... 0xc5f4f270, 0xc5823820
Caught exception in looking up space point ptr... 0xc5823820, 0xc5823720
Caught exception in looking up space point ptr... 0x97e600a0, 0x97e5f5e0
Caught exception in looking up space point ptr... 0x97e5f5e0, 0x97e600a0
Caught exception in looking up space point ptr... 0x97e600a0, 0x97e64fd0
Caught exception in looking up space point ptr... 0x97e64fd0, 0x97e600a0
Caught exception in looking up space point ptr... 0x97e600a0, 0x97e627d0
Caught exception in looking up space point ptr... 0x97e627d0, 0x97e600a0
Caught exception in looking up space point ptr... 0xa28cf9a0, 0xa2901aa0
Caught exception in looking up space point ptr... 0xa2901aa0, 0xa28cf9a0
Caught exception in looking up space point ptr... 0x3dacfd90, 0x3dad20a0
Caught exception in looking up space point ptr... 0x3dad20a0, 0x3dacfd90
Caught exception in looking up space point ptr... 0x97fb3810, 0x94e750e0
Caught exception in looking up space point ptr... 0x94e750e0, 0x97fbab40
Caught exception in looking up space point ptr... 0x97fbab40, 0x97fb3d30
Caught exception in looking up space point ptr... 0x97fb3d30, 0x97fb3aa0
Caught exception in looking up space point ptr... 0x97fb3aa0, 0x97fb3810
Caught exception in looking up space point ptr... 0xa2035ad0, 0xa20359a0
Caught exception in looking up space point ptr... 0xa20359a0, 0xa2035ad0
Caught exception in looking up space point ptr... 0xa185fc40, 0xa12a5420
Caught exception in looking up space point ptr... 0xa12a5420, 0xa185fc40
++++>>>> total num hits: 8192419, num free: 7976025
04-Feb-2026 02:17:19 CET Opened output file with pattern "%ifb_reco_stage0_%tc_offline.root"
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 1641.83 1641.83 1641.83 1641.83 0 1
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000242783 0.000242783 0.000242783 0.000242783 0 1
produce:tpcrawdecoder:PDVDTPCReader 3.98702 3.98702 3.98702 3.98702 0 1
produce:triggerrawdecoder:PDVDTriggerReader4 0.0167112 0.0167112 0.0167112 0.0167112 0 1
produce:timingrawdecoder:PDHDTimingRawDecoder 0.00084396 0.00084396 0.00084396 0.00084396 0 1
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.00240498 0.00240498 0.00240498 0.00240498 0 1
produce:beamevent:BeamEvent 0.00104789 0.00104789 0.00104789 0.00104789 0 1
produce:pdvddaphne:DAPHNEReaderPDVD 2.42711 2.42711 2.42711 2.42711 0 1
produce:ophit:OpHitFinder 0.0320519 0.0320519 0.0320519 0.0320519 0 1
produce:wclsdatavd:WireCellToolkit 62.5632 62.5632 62.5632 62.5632 0 1
produce:gaushit:GausHitFinder 7.91765 7.91765 7.91765 7.91765 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.00120304 0.00120304 0.00120304 0.00120304 0 1
produce:reco3d:SpacePointSolver 163.749 163.749 163.749 163.749 0 1
produce:hitpdune:DisambigFromSpacePoints 4.64818 4.64818 4.64818 4.64818 0 1
produce:cluster3d:Cluster3D 1396.35 1396.35 1396.35 1396.35 0 1
[art]:TriggerResults:TriggerResultInserter 0.000148972 0.000148972 0.000148972 0.000148972 0 1
end_path:out1:RootOutput 5.206e-05 5.206e-05 5.206e-05 5.206e-05 0 1
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 18054.3 MB
Peak resident set size usage (VmHWM): 15640.7 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 02:17:40 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 1 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 