Jobsub ID 297917.75@dunegpschedd02.fnal.gov

Jobsub ID	297917.75@dunegpschedd02.fnal.gov
Workflow ID	12717
Stage ID	1
User name	ykermaid@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
Submitted time	2026-02-03 22:48:25
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce513
Last heartbeat	2026-02-03 23:58:35
From worker node	Hostname	b9p35p0008.cern.ch
	cpuinfo	AMD EPYC 9655P 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-03 23:16:58
Input files	vd-protodune:np02vd_raw_run042421_0050_df-s02-d0_dw_0_20260203T135802.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-03 23:58:35
Saved logs	justin-logs:297917.75-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

tr... 0x127be4120, 0x12659aa80
Caught exception in looking up space point ptr... 0x12659aa80, 0x128a2e030
Caught exception in looking up space point ptr... 0x1016c6700, 0x1016c67f0
Caught exception in looking up space point ptr... 0x1016c67f0, 0x1010102d0
Caught exception in looking up space point ptr... 0x101476360, 0x1014764f0
Caught exception in looking up space point ptr... 0x1014764f0, 0x1016c6700
Caught exception in looking up space point ptr... 0x77e04400, 0x77e040e0
Caught exception in looking up space point ptr... 0x77e040e0, 0x77e04270
Caught exception in looking up space point ptr... 0x10ce25b70, 0x11ddf3370
Caught exception in looking up space point ptr... 0x11ddf3370, 0x10ce25b70
Caught exception in looking up space point ptr... 0x12329a370, 0x1328e5250
Caught exception in looking up space point ptr... 0x1328e5250, 0x12329a370
Caught exception in looking up space point ptr... 0x8903b230, 0x889949f0
Caught exception in looking up space point ptr... 0x889b2e30, 0x8903b230
Caught exception in looking up space point ptr... 0x5532e980, 0x4435e5f0
Caught exception in looking up space point ptr... 0x5540da40, 0x5532e980
Caught exception in looking up space point ptr... 0x33726d40, 0x33727bf0
Caught exception in looking up space point ptr... 0x33727bf0, 0x33726d40
Caught exception in looking up space point ptr... 0x33726d40, 0x337284f0
Caught exception in looking up space point ptr... 0x337284f0, 0x33726d40
Caught exception in looking up space point ptr... 0x33726d40, 0x33726590
Caught exception in looking up space point ptr... 0x33726590, 0x33726d40
Caught exception in looking up space point ptr... 0x268b2450, 0x268acd90
Caught exception in looking up space point ptr... 0x268acd90, 0x268b2450
Caught exception in looking up space point ptr... 0x9f605aa0, 0x9f605c30
Caught exception in looking up space point ptr... 0x3d3549e0, 0x9f605aa0
Caught exception in looking up space point ptr... 0x71fb0650, 0x759869e0
Caught exception in looking up space point ptr... 0x768a6350, 0x71fb0650
Caught exception in looking up space point ptr... 0x4e88fb80, 0x4e85e0e0
Caught exception in looking up space point ptr... 0x4e85e0e0, 0x4e844140
Caught exception in looking up space point ptr... 0x62cc5360, 0x3bf56430
Caught exception in looking up space point ptr... 0x3bf56430, 0x62cc5360
Caught exception in looking up space point ptr... 0x451a2b10, 0x4cec6830
Caught exception in looking up space point ptr... 0x4cec6830, 0x451a2b10
Caught exception in looking up space point ptr... 0x1017b5860, 0x1017b53b0
Caught exception in looking up space point ptr... 0x1017b53b0, 0x1017b5860
Caught exception in looking up space point ptr... 0x28603ba0, 0x286036f0
Caught exception in looking up space point ptr... 0x286036f0, 0x28603ba0
Caught exception in looking up space point ptr... 0xdba7fc30, 0xdab34230
Caught exception in looking up space point ptr... 0xdab34230, 0xdba7fc30
Caught exception in looking up space point ptr... 0x1012c4f50, 0x1012c4e60
Caught exception in looking up space point ptr... 0x1012c4e60, 0x1012c4f50
Caught exception in looking up space point ptr... 0x7c1d0660, 0x7d680da0
Caught exception in looking up space point ptr... 0x7d680da0, 0x7c1d07f0
Caught exception in looking up space point ptr... 0xad40e480, 0xad40e610
Caught exception in looking up space point ptr... 0xad40e610, 0xad40e2f0
Caught exception in looking up space point ptr... 0x3a3a9300, 0x51976ad0
Caught exception in looking up space point ptr... 0x51976ad0, 0x4eb1a210
Caught exception in looking up space point ptr... 0x1095229e0, 0x119a80050
Caught exception in looking up space point ptr... 0x119a80050, 0x1095229e0
Caught exception in looking up space point ptr... 0x9ad670f0, 0x9b46e020
Caught exception in looking up space point ptr... 0x9b46e020, 0x9ad670f0
Caught exception in looking up space point ptr... 0x10255aa90, 0x10255a9f0
Caught exception in looking up space point ptr... 0x10255a9f0, 0x10255aa90
Caught exception in looking up space point ptr... 0x1020dc050, 0x1020dc3e0
Caught exception in looking up space point ptr... 0x1020dc3e0, 0x1020dc2b0
Caught exception in looking up space point ptr... 0xafe99cf0, 0xafe9a010
Caught exception in looking up space point ptr... 0xafe9a010, 0xafe9a1a0
Caught exception in looking up space point ptr... 0x100438270, 0xffc60970
Caught exception in looking up space point ptr... 0xffc60970, 0x100438270
Caught exception in looking up space point ptr... 0xd4e57890, 0xd49275a0
Caught exception in looking up space point ptr... 0xd49275a0, 0xd4e57700
Caught exception in looking up space point ptr... 0xe10ec0d0, 0xe7cb6100
Caught exception in looking up space point ptr... 0xe7cb6100, 0xe10ec0d0
Caught exception in looking up space point ptr... 0xa259cad0, 0x94891c40
Caught exception in looking up space point ptr... 0x94891c40, 0x8b716720
Caught exception in looking up space point ptr... 0x11def2f50, 0x11351f020
Caught exception in looking up space point ptr... 0x11351f020, 0x11bfa4630
Caught exception in looking up space point ptr... 0x101707170, 0x1017b14f0
Caught exception in looking up space point ptr... 0x1017b14f0, 0x101707170
Caught exception in looking up space point ptr... 0xe19731c0, 0xeb966200
Caught exception in looking up space point ptr... 0xeb966200, 0xe19731c0
Caught exception in looking up space point ptr... 0xe19731c0, 0xe0624470
Caught exception in looking up space point ptr... 0xe0624470, 0xe19731c0
Caught exception in looking up space point ptr... 0xfedc54b0, 0xfe797040
Caught exception in looking up space point ptr... 0xfed51120, 0xfedc54b0
Caught exception in looking up space point ptr... 0x13405f9e0, 0x13405f530
Caught exception in looking up space point ptr... 0x13405f530, 0x13405f9e0
Caught exception in looking up space point ptr... 0x9849840, 0x39ecbb10
Caught exception in looking up space point ptr... 0x39ecbb10, 0x98480c0
++++>>>> total num hits: 12659829, num free: 12354771

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         65.7517       1197.52       2329.29       1197.52       1131.77         2     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.034e-05    0.000118136   0.000175931   0.000118136   5.77955e-05       2     
produce:tpcrawdecoder:PDVDTPCReader                3.63014       4.24245       4.85477       4.24245      0.612316         2     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00988291     0.0260235     0.042164      0.0260235     0.0161406        2     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000481663   0.000581618   0.000681574   0.000581618   9.99555e-05       2     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00165728    0.00182055    0.00198383    0.00182055    0.000163276       2     
produce:beamevent:BeamEvent                       7.952e-05    0.000437902   0.000796284   0.000437902   0.000358382       2     
produce:pdvddaphne:DAPHNEReaderPDVD                2.09963       2.20213       2.30462       2.20213      0.102492         2     
produce:ophit:OpHitFinder                         0.0243308     0.0274315     0.0305322     0.0274315    0.00310073        2     
produce:wclsdatavd:WireCellToolkit                 48.1229       52.2004       56.2779       52.2004       4.07749         2     
produce:gaushit:GausHitFinder                     0.581768       5.13052       9.67927       5.13052       4.54875         2     
produce:nhitsfilter:NumberOfHitsFilter           0.000327092   0.000469142   0.000611193   0.000469142   0.00014205        2     
produce:reco3d:SpacePointSolver                    6.33061       124.187       242.043       124.187       117.856         2     
produce:hitpdune:DisambigFromSpacePoints          0.0850355      2.84015       5.59526       2.84015       2.75511         2     
produce:cluster3d:Cluster3D                       0.941464       1004.8        2008.65       1004.8        1003.86         2     
[art]:TriggerResults:TriggerResultInserter        2.785e-05     5.083e-05     7.381e-05     5.083e-05     2.298e-05        2     
end_path:out1:RootOutput                          4.37e-06     1.1055e-05     1.774e-05    1.1055e-05     6.685e-06        2     
end_path:out1:RootOutput(write)                    3.69566       3.69566       3.69566       3.69566          0            1     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 22518.5 MB
  Peak resident set size usage (VmHWM): 19793.5 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 04-Feb-2026 00:57:59 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 2 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1