Jobsub ID 297935.0@dunegpschedd02.fnal.gov

Jobsub ID	297935.0@dunegpschedd02.fnal.gov
Workflow ID	12718
Stage ID	1
User name	ykermaid@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
Submitted time	2026-02-03 23:30:25
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce01
Last heartbeat	2026-02-04 01:26:30
From worker node	Hostname	wn5918241.tier2.hep.manchester.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-04 00:13:43
Input files	vd-protodune:np02vd_raw_run042422_0116_df-s04-d1_dw_0_20260203T161728.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-04 01:26:30
Saved logs	justin-logs:297935.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

xbf4575c0
Caught exception in looking up space point ptr... 0xea3a39c0, 0xea3a3c50
Caught exception in looking up space point ptr... 0xea3a3c50, 0xea3a39c0
Caught exception in looking up space point ptr... 0x55602580, 0x543eab80
Caught exception in looking up space point ptr... 0x543eab80, 0x55602580
Caught exception in looking up space point ptr... 0x7c1a6ce0, 0x805a7830
Caught exception in looking up space point ptr... 0x805a7830, 0x7c1a6ce0
Caught exception in looking up space point ptr... 0x60878a30, 0x608788a0
Caught exception in looking up space point ptr... 0x608788a0, 0x5a6b6190
Caught exception in looking up space point ptr... 0xc3257570, 0xc33124d0
Caught exception in looking up space point ptr... 0xc33124d0, 0xc3257570
Caught exception in looking up space point ptr... 0x6cc36980, 0x6e29b690
Caught exception in looking up space point ptr... 0x6e29b690, 0x6cc36980
Caught exception in looking up space point ptr... 0xa4949cf0, 0xa5e4caa0
Caught exception in looking up space point ptr... 0xa5e4caa0, 0xa323b100
Caught exception in looking up space point ptr... 0xec152910, 0xd944d110
Caught exception in looking up space point ptr... 0xd944d110, 0xdde0b390
Caught exception in looking up space point ptr... 0x8a5edf30, 0x89c02bd0
Caught exception in looking up space point ptr... 0x89c02bd0, 0x8a5edf30
Caught exception in looking up space point ptr... 0x625a2050, 0x61d52ed0
Caught exception in looking up space point ptr... 0x61d52ed0, 0x625a2050
Caught exception in looking up space point ptr... 0x294a1a50, 0x261bf2a0
Caught exception in looking up space point ptr... 0x261bf2a0, 0x294a1a50
Caught exception in looking up space point ptr... 0x294a1a50, 0x2949e220
Caught exception in looking up space point ptr... 0x2949e220, 0x294a1a50
Caught exception in looking up space point ptr... 0x8a71ee00, 0x7eb7a1c0
Caught exception in looking up space point ptr... 0x7eb7a1c0, 0x8a71ee00
Caught exception in looking up space point ptr... 0xc6853d80, 0xc6853f60
Caught exception in looking up space point ptr... 0xc6853f60, 0xc6853d80
Caught exception in looking up space point ptr... 0xd6094ee0, 0xd970e110
Caught exception in looking up space point ptr... 0xd970e110, 0xd6094ee0
Caught exception in looking up space point ptr... 0x6bea5030, 0x35c4f970
Caught exception in looking up space point ptr... 0x35c4f970, 0x6bea5030
Caught exception in looking up space point ptr... 0x5a868330, 0x3b177d80
Caught exception in looking up space point ptr... 0x3b177d80, 0x5a868330
Caught exception in looking up space point ptr... 0x5a868330, 0x29631b50
Caught exception in looking up space point ptr... 0x29631b50, 0x5a868330
Caught exception in looking up space point ptr... 0x533f4570, 0x533f4700
Caught exception in looking up space point ptr... 0x533f4700, 0x537d3f80
Caught exception in looking up space point ptr... 0xb0d07370, 0xb39c8090
Caught exception in looking up space point ptr... 0xb39c8090, 0xaefde230
Caught exception in looking up space point ptr... 0x49accf30, 0x39fb0310
Caught exception in looking up space point ptr... 0x39fb0310, 0x3db5aa60
Caught exception in looking up space point ptr... 0xc7fafa60, 0xc7ee8680
Caught exception in looking up space point ptr... 0xc7ee8680, 0xc7fafa60
Caught exception in looking up space point ptr... 0xc7fafa60, 0xc7ea3120
Caught exception in looking up space point ptr... 0xc7ea3120, 0xc7fafa60
Caught exception in looking up space point ptr... 0x387dad70, 0x3882bfc0
Caught exception in looking up space point ptr... 0x3882bfc0, 0x387dad70
Caught exception in looking up space point ptr... 0x62f283a0, 0x62f28250
Caught exception in looking up space point ptr... 0x62f28250, 0x62f283a0
Caught exception in looking up space point ptr... 0x6e3aae50, 0x85e67f50
Caught exception in looking up space point ptr... 0x90eac020, 0x6e3aae50
Caught exception in looking up space point ptr... 0xcb779eb0, 0xcb779dc0
Caught exception in looking up space point ptr... 0xcb779dc0, 0xcb779cd0
Caught exception in looking up space point ptr... 0x4bdc60e0, 0x53e04c80
Caught exception in looking up space point ptr... 0x53e04c80, 0x4bdc60e0
Caught exception in looking up space point ptr... 0x840d5300, 0x826c5100
Caught exception in looking up space point ptr... 0x826c5100, 0x840d5300
Caught exception in looking up space point ptr... 0x639cf840, 0x639cda40
Caught exception in looking up space point ptr... 0x639cda40, 0x639cf840
Caught exception in looking up space point ptr... 0x54433750, 0x764555d0
Caught exception in looking up space point ptr... 0x764555d0, 0x5be4bab0
Caught exception in looking up space point ptr... 0xa5184600, 0xa5184920
Caught exception in looking up space point ptr... 0xa5184920, 0xa5184ab0
Caught exception in looking up space point ptr... 0xd83511b0, 0xd7f48600
Caught exception in looking up space point ptr... 0xd7f48600, 0xd83511b0
Caught exception in looking up space point ptr... 0x5afe8b60, 0x25453490
Caught exception in looking up space point ptr... 0x25453490, 0x5afe8b60
Caught exception in looking up space point ptr... 0x5afe8b60, 0x1820f0e0
Caught exception in looking up space point ptr... 0x1820f0e0, 0x5afe8b60
Caught exception in looking up space point ptr... 0xc92439c0, 0xc74f5870
Caught exception in looking up space point ptr... 0xc74f5870, 0xc92439c0
Caught exception in looking up space point ptr... 0x9bcbe280, 0xa0f74310
Caught exception in looking up space point ptr... 0xa0f74310, 0x9bcbe280
Caught exception in looking up space point ptr... 0x6070e630, 0x6956f400
Caught exception in looking up space point ptr... 0x6956f400, 0x6070e630
Caught exception in looking up space point ptr... 0x6070e630, 0x6071f690
Caught exception in looking up space point ptr... 0x6071f690, 0x6070e630
Caught exception in looking up space point ptr... 0x4dd10170, 0x4dd190e0
Caught exception in looking up space point ptr... 0x4dd190e0, 0x4dd19270
++++>>>> total num hits: 8947053, num free: 8697868

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         367.107       2032.95       3698.8        2032.95       1665.85         2     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000160841   0.000224628   0.000288415   0.000224628   6.3787e-05        2     
produce:tpcrawdecoder:PDVDTPCReader                154.683       159.888       165.093       159.888       5.20503         2     
produce:triggerrawdecoder:PDVDTriggerReader4       1.44083       1.48006       1.51929       1.48006      0.0392308        2     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0223883     0.0257552     0.0291221     0.0257552    0.00336687        2     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.109651      0.109694      0.109737      0.109694     4.31755e-05       2     
produce:beamevent:BeamEvent                      0.000205884   0.000924578   0.00164327    0.000924578   0.000718694       2     
produce:pdvddaphne:DAPHNEReaderPDVD                25.4013       27.3481       29.2949       27.3481       1.94681         2     
produce:ophit:OpHitFinder                         0.0491654     0.0687253     0.0882851     0.0687253     0.0195598        2     
produce:wclsdatavd:WireCellToolkit                 139.898       147.597       155.295       147.597       7.69847         2     
produce:gaushit:GausHitFinder                      16.1128       16.8454       17.578        16.8454      0.732582         2     
produce:nhitsfilter:NumberOfHitsFilter           0.00302026    0.00320733     0.0033944    0.00320733    0.000187072       2     
[art]:TriggerResults:TriggerResultInserter       0.000129866   0.000137266   0.000144666   0.000137266     7.4e-06         2     
end_path:out1:RootOutput                         2.6863e-05    3.28015e-05    3.874e-05    3.28015e-05   5.9385e-06        2     
end_path:out1:RootOutput(write)                    9.92933       9.92933       9.92933       9.92933          0            1     
produce:reco3d:SpacePointSolver                    353.062       353.062       353.062       353.062          0            1     
produce:hitpdune:DisambigFromSpacePoints           20.3833       20.3833       20.3833       20.3833          0            1     
produce:cluster3d:Cluster3D                        2975.69       2975.69       2975.69       2975.69          0            1     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 18234.9 MB
  Peak resident set size usage (VmHWM): 15734.1 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 04-Feb-2026 01:25:08 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 2 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1