Jobsub ID 303878.0@dunegpschedd01.fnal.gov

Jobsub ID	303878.0@dunegpschedd01.fnal.gov
Workflow ID	12717
Stage ID	1
User name	ykermaid@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
Submitted time	2026-02-04 05:36:42
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce02
Last heartbeat	2026-02-04 06:38:59
From worker node	Hostname	wn1904241.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7513 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-04 05:37:44
Input files	vd-protodune:np02vd_raw_run042421_0050_df-s02-d1_dw_0_20260203T135801.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-04 06:38:59
Saved logs	justin-logs:303878.0-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

0x7d7fe30
Caught exception in looking up space point ptr... 0xb2303070, 0xb23ff0b0
Caught exception in looking up space point ptr... 0xb23ff0b0, 0xb2303070
Caught exception in looking up space point ptr... 0x121f3720, 0x121ec810
Caught exception in looking up space point ptr... 0x121ec810, 0x121e8230
Caught exception in looking up space point ptr... 0x91ebfda0, 0x8ab56d10
Caught exception in looking up space point ptr... 0x8ab56d10, 0x8b22d900
Caught exception in looking up space point ptr... 0x2bc79c40, 0x4997d0c0
Caught exception in looking up space point ptr... 0x4997d0c0, 0x4997d250
Caught exception in looking up space point ptr... 0x58dd26e0, 0x58e1b000
Caught exception in looking up space point ptr... 0x58e1b000, 0x58dd26e0
Caught exception in looking up space point ptr... 0x5ca11550, 0x5ca113c0
Caught exception in looking up space point ptr... 0x5ca113c0, 0x5ca11550
Caught exception in looking up space point ptr... 0x2519e7f0, 0x2519e980
Caught exception in looking up space point ptr... 0x2519e980, 0x252e0550
Caught exception in looking up space point ptr... 0x252e0550, 0x252e0450
Caught exception in looking up space point ptr... 0x252e0450, 0x2519e7f0
Caught exception in looking up space point ptr... 0x8e7b41e0, 0x8f7fb090
Caught exception in looking up space point ptr... 0x8f7fb090, 0x8e7b43b0
Caught exception in looking up space point ptr... 0x85c148b0, 0xad706bb0
Caught exception in looking up space point ptr... 0xad706bb0, 0x85c148b0
Caught exception in looking up space point ptr... 0xb25ceb20, 0xb25ce780
Caught exception in looking up space point ptr... 0xb25ce780, 0xb25ceb20
Caught exception in looking up space point ptr... 0xa10732d0, 0xa1073520
Caught exception in looking up space point ptr... 0xa1073520, 0xa10732d0
Caught exception in looking up space point ptr... 0x6b692ad0, 0x6b32b5f0
Caught exception in looking up space point ptr... 0x6b32b840, 0x6b692ad0
Caught exception in looking up space point ptr... 0x69f8eb90, 0x69f8f0b0
Caught exception in looking up space point ptr... 0x69f8f0b0, 0x69f8eb90
Caught exception in looking up space point ptr... 0x4594b6f0, 0x483b9c30
Caught exception in looking up space point ptr... 0x483b9c30, 0x4b0c08f0
Caught exception in looking up space point ptr... 0x4b0c08f0, 0x483b9c30
Caught exception in looking up space point ptr... 0x483b9c30, 0x5595d210
Caught exception in looking up space point ptr... 0x90ddfcc0, 0x8ca572b0
Caught exception in looking up space point ptr... 0x8ca572b0, 0x90ddfcc0
Caught exception in looking up space point ptr... 0xc523f540, 0xc8e89ee0
Caught exception in looking up space point ptr... 0xc8e89ee0, 0xc523f540
Caught exception in looking up space point ptr... 0xc523f540, 0xbde5bae0
Caught exception in looking up space point ptr... 0xbde5bae0, 0xc523f540
Caught exception in looking up space point ptr... 0x8268f200, 0x826e66f0
Caught exception in looking up space point ptr... 0x826e66f0, 0x8268f200
Caught exception in looking up space point ptr... 0x8268f200, 0x8269e0b0
Caught exception in looking up space point ptr... 0x8269e0b0, 0x8268f200
Caught exception in looking up space point ptr... 0xbb01af60, 0xba877150
Caught exception in looking up space point ptr... 0xba877150, 0xbb01af60
Caught exception in looking up space point ptr... 0xc77aabb0, 0xc6631e90
Caught exception in looking up space point ptr... 0xc6631e90, 0xc77aabb0
Caught exception in looking up space point ptr... 0x76282cb0, 0x727e9b70
Caught exception in looking up space point ptr... 0x727e9b70, 0x76282cb0
Caught exception in looking up space point ptr... 0xa22c3ee0, 0xa0ec4ff0
Caught exception in looking up space point ptr... 0xa0ec4ff0, 0xa22c3ee0
Caught exception in looking up space point ptr... 0xa22c3ee0, 0xa38ed020
Caught exception in looking up space point ptr... 0xa38ed020, 0xa22c3ee0
Caught exception in looking up space point ptr... 0x78cdbd70, 0x789125a0
Caught exception in looking up space point ptr... 0x789125a0, 0x78cdbd70
Caught exception in looking up space point ptr... 0x831c96e0, 0x831c9ab0
Caught exception in looking up space point ptr... 0x831c9ab0, 0x831c9450
Caught exception in looking up space point ptr... 0x967d13b0, 0x9683f7a0
Caught exception in looking up space point ptr... 0x9683f7a0, 0x968eef20
Caught exception in looking up space point ptr... 0xce934180, 0xcddff8a0
Caught exception in looking up space point ptr... 0xcd68e9b0, 0xcde5fc00
Caught exception in looking up space point ptr... 0xcde5fc00, 0xcebf6e00
Caught exception in looking up space point ptr... 0xcebf6e00, 0xce934180
Caught exception in looking up space point ptr... 0x5e7180d0, 0x44187bf0
Caught exception in looking up space point ptr... 0x44187bf0, 0x5e7180d0
Caught exception in looking up space point ptr... 0x701d62a0, 0x70184cf0
Caught exception in looking up space point ptr... 0x70184cf0, 0x702aa030
Caught exception in looking up space point ptr... 0x47346e60, 0x4733b1b0
Caught exception in looking up space point ptr... 0x4733b1b0, 0x47346e60
Caught exception in looking up space point ptr... 0xa11445b0, 0xaadcc330
Caught exception in looking up space point ptr... 0xaadcc330, 0xa11445b0
Caught exception in looking up space point ptr... 0xa11445b0, 0xad2b6380
Caught exception in looking up space point ptr... 0xad2b6380, 0xa11445b0
Caught exception in looking up space point ptr... 0xa11445b0, 0xa7564380
Caught exception in looking up space point ptr... 0xa7564380, 0xa11445b0
Caught exception in looking up space point ptr... 0x35972260, 0x44f8db20
Caught exception in looking up space point ptr... 0x44f8db20, 0x35972260
Caught exception in looking up space point ptr... 0x95ad1a80, 0xa6ca1580
Caught exception in looking up space point ptr... 0xa6ca1580, 0x8264b1b0
Caught exception in looking up space point ptr... 0x72f0dbf0, 0x6e4cd860
Caught exception in looking up space point ptr... 0x6e4cd860, 0x70ba35c0
++++>>>> total num hits: 7762182, num free: 7557525

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         304.964       1627.49       2950.02       1627.49       1322.53         2     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000128071   0.00018628    0.00024449    0.00018628    5.82095e-05       2     
produce:tpcrawdecoder:PDVDTPCReader                168.249       178.703       189.156       178.703       10.4532         2     
produce:triggerrawdecoder:PDVDTriggerReader4      0.893022      0.914535      0.936047      0.914535      0.0215127        2     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0253224     0.053175      0.0810277     0.053175      0.0278527        2     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.108214      0.112593      0.116972      0.112593     0.00437908        2     
produce:beamevent:BeamEvent                      0.000166654   0.000742413   0.00131817    0.000742413   0.000575759       2     
produce:pdvddaphne:DAPHNEReaderPDVD                22.4386       22.8103       23.1821       22.8103       0.37177         2     
produce:ophit:OpHitFinder                         0.0413868     0.0449277     0.0484686     0.0449277    0.00354091        2     
produce:wclsdatavd:WireCellToolkit                 79.7695       83.9154       88.0614       83.9154       4.14595         2     
produce:gaushit:GausHitFinder                     0.756232       5.55353       10.3508       5.55353       4.79729         2     
produce:nhitsfilter:NumberOfHitsFilter           0.000599729   0.00113721    0.00167468    0.00113721    0.000537477       2     
produce:reco3d:SpacePointSolver                    5.5239        115.432       225.341       115.432       109.909         2     
produce:hitpdune:DisambigFromSpacePoints          0.0713934      4.10849       8.14559       4.10849       4.0371          2     
produce:cluster3d:Cluster3D                        1.09875       1213.31       2425.53       1213.31       1212.22         2     
[art]:TriggerResults:TriggerResultInserter       7.0322e-05     8.255e-05    9.4778e-05     8.255e-05    1.2228e-05        2     
end_path:out1:RootOutput                         1.3265e-05    2.20865e-05   3.0908e-05    2.20865e-05   8.8215e-06        2     
end_path:out1:RootOutput(write)                    5.00133       5.00133       5.00133       5.00133          0            1     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 16935.4 MB
  Peak resident set size usage (VmHWM): 14945.2 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 04-Feb-2026 06:33:53 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 2 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1