Jobsub ID 297916.88@dunegpschedd02.fnal.gov

Jobsub ID	297916.88@dunegpschedd02.fnal.gov
Workflow ID	12718
Stage ID	1
User name	ykermaid@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
Submitted time	2026-02-03 22:46:26
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2026-02-04 00:56:33
From worker node	Hostname	heplnc168.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-03 22:48:32
Input files	vd-protodune:np02vd_raw_run042422_0078_df-s03-d0_dw_0_20260203T155656.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-04 00:56:33
Saved logs	justin-logs:297916.88-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

c10
Caught exception in looking up space point ptr... 0x1061c04b0, 0x154ed8980
Caught exception in looking up space point ptr... 0xd41451f0, 0xd4145710
Caught exception in looking up space point ptr... 0xd4145480, 0xd41451f0
Caught exception in looking up space point ptr... 0x106a2c680, 0x106a2c360
Caught exception in looking up space point ptr... 0x106a2c360, 0x106a2c680
Caught exception in looking up space point ptr... 0x87211410, 0x10e40c090
Caught exception in looking up space point ptr... 0x11d256450, 0x87211410
Caught exception in looking up space point ptr... 0x6db9ac40, 0x6f493790
Caught exception in looking up space point ptr... 0x162647ae0, 0x6db9ac40
Caught exception in looking up space point ptr... 0xaa6f1a40, 0x86d1db70
Caught exception in looking up space point ptr... 0x86d1db70, 0xaa6f1a40
Caught exception in looking up space point ptr... 0x1115acad0, 0x116021200
Caught exception in looking up space point ptr... 0x116021200, 0x1115acad0
Caught exception in looking up space point ptr... 0x14c955430, 0x14c916ec0
Caught exception in looking up space point ptr... 0x14c916ec0, 0x6edd9090
Caught exception in looking up space point ptr... 0x10f6e8b20, 0x1643630f0
Caught exception in looking up space point ptr... 0x1643630f0, 0x10f6e8b20
Caught exception in looking up space point ptr... 0x47025fb0, 0x47025e60
Caught exception in looking up space point ptr... 0x47025e60, 0x47026250
Caught exception in looking up space point ptr... 0x47026250, 0x470263a0
Caught exception in looking up space point ptr... 0x14c855160, 0x14cfd7610
Caught exception in looking up space point ptr... 0x14cfd7610, 0x161f09540
Caught exception in looking up space point ptr... 0x7f420de0, 0x7e890e70
Caught exception in looking up space point ptr... 0xe9a8cbe0, 0x7f420de0
Caught exception in looking up space point ptr... 0x4e68a210, 0x46e534a0
Caught exception in looking up space point ptr... 0xbb5f0620, 0x4e68a210
Caught exception in looking up space point ptr... 0xb92381c0, 0x111caba70
Caught exception in looking up space point ptr... 0x52365c30, 0xb92381c0
Caught exception in looking up space point ptr... 0x147ac9280, 0x174c66060
Caught exception in looking up space point ptr... 0x174c66060, 0x147ac9280
Caught exception in looking up space point ptr... 0xa8f72a30, 0xa9179a80
Caught exception in looking up space point ptr... 0xa9179a80, 0xa8f72a30
Caught exception in looking up space point ptr... 0xd65c33f0, 0xd65c3680
Caught exception in looking up space point ptr... 0xd65c3680, 0x544867c0
Caught exception in looking up space point ptr... 0x1729c12d0, 0x14e7ed1c0
Caught exception in looking up space point ptr... 0x14e7ed1c0, 0x1729c12d0
Caught exception in looking up space point ptr... 0xe9a89620, 0x107e44850
Caught exception in looking up space point ptr... 0x107e44850, 0xe9a89620
Caught exception in looking up space point ptr... 0x5957e8c0, 0xe060a850
Caught exception in looking up space point ptr... 0xe060a850, 0x5957e8c0
Caught exception in looking up space point ptr... 0x6dc49570, 0x71291050
Caught exception in looking up space point ptr... 0x71291050, 0x6dc49570
Caught exception in looking up space point ptr... 0x107620440, 0xe9a3bf80
Caught exception in looking up space point ptr... 0xe9a3bf80, 0xe9a8f450
Caught exception in looking up space point ptr... 0x6f72c770, 0x53c9a0f0
Caught exception in looking up space point ptr... 0x53c9a0f0, 0x6f72c770
Caught exception in looking up space point ptr... 0x17a699c20, 0x1501acca0
Caught exception in looking up space point ptr... 0x1501acca0, 0x17a699c20
Caught exception in looking up space point ptr... 0x17a699c20, 0x1647e2f10
Caught exception in looking up space point ptr... 0x1647e2f10, 0x17a699c20
Caught exception in looking up space point ptr... 0x17a699c20, 0x8527bfb0
Caught exception in looking up space point ptr... 0x8527bfb0, 0x17a699c20
Caught exception in looking up space point ptr... 0x53d215a0, 0x10a43e070
Caught exception in looking up space point ptr... 0x24d72cf0, 0x53d215a0
Caught exception in looking up space point ptr... 0x5bb6ff60, 0x3adf96e0
Caught exception in looking up space point ptr... 0x3adf96e0, 0x5bb919d0
Caught exception in looking up space point ptr... 0x5bb919d0, 0x3adf9880
Caught exception in looking up space point ptr... 0x107412cd0, 0x10740b0d0
Caught exception in looking up space point ptr... 0x107a88a00, 0x107412cd0
Caught exception in looking up space point ptr... 0x108ca02a0, 0x108ca0430
Caught exception in looking up space point ptr... 0x108ca0430, 0x108ca0110
Caught exception in looking up space point ptr... 0x1276348e0, 0x127634430
Caught exception in looking up space point ptr... 0x127634430, 0x1276348e0
Caught exception in looking up space point ptr... 0x3ae077f0, 0x3ae07660
Caught exception in looking up space point ptr... 0x3ae07660, 0x3ae071b0
Caught exception in looking up space point ptr... 0x4d21b880, 0x552ab5d0
Caught exception in looking up space point ptr... 0x552ab5d0, 0x453c1a80
Caught exception in looking up space point ptr... 0x107503670, 0x107503800
Caught exception in looking up space point ptr... 0x107503800, 0x107503990
Caught exception in looking up space point ptr... 0x109828810, 0x5b355570
Caught exception in looking up space point ptr... 0x5b355570, 0x109828810
Caught exception in looking up space point ptr... 0x1277c7650, 0x127638f30
Caught exception in looking up space point ptr... 0x127638f30, 0x1277c7650
Caught exception in looking up space point ptr... 0x18112e560, 0x13a85e750
Caught exception in looking up space point ptr... 0x13a85e750, 0x18112e560
Caught exception in looking up space point ptr... 0xea3f0370, 0xea3efa90
Caught exception in looking up space point ptr... 0xea3efa90, 0xea3f0370
Caught exception in looking up space point ptr... 0x15086a170, 0x150869cd0
Caught exception in looking up space point ptr... 0x150869cd0, 0x15086a170
++++>>>> total num hits: 8128493, num free: 7916479

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         126.551       412.303       2525.16       171.463       613.422        18     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.4649e-05    0.000142537   0.000349516   0.000113388   6.54331e-05      18     
produce:tpcrawdecoder:PDVDTPCReader                12.3834       12.911        13.4477       12.9245      0.305026        18     
produce:triggerrawdecoder:PDVDTriggerReader4      0.540687      0.601459       0.69951      0.599586      0.0352067       18     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0175371     0.0191128     0.0280414     0.017914     0.00312336       18     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0871666     0.0938402     0.106353      0.0885456    0.00809735       18     
produce:beamevent:BeamEvent                      0.000115327   0.000229885   0.00145452    0.000168341   0.000298433      18     
produce:pdvddaphne:DAPHNEReaderPDVD                3.38504       4.01945       6.42123       3.86432      0.629889        18     
produce:ophit:OpHitFinder                         0.0406244     0.0472032     0.052952      0.0473387    0.00358156       18     
produce:wclsdatavd:WireCellToolkit                 86.7363       93.5854       108.461        92.34        4.99475        18     
produce:gaushit:GausHitFinder                     0.897772       2.62045       12.3687       1.85935       2.62705        18     
produce:nhitsfilter:NumberOfHitsFilter           0.000246423   0.000526897   0.00169222    0.000413251   0.000365969      18     
produce:reco3d:SpacePointSolver                    6.33073       33.8848       209.968       12.6393       54.6343        18     
produce:hitpdune:DisambigFromSpacePoints          0.107068       1.4267        8.77221      0.220748       2.72995        18     
produce:cluster3d:Cluster3D                        1.28499       255.038       2167.36       46.0365       547.859        18     
[art]:TriggerResults:TriggerResultInserter       2.6039e-05    4.52076e-05   0.00013307    3.79265e-05   2.53931e-05      18     
end_path:out1:RootOutput                          7.173e-06    1.19779e-05   3.1519e-05    1.0059e-05    6.22096e-06      18     
end_path:out1:RootOutput(write)                    4.17129       8.48986       33.3315       6.12203       7.46273        17     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19622.7 MB
  Peak resident set size usage (VmHWM): 17243.6 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 04-Feb-2026 00:53:58 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 17 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 18 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1