Jobsub ID 297997.0@dunegpschedd02.fnal.gov

Jobsub ID	297997.0@dunegpschedd02.fnal.gov
Workflow ID	12717
Stage ID	1
User name	ykermaid@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
Submitted time	2026-02-04 01:24:30
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce01
Last heartbeat	2026-02-04 02:15:49
From worker node	Hostname	wn1909240.tier2.hep.manchester.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-04 01:25:50
Input files	vd-protodune:np02vd_raw_run042421_0050_df-s02-d0_dw_0_20260203T135802.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-04 02:15:49
Saved logs	justin-logs:297997.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

, 0x5dafb460
Caught exception in looking up space point ptr... 0x6b583020, 0x6b4a5a00
Caught exception in looking up space point ptr... 0x6b4a5a00, 0x6b7c37e0
Caught exception in looking up space point ptr... 0x6b7c37e0, 0x6b582e90
Caught exception in looking up space point ptr... 0x6b582e90, 0x6b4a5b40
Caught exception in looking up space point ptr... 0x6b4a5b40, 0x6b4a5a00
Caught exception in looking up space point ptr... 0x6b4a5a00, 0x6b4a5c80
Caught exception in looking up space point ptr... 0x6b58c170, 0x6b4a6130
Caught exception in looking up space point ptr... 0x6b4a6130, 0x6b4a5fa0
Caught exception in looking up space point ptr... 0x6b4a5fa0, 0x6b58bfe0
Caught exception in looking up space point ptr... 0x6b58bfe0, 0x6b4a62c0
Caught exception in looking up space point ptr... 0x6b4a62c0, 0x6b4a6130
Caught exception in looking up space point ptr... 0x6b4a6130, 0x6b4a6450
Caught exception in looking up space point ptr... 0xa01f21c0, 0xa01f1ea0
Caught exception in looking up space point ptr... 0xa01f1ea0, 0xa01f21c0
Caught exception in looking up space point ptr... 0xcda5a5b0, 0xac2ec8d0
Caught exception in looking up space point ptr... 0xac2ec8d0, 0xcda5a5b0
Caught exception in looking up space point ptr... 0x5aae4c70, 0x5aae49e0
Caught exception in looking up space point ptr... 0x5aae49e0, 0x5aae4c70
Caught exception in looking up space point ptr... 0xa65c0470, 0xa65c05a0
Caught exception in looking up space point ptr... 0xa65c05a0, 0xa65c0470
Caught exception in looking up space point ptr... 0x8d833b50, 0x8d559600
Caught exception in looking up space point ptr... 0x8d559600, 0x8d87c4f0
Caught exception in looking up space point ptr... 0x6ba04090, 0x6aa23080
Caught exception in looking up space point ptr... 0x6aa23080, 0x6ba04370
Caught exception in looking up space point ptr... 0x136156e0, 0x8c1c6d0
Caught exception in looking up space point ptr... 0x8c1c6d0, 0x8c1c9f0
Caught exception in looking up space point ptr... 0x41a5d590, 0x41a5d400
Caught exception in looking up space point ptr... 0x41a5d400, 0x41a5d590
Caught exception in looking up space point ptr... 0x5dc6b910, 0x5d26e460
Caught exception in looking up space point ptr... 0x5d26e460, 0x5dc6b910
Caught exception in looking up space point ptr... 0x27792610, 0x4df8fc30
Caught exception in looking up space point ptr... 0x4df8fc30, 0x4da0f4e0
Caught exception in looking up space point ptr... 0x6d401310, 0x6d6c03d0
Caught exception in looking up space point ptr... 0x6d6c03d0, 0x6d6c0650
Caught exception in looking up space point ptr... 0x38cc6350, 0x5d3ed640
Caught exception in looking up space point ptr... 0x5d3ed640, 0x38cc6350
Caught exception in looking up space point ptr... 0xd69bebc0, 0xd69be720
Caught exception in looking up space point ptr... 0xd69be720, 0xd3d656a0
Caught exception in looking up space point ptr... 0x24515b50, 0x24515a50
Caught exception in looking up space point ptr... 0x24515a50, 0x2328fec0
Caught exception in looking up space point ptr... 0x613c71c0, 0x613c6ea0
Caught exception in looking up space point ptr... 0x613c6ea0, 0x613c71c0
Caught exception in looking up space point ptr... 0x613c71c0, 0x613c6d10
Caught exception in looking up space point ptr... 0x613c6d10, 0x613c71c0
Caught exception in looking up space point ptr... 0xb7706570, 0xb792f000
Caught exception in looking up space point ptr... 0xb792f000, 0xb7706570
Caught exception in looking up space point ptr... 0x9f042950, 0xa0382ff0
Caught exception in looking up space point ptr... 0xa0382ff0, 0x9f042950
Caught exception in looking up space point ptr... 0x62e15f70, 0x62f34b20
Caught exception in looking up space point ptr... 0x62f34b20, 0x62e16100
Caught exception in looking up space point ptr... 0x60ae8790, 0x4d957990
Caught exception in looking up space point ptr... 0x4d957990, 0x60ae8790
Caught exception in looking up space point ptr... 0x60ae8790, 0x40ca05c0
Caught exception in looking up space point ptr... 0x40ca05c0, 0x60ae8790
Caught exception in looking up space point ptr... 0xaa6fca90, 0xaa6fc8b0
Caught exception in looking up space point ptr... 0xaa6fc8b0, 0xaa6fca90
Caught exception in looking up space point ptr... 0x38977910, 0x63c62810
Caught exception in looking up space point ptr... 0x63c62810, 0x63c62b30
Caught exception in looking up space point ptr... 0x6d6cdb40, 0x6d400370
Caught exception in looking up space point ptr... 0x6d400370, 0x6d4001e0
Caught exception in looking up space point ptr... 0x43f9fba0, 0x43bed140
Caught exception in looking up space point ptr... 0x43bed140, 0x43f9fba0
Caught exception in looking up space point ptr... 0x6a47ab80, 0x6a0e85b0
Caught exception in looking up space point ptr... 0x6a0e85b0, 0x6a47ab80
Caught exception in looking up space point ptr... 0x55587050, 0x55586be0
Caught exception in looking up space point ptr... 0x55586be0, 0x55586d30
Caught exception in looking up space point ptr... 0x290327d0, 0x51b3ea20
Caught exception in looking up space point ptr... 0x51b3ea20, 0x3d51f850
Caught exception in looking up space point ptr... 0x613dc520, 0x613dc390
Caught exception in looking up space point ptr... 0x613dc390, 0x5eca7ea0
Caught exception in looking up space point ptr... 0x5dfd6870, 0x5dbb3c90
Caught exception in looking up space point ptr... 0x5dbb3c90, 0x5dfd6870
Caught exception in looking up space point ptr... 0xb0059030, 0xb51659f0
Caught exception in looking up space point ptr... 0xb51659f0, 0xb0059030
Caught exception in looking up space point ptr... 0x9fa31cb0, 0x9fa31b20
Caught exception in looking up space point ptr... 0x9fa31b20, 0x9fa31e40
Caught exception in looking up space point ptr... 0x5c9b9b90, 0x3df31230
Caught exception in looking up space point ptr... 0x3df31230, 0x5c9b8410
Caught exception in looking up space point ptr... 0x3a38f210, 0x39d6eed0
Caught exception in looking up space point ptr... 0x39d6eed0, 0x3a38f210
++++>>>> total num hits: 8170777, num free: 7972049

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         531.752       1368.4        2205.06       1368.4        836.652         2     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.2027e-05    0.00012919    0.000186353   0.00012919    5.7163e-05        2     
produce:tpcrawdecoder:PDVDTPCReader                350.352       357.798       365.244       357.798       7.44608         2     
produce:triggerrawdecoder:PDVDTriggerReader4       9.22116       10.1442       11.0672       10.1442      0.923037         2     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.141317      0.293662      0.446008      0.293662      0.152345         2     
produce:ctbrawdecoder:PDHDCTBRawDecoder            2.01085       2.5917        3.17254       2.5917       0.580845         2     
produce:beamevent:BeamEvent                      0.000101019   0.000563015   0.00102501    0.000563015   0.000461996       2     
produce:pdvddaphne:DAPHNEReaderPDVD                21.8475       30.2185       38.5896       30.2185       8.37105         2     
produce:ophit:OpHitFinder                         0.0400315     0.0422989     0.0445663     0.0422989     0.0022674        2     
produce:wclsdatavd:WireCellToolkit                 96.2583       101.567       106.876       101.567       5.3086          2     
produce:gaushit:GausHitFinder                      1.17097       4.47589       7.78081       4.47589       3.30492         2     
produce:nhitsfilter:NumberOfHitsFilter           0.000668337   0.00132604    0.00198375    0.00132604    0.000657706       2     
produce:reco3d:SpacePointSolver                    10.1732       91.5739       172.975       91.5739       81.4007         2     
produce:hitpdune:DisambigFromSpacePoints          0.107214       4.69233       9.27744       4.69233       4.58511         2     
produce:cluster3d:Cluster3D                        1.20005       762.852       1524.5        762.852       761.652         2     
[art]:TriggerResults:TriggerResultInserter       5.3364e-05    6.70745e-05   8.0785e-05    6.70745e-05   1.37105e-05       2     
end_path:out1:RootOutput                         1.7102e-05    2.01415e-05   2.3181e-05    2.01415e-05   3.0395e-06        2     
end_path:out1:RootOutput(write)                    4.25245       4.25245       4.25245       4.25245          0            1     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 18121.4 MB
  Peak resident set size usage (VmHWM): 15690.3 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 04-Feb-2026 02:13:49 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 2 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1