Jobsub ID 303847.2@dunegpschedd01.fnal.gov

Jobsub ID	303847.2@dunegpschedd01.fnal.gov
Workflow ID	12717
Stage ID	1
User name	ykermaid@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
Submitted time	2026-02-04 03:50:37
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce02
Last heartbeat	2026-02-04 05:01:55
From worker node	Hostname	wn2208241.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7513 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-04 03:51:05
Input files	vd-protodune:np02vd_raw_run042421_0050_df-s02-d0_dw_0_20260203T135802.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-04 05:01:55
Saved logs	justin-logs:303847.2-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

Caught exception in looking up space point ptr... 0xbbbc7250, 0xbbbcb4c0
Caught exception in looking up space point ptr... 0xa5321280, 0xa531eb90
Caught exception in looking up space point ptr... 0xa531eb90, 0xa5321280
Caught exception in looking up space point ptr... 0xdeb02980, 0xdeb03230
Caught exception in looking up space point ptr... 0xdeb02460, 0xdeb02980
Caught exception in looking up space point ptr... 0x5285af80, 0x61b68fd0
Caught exception in looking up space point ptr... 0x61b68fd0, 0x5285af80
Caught exception in looking up space point ptr... 0xb87ec9a0, 0xb8759990
Caught exception in looking up space point ptr... 0xb8759990, 0xb8765b10
Caught exception in looking up space point ptr... 0x68913d00, 0x68913f90
Caught exception in looking up space point ptr... 0x68913f90, 0x68900c80
Caught exception in looking up space point ptr... 0x9100c6d0, 0x8e988f00
Caught exception in looking up space point ptr... 0x8e988f00, 0x9100c6d0
Caught exception in looking up space point ptr... 0x76e31650, 0x76e313c0
Caught exception in looking up space point ptr... 0x76e313c0, 0x76e30ea0
Caught exception in looking up space point ptr... 0xe1cfa8d0, 0xe1cfa740
Caught exception in looking up space point ptr... 0xe1cfa740, 0xe1cfa8d0
Caught exception in looking up space point ptr... 0x106c8fb0, 0x106c38f0
Caught exception in looking up space point ptr... 0x106c38f0, 0x106c8fb0
Caught exception in looking up space point ptr... 0xfe39be60, 0xfe3115c0
Caught exception in looking up space point ptr... 0xfe3115c0, 0xfe2178b0
Caught exception in looking up space point ptr... 0xe1cfbb90, 0xe1cfba00
Caught exception in looking up space point ptr... 0xe1cfba00, 0xe1cfbb90
Caught exception in looking up space point ptr... 0xeab7e260, 0xea8e37d0
Caught exception in looking up space point ptr... 0xea8e37d0, 0xea8e3730
Caught exception in looking up space point ptr... 0xfe351660, 0xfe1ceaa0
Caught exception in looking up space point ptr... 0xfe1ceaa0, 0xfe1ceba0
Caught exception in looking up space point ptr... 0x770d0080, 0x6192bca0
Caught exception in looking up space point ptr... 0x6192bca0, 0x69522de0
Caught exception in looking up space point ptr... 0x898aaf30, 0x8a5a6840
Caught exception in looking up space point ptr... 0x8a5a6840, 0x898aaf30
Caught exception in looking up space point ptr... 0xfd8f6060, 0xfd18a7c0
Caught exception in looking up space point ptr... 0xfd18a7c0, 0xfd8f6060
Caught exception in looking up space point ptr... 0xfd8f6060, 0xfd18a680
Caught exception in looking up space point ptr... 0xfd18a680, 0xfd8f6060
Caught exception in looking up space point ptr... 0x106f60c0, 0x106fba10
Caught exception in looking up space point ptr... 0x106fba10, 0x106f6970
Caught exception in looking up space point ptr... 0xb001a320, 0xaedfa4b0
Caught exception in looking up space point ptr... 0xaedfa4b0, 0xb001a320
Caught exception in looking up space point ptr... 0x11f5677a0, 0x10e3822a0
Caught exception in looking up space point ptr... 0x10e3822a0, 0x11f5677a0
Caught exception in looking up space point ptr... 0x237140b0, 0x23713f60
Caught exception in looking up space point ptr... 0x23713f60, 0x2371ccc0
Caught exception in looking up space point ptr... 0x8e74f000, 0x9008ceb0
Caught exception in looking up space point ptr... 0x9008ceb0, 0x8b86f580
Caught exception in looking up space point ptr... 0x8b86f580, 0x92a2e950
Caught exception in looking up space point ptr... 0x92a2e950, 0x92abd6a0
Caught exception in looking up space point ptr... 0x10df04e00, 0x10d2b2280
Caught exception in looking up space point ptr... 0x10d2b2280, 0x10d2b1de0
Caught exception in looking up space point ptr... 0xb767cd30, 0xb4fd64a0
Caught exception in looking up space point ptr... 0xb4fd64a0, 0xb767cd30
Caught exception in looking up space point ptr... 0xd6c7fc00, 0xd8cb69e0
Caught exception in looking up space point ptr... 0xd8cb69e0, 0xd6c7fc00
Caught exception in looking up space point ptr... 0x98cb2580, 0x98cb2260
Caught exception in looking up space point ptr... 0x98cb2260, 0x98cb2580
Caught exception in looking up space point ptr... 0xe0368ed0, 0xe027b2f0
Caught exception in looking up space point ptr... 0xe027b2f0, 0xe027b160
Caught exception in looking up space point ptr... 0xcc6cf600, 0xc4f08290
Caught exception in looking up space point ptr... 0xc4f08290, 0xcc6cf600
Caught exception in looking up space point ptr... 0x97a86750, 0x970ec340
Caught exception in looking up space point ptr... 0x970ec340, 0x970ec4d0
Caught exception in looking up space point ptr... 0xb480e810, 0xb44f9470
Caught exception in looking up space point ptr... 0xb44f9470, 0xb480e810
Caught exception in looking up space point ptr... 0xfdc83d30, 0xfdc83e80
Caught exception in looking up space point ptr... 0xfdc83e80, 0xfdc84120
Caught exception in looking up space point ptr... 0x8bbd3650, 0x8c45c7f0
Caught exception in looking up space point ptr... 0x8c45c7f0, 0x8bbd3650
Caught exception in looking up space point ptr... 0x97c14220, 0x97c14390
Caught exception in looking up space point ptr... 0x97c14390, 0x97c14220
Caught exception in looking up space point ptr... 0x97c14220, 0x97be8600
Caught exception in looking up space point ptr... 0x97be8600, 0x97c14220
Caught exception in looking up space point ptr... 0x97c14220, 0x954ee880
Caught exception in looking up space point ptr... 0x954ee880, 0x97c14220
Caught exception in looking up space point ptr... 0xa47a8350, 0xa4f0d6a0
Caught exception in looking up space point ptr... 0xa4f0d6a0, 0xa47a8350
Caught exception in looking up space point ptr... 0xfdef6530, 0xfdc58e60
Caught exception in looking up space point ptr... 0xfdd0a970, 0xfdd0ab00
Caught exception in looking up space point ptr... 0xfdd0ab00, 0xfdef6530
Caught exception in looking up space point ptr... 0x54d44440, 0x5d757a80
Caught exception in looking up space point ptr... 0x5d757a80, 0x54d42cc0
++++>>>> total num hits: 12237028, num free: 11962018

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         333.904       2024.93       3715.95       2024.93       1691.02         2     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.00011172    0.000226536   0.000341352   0.000226536   0.000114816       2     
produce:tpcrawdecoder:PDVDTPCReader                201.323       205.026       208.728       205.026       3.70238         2     
produce:triggerrawdecoder:PDVDTriggerReader4      0.738024      0.854718      0.971411      0.854718      0.116693         2     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0225483     0.0228391      0.02313      0.0228391    0.000290852       2     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.131927       0.13252      0.133113       0.13252     0.00059292        2     
produce:beamevent:BeamEvent                      0.000122891   0.000823088   0.00152329    0.000823088   0.000700198       2     
produce:pdvddaphne:DAPHNEReaderPDVD                23.3837       23.644        23.9043       23.644       0.260304         2     
produce:ophit:OpHitFinder                         0.0415132     0.0436756     0.045838      0.0436756    0.00216238        2     
produce:wclsdatavd:WireCellToolkit                 90.0101       95.7491       101.488       95.7491       5.73901         2     
produce:gaushit:GausHitFinder                      1.25748       9.28307       17.3086       9.28307       8.02558         2     
produce:nhitsfilter:NumberOfHitsFilter           0.000816466   0.00129339    0.00177031    0.00129339    0.000476923       2     
produce:reco3d:SpacePointSolver                    10.2214       169.066       327.911       169.066       158.845         2     
produce:hitpdune:DisambigFromSpacePoints          0.113357       4.5226        8.93184       4.5226        4.40924         2     
produce:cluster3d:Cluster3D                        1.64454       1514.05       3026.46       1514.05       1512.41         2     
[art]:TriggerResults:TriggerResultInserter       8.5872e-05    0.000110283   0.000134693   0.000110283   2.44105e-05       2     
end_path:out1:RootOutput                         2.4957e-05    3.3864e-05    4.2771e-05    3.3864e-05     8.907e-06        2     
end_path:out1:RootOutput(write)                    4.98122       4.98122       4.98122       4.98122          0            1     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 22783.6 MB
  Peak resident set size usage (VmHWM): 20694.3 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 04-Feb-2026 05:00:41 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 2 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1