Jobsub ID 303652.57@dunegpschedd01.fnal.gov

Jobsub ID	303652.57@dunegpschedd01.fnal.gov
Workflow ID	12719
Stage ID	1
User name	ykermaid@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
Submitted time	2026-02-03 22:50:26
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce513
Last heartbeat	2026-02-03 23:58:39
From worker node	Hostname	b9p12p8471.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-03 23:27:49
Input files	vd-protodune:np02vd_raw_run042433_0532_df-s04-d0_dw_0_20260203T220419.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-03 23:58:39
Saved logs	justin-logs:303652.57-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

x510b0620
Caught exception in looking up space point ptr... 0x91eddf10, 0x8f791680
Caught exception in looking up space point ptr... 0x8f791680, 0x91eddf10
Caught exception in looking up space point ptr... 0x9e2b3460, 0x9d782d00
Caught exception in looking up space point ptr... 0x9d782d00, 0x9e2b3460
Caught exception in looking up space point ptr... 0x598b9ea0, 0x598ba130
Caught exception in looking up space point ptr... 0x598ba130, 0x598b9c10
Caught exception in looking up space point ptr... 0x2ebee710, 0x2ebee1f0
Caught exception in looking up space point ptr... 0x2ebedf60, 0x2ebee710
Caught exception in looking up space point ptr... 0x4b819500, 0x4b819790
Caught exception in looking up space point ptr... 0x4b819790, 0x4b818f20
Caught exception in looking up space point ptr... 0xd34e5810, 0xd267fa90
Caught exception in looking up space point ptr... 0xd267fa90, 0xd34e5810
Caught exception in looking up space point ptr... 0x6d4c2650, 0x6d4c2550
Caught exception in looking up space point ptr... 0x6d4c2550, 0x6d4c2650
Caught exception in looking up space point ptr... 0xbbca4ca0, 0xbbdaa590
Caught exception in looking up space point ptr... 0xbbdaa590, 0xbbca4ca0
Caught exception in looking up space point ptr... 0xa09c0c60, 0x9f333f30
Caught exception in looking up space point ptr... 0x9f333f30, 0x9e916f30
Caught exception in looking up space point ptr... 0x438b9d10, 0x310f1e10
Caught exception in looking up space point ptr... 0x310f1e10, 0x438b9d10
Caught exception in looking up space point ptr... 0x8b6b89c0, 0x8b6b8830
Caught exception in looking up space point ptr... 0x8b6b8830, 0x8b6b81f0
Caught exception in looking up space point ptr... 0x794c5920, 0x791b7d40
Caught exception in looking up space point ptr... 0x791b7d40, 0x794c5920
Caught exception in looking up space point ptr... 0x196e7ac0, 0x196e7930
Caught exception in looking up space point ptr... 0x196e7930, 0x562f04e0
Caught exception in looking up space point ptr... 0xbb803360, 0xbb803540
Caught exception in looking up space point ptr... 0xbb803540, 0xbb803360
Caught exception in looking up space point ptr... 0x55965a80, 0x55966740
Caught exception in looking up space point ptr... 0x55966740, 0x55964340
Caught exception in looking up space point ptr... 0xac3d5ca0, 0xaddc5b50
Caught exception in looking up space point ptr... 0xaddc5b50, 0xac3d5ca0
Caught exception in looking up space point ptr... 0xb172c310, 0xafcde2d0
Caught exception in looking up space point ptr... 0xafcde2d0, 0xb172c310
Caught exception in looking up space point ptr... 0x5bb04fd0, 0x5ba87300
Caught exception in looking up space point ptr... 0x5ba87300, 0x5bb04fd0
Caught exception in looking up space point ptr... 0x99f0fe10, 0x99f0f8f0
Caught exception in looking up space point ptr... 0x99f0f8f0, 0x99f0fb80
Caught exception in looking up space point ptr... 0xaddad600, 0xac7e3900
Caught exception in looking up space point ptr... 0xac7e3900, 0xaddad600
Caught exception in looking up space point ptr... 0x65396470, 0x53e87820
Caught exception in looking up space point ptr... 0x53e87820, 0x653962e0
Caught exception in looking up space point ptr... 0xbcbaed60, 0xb96d51a0
Caught exception in looking up space point ptr... 0xb96d51a0, 0xbcbaec20
Caught exception in looking up space point ptr... 0x6538b890, 0x6538b700
Caught exception in looking up space point ptr... 0x6538b700, 0x6538b890
Caught exception in looking up space point ptr... 0x7d9edf90, 0x7da03650
Caught exception in looking up space point ptr... 0x7d9ede50, 0x52a30130
Caught exception in looking up space point ptr... 0x52a30130, 0x52a302c0
Caught exception in looking up space point ptr... 0x52a302c0, 0x7d9edf90
Caught exception in looking up space point ptr... 0xbbe69940, 0xbbcf4840
Caught exception in looking up space point ptr... 0xbbcf4840, 0xbbe69940
Caught exception in looking up space point ptr... 0xb787b740, 0xb3445690
Caught exception in looking up space point ptr... 0xb3445690, 0xb787b740
Caught exception in looking up space point ptr... 0x4a8670a0, 0x6ccb4ab0
Caught exception in looking up space point ptr... 0x6ccb4ab0, 0x4a8670a0
Caught exception in looking up space point ptr... 0xbbb42530, 0xbbb44c10
Caught exception in looking up space point ptr... 0xbbb44c10, 0xbbb42530
Caught exception in looking up space point ptr... 0xbbb42530, 0xbbb4fac0
Caught exception in looking up space point ptr... 0xbbb4fac0, 0xbbb42530
Caught exception in looking up space point ptr... 0xa66a0cb0, 0xb37a7030
Caught exception in looking up space point ptr... 0xb37a7030, 0xa6eee4c0
Caught exception in looking up space point ptr... 0xa16d5fb0, 0xa2a447b0
Caught exception in looking up space point ptr... 0xa2a447b0, 0xa4907ed0
Caught exception in looking up space point ptr... 0x2c68ddf0, 0x2c68dcd0
Caught exception in looking up space point ptr... 0x2c68dcd0, 0x2c68ddf0
Caught exception in looking up space point ptr... 0x8b6baa90, 0x88dc9e90
Caught exception in looking up space point ptr... 0x88dc9e90, 0x8b6baa90
Caught exception in looking up space point ptr... 0x8b6baa90, 0x8af0f3f0
Caught exception in looking up space point ptr... 0x8af0f3f0, 0x8b6baa90
Caught exception in looking up space point ptr... 0x8b6baa90, 0x8af0f710
Caught exception in looking up space point ptr... 0x8af0f710, 0x8b6baa90
Caught exception in looking up space point ptr... 0x2c761880, 0x2c761a10
Caught exception in looking up space point ptr... 0x2c761a10, 0x2c761560
Caught exception in looking up space point ptr... 0xc3a87260, 0xc31cd8f0
Caught exception in looking up space point ptr... 0xc31cd8f0, 0xc2e78db0
Caught exception in looking up space point ptr... 0x860f21f0, 0x65fb9350
Caught exception in looking up space point ptr... 0x65fb9350, 0x860f21f0
Caught exception in looking up space point ptr... 0xd0141800, 0xd48e8780
Caught exception in looking up space point ptr... 0xd48e8780, 0xd4a22b50
++++>>>> total num hits: 7274182, num free: 7072056

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         131.786       863.458       1595.13       863.458       731.672         2     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.5451e-05    0.000108767   0.000142082   0.000108767   3.33155e-05       2     
produce:tpcrawdecoder:PDVDTPCReader                8.60906       8.61441       8.61975       8.61441     0.00534721        2     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0414291     0.0726094      0.10379      0.0726094     0.0311803        2     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000561017   0.000780559    0.0010001    0.000780559   0.000219542       2     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000491926   0.000641303   0.00079068    0.000641303   0.000149377       2     
produce:beamevent:BeamEvent                      0.000102601   0.000577472   0.00105234    0.000577472   0.00047487        2     
produce:pdvddaphne:DAPHNEReaderPDVD                2.60318       2.69442       2.78566       2.69442      0.0912362        2     
produce:ophit:OpHitFinder                         0.0321222     0.0348064     0.0374906     0.0348064    0.00268423        2     
produce:wclsdatavd:WireCellToolkit                 99.0099       105.692       112.375       105.692       6.68241         2     
produce:gaushit:GausHitFinder                      1.23605       4.68928       8.14252       4.68928       3.45323         2     
produce:nhitsfilter:NumberOfHitsFilter           0.000536386    0.0010503    0.00156421     0.0010503    0.000513911       2     
produce:reco3d:SpacePointSolver                    12.8588       94.0717       175.285       94.0717       81.2129         2     
produce:hitpdune:DisambigFromSpacePoints          0.175439       3.39056       6.60567       3.39056       3.21512         2     
produce:cluster3d:Cluster3D                        1.64962       641.52        1281.39       641.52        639.87          2     
[art]:TriggerResults:TriggerResultInserter       3.2781e-05    6.6796e-05    0.000100811   6.6796e-05    3.4015e-05        2     
end_path:out1:RootOutput                          7.76e-06     1.92205e-05   3.0681e-05    1.92205e-05   1.14605e-05       2     
end_path:out1:RootOutput(write)                    5.29889       5.29889       5.29889       5.29889          0            1     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 16757.3 MB
  Peak resident set size usage (VmHWM): 14557.9 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 04-Feb-2026 00:57:42 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 2 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1