justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.147@dunegpschedd01.fnal.gov

Jobsub ID	303647.147@dunegpschedd01.fnal.gov
Workflow ID	12717
Stage ID	1
User name	ykermaid@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
Submitted time	2026-02-03 22:46:24
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce503
Last heartbeat	2026-02-03 23:14:30
From worker node	Hostname	b9p05p0564.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-03 22:46:58
Input files	vd-protodune:np02vd_raw_run042421_0050_df-s02-d0_dw_0_20260203T135802.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-03 23:14:30
Saved logs	justin-logs:303647.147-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

x73308e00
Caught exception in looking up space point ptr... 0x73308e00, 0x73309070
Caught exception in looking up space point ptr... 0xd4158b70, 0xd4158c10
Caught exception in looking up space point ptr... 0xd4158c10, 0xd23068a0
Caught exception in looking up space point ptr... 0xd1ebaff0, 0xd4158d50
Caught exception in looking up space point ptr... 0xd4158d50, 0xd1ebaff0
Caught exception in looking up space point ptr... 0xd41e81f0, 0xd41e7f90
Caught exception in looking up space point ptr... 0xd41e7f90, 0xd41e81f0
Caught exception in looking up space point ptr... 0xd41e81f0, 0xd16cfe80
Caught exception in looking up space point ptr... 0xd16cfe80, 0xd41e81f0
Caught exception in looking up space point ptr... 0xcf974700, 0xcf8ed920
Caught exception in looking up space point ptr... 0xcf8ed920, 0xcf974700
Caught exception in looking up space point ptr... 0xd08dc250, 0xd03561f0
Caught exception in looking up space point ptr... 0xd03561f0, 0xd08dc250
Caught exception in looking up space point ptr... 0xa21abac0, 0xa21ab930
Caught exception in looking up space point ptr... 0xa21ab930, 0xa19b2060
Caught exception in looking up space point ptr... 0xd102ceb0, 0xd421e560
Caught exception in looking up space point ptr... 0xd421e560, 0xd0f6b290
Caught exception in looking up space point ptr... 0x8c0b1890, 0x8c30b8b0
Caught exception in looking up space point ptr... 0x8c30b8b0, 0x8c0a9a50
Caught exception in looking up space point ptr... 0xa21ac5b0, 0xa12c3b10
Caught exception in looking up space point ptr... 0xa12c3b10, 0xa21ac420
Caught exception in looking up space point ptr... 0xa21ac420, 0xa12c3980
Caught exception in looking up space point ptr... 0x775919e0, 0x773b4840
Caught exception in looking up space point ptr... 0x773b4840, 0x773b4700
Caught exception in looking up space point ptr... 0xb14ddf30, 0xb03ba1a0
Caught exception in looking up space point ptr... 0xb03ba1a0, 0xb03ba010
Caught exception in looking up space point ptr... 0xd205f5b0, 0xd20880a0
Caught exception in looking up space point ptr... 0xd218ed50, 0xd205f5b0
Caught exception in looking up space point ptr... 0xcf73f2b0, 0xcf942d90
Caught exception in looking up space point ptr... 0xd2320d50, 0xcf73f2b0
Caught exception in looking up space point ptr... 0xc8b1e6f0, 0xc979af80
Caught exception in looking up space point ptr... 0xc979af80, 0xc8b1e6f0
Caught exception in looking up space point ptr... 0x28bd93b0, 0x4ed74b20
Caught exception in looking up space point ptr... 0x4ed74b20, 0x28bd93b0
Caught exception in looking up space point ptr... 0x8b1c3290, 0x8b1c35b0
Caught exception in looking up space point ptr... 0x8b1c35b0, 0x8b1c3740
Caught exception in looking up space point ptr... 0xf4b2b800, 0xf4b2b700
Caught exception in looking up space point ptr... 0xf4b2b700, 0xf4b2b800
Caught exception in looking up space point ptr... 0x9e24eb70, 0x9f014610
Caught exception in looking up space point ptr... 0x9f014610, 0x9e24eb70
Caught exception in looking up space point ptr... 0x9edaffa0, 0x9edafc80
Caught exception in looking up space point ptr... 0x9edafc80, 0x9edaffa0
Caught exception in looking up space point ptr... 0x9edaffa0, 0x9edafaf0
Caught exception in looking up space point ptr... 0x9edafaf0, 0x9edaffa0
Caught exception in looking up space point ptr... 0xf4b052e0, 0xf4ac9520
Caught exception in looking up space point ptr... 0xf4ac9520, 0xf49c2430
Caught exception in looking up space point ptr... 0x8e63cdc0, 0x8ffb4fa0
Caught exception in looking up space point ptr... 0x8ffb4fa0, 0x8ee20040
Caught exception in looking up space point ptr... 0x8ee20040, 0x8ec99990
Caught exception in looking up space point ptr... 0x8ec99990, 0x944c15e0
Caught exception in looking up space point ptr... 0x944c15e0, 0x8e413620
Caught exception in looking up space point ptr... 0x8e413620, 0x8e63cdc0
Caught exception in looking up space point ptr... 0x66d48a70, 0x517e0080
Caught exception in looking up space point ptr... 0x517e0080, 0x517e0320
Caught exception in looking up space point ptr... 0x831415d0, 0x83247c50
Caught exception in looking up space point ptr... 0x83247c50, 0x831415d0
Caught exception in looking up space point ptr... 0x8ce16ef0, 0x8ce16bd0
Caught exception in looking up space point ptr... 0x8ce16bd0, 0x8ce16d60
Caught exception in looking up space point ptr... 0x8d0af010, 0x8d0aeb60
Caught exception in looking up space point ptr... 0x8d0aeb60, 0x8d0af010
Caught exception in looking up space point ptr... 0xa1b1dcd0, 0xa1b1db40
Caught exception in looking up space point ptr... 0xa1b1db40, 0xa1b1dff0
Caught exception in looking up space point ptr... 0xa1a9d1c0, 0xa1a9d030
Caught exception in looking up space point ptr... 0xa1a9d030, 0xa1a9d4e0
Caught exception in looking up space point ptr... 0xe8500000, 0xebb207f0
Caught exception in looking up space point ptr... 0xebb207f0, 0xe8500000
Caught exception in looking up space point ptr... 0xa1b21510, 0xa1b21060
Caught exception in looking up space point ptr... 0xa1b21060, 0xa1b211f0
Caught exception in looking up space point ptr... 0xa1aa0b90, 0xa1aa06e0
Caught exception in looking up space point ptr... 0xa1aa06e0, 0xa1aa0870
Caught exception in looking up space point ptr... 0xa1a99fc0, 0xa1a99b10
Caught exception in looking up space point ptr... 0xa1a99b10, 0xa1a99fc0
Caught exception in looking up space point ptr... 0x8d0acc20, 0x8ce18fc0
Caught exception in looking up space point ptr... 0x8ce18fc0, 0x8d0acf40
Caught exception in looking up space point ptr... 0x8d0af970, 0x8d0af7e0
Caught exception in looking up space point ptr... 0x8d0af7e0, 0x8d0afc90
Caught exception in looking up space point ptr... 0x8d0aad20, 0x8d0aa870
Caught exception in looking up space point ptr... 0x8d0aa870, 0x8d0aad20
Caught exception in looking up space point ptr... 0x47989550, 0x5094e410
Caught exception in looking up space point ptr... 0x5094e410, 0x47987dd0
++++>>>> total num hits: 9814576, num free: 9579851

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         76.6853       777.675       1478.66       777.675       700.989         2     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.6281e-05    0.000217392   0.000338504   0.000217392   0.000121111       2     
produce:tpcrawdecoder:PDVDTPCReader                3.8782        3.92673       3.97526       3.92673      0.0485261        2     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0113327     0.0114945     0.0116562     0.0114945    0.000161767       2     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000379435   0.000542807   0.000706179   0.000542807   0.000163372       2     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00191125    0.00211173    0.00231221    0.00211173    0.000200478       2     
produce:beamevent:BeamEvent                      6.2901e-05    0.000591232   0.00111956    0.000591232   0.000528331       2     
produce:pdvddaphne:DAPHNEReaderPDVD                2.29146       2.53777       2.78408       2.53777      0.246313         2     
produce:ophit:OpHitFinder                         0.0276941     0.0333465     0.0389989     0.0333465     0.0056524        2     
produce:wclsdatavd:WireCellToolkit                 57.2483       59.1881       61.1279       59.1881       1.9398          2     
produce:gaushit:GausHitFinder                      0.67036       3.07047       5.47058       3.07047       2.40011         2     
produce:nhitsfilter:NumberOfHitsFilter           0.000388505   0.000662943   0.000937382   0.000662943   0.000274439       2     
produce:reco3d:SpacePointSolver                    7.29807       92.6583       178.019       92.6583       85.3602         2     
produce:hitpdune:DisambigFromSpacePoints          0.101208       2.12996       4.1587        2.12996       2.02875         2     
produce:cluster3d:Cluster3D                        1.31255       612.438       1223.56       612.438       611.126         2     
[art]:TriggerResults:TriggerResultInserter       3.3211e-05    9.57915e-05   0.000158372   9.57915e-05   6.25805e-05       2     
end_path:out1:RootOutput                          7.94e-06     3.16705e-05   5.5401e-05    3.16705e-05   2.37305e-05       2     
end_path:out1:RootOutput(write)                    3.32251       3.32251       3.32251       3.32251          0            1     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 18839.7 MB
  Peak resident set size usage (VmHWM): 16309.4 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 04-Feb-2026 00:13:56 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 2 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1

justIN time: 2026-02-04 04:50:34 UTC       justIN version: 01.06.00