justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.33@dunegpschedd01.fnal.gov

Jobsub ID	303653.33@dunegpschedd01.fnal.gov
Workflow ID	12717
Stage ID	1
User name	ykermaid@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
Submitted time	2026-02-03 22:52:24
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2026-02-04 01:10:49
From worker node	Hostname	heplnc145.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-03 23:39:33
Input files	vd-protodune:np02vd_raw_run042421_0015_df-s04-d1_dw_0_20260203T133907.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-04 01:10:49
Saved logs	justin-logs:303653.33-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e80, 0x4c076fc0
Caught exception in looking up space point ptr... 0x8dcc97e0, 0x94d6c7f0
Caught exception in looking up space point ptr... 0x94d6c7f0, 0x8f347800
Caught exception in looking up space point ptr... 0xb50a0b80, 0xb4223b00
Caught exception in looking up space point ptr... 0xb38df510, 0xb50a0b80
Caught exception in looking up space point ptr... 0x14719db0, 0x54323440
Caught exception in looking up space point ptr... 0x54323440, 0x14719db0
Caught exception in looking up space point ptr... 0x166c6570, 0x166c6800
Caught exception in looking up space point ptr... 0x166c6800, 0x166c62e0
Caught exception in looking up space point ptr... 0x94b319d0, 0x92727dc0
Caught exception in looking up space point ptr... 0x95053860, 0x94b319d0
Caught exception in looking up space point ptr... 0x278c0e50, 0x278c1050
Caught exception in looking up space point ptr... 0x278c1050, 0x278c0e50
Caught exception in looking up space point ptr... 0x540bdd50, 0x542608f0
Caught exception in looking up space point ptr... 0x54260660, 0x540bdd50
Caught exception in looking up space point ptr... 0xe2ef0710, 0xe2ef1be0
Caught exception in looking up space point ptr... 0xe2ef1be0, 0xe8f23c40
Caught exception in looking up space point ptr... 0xee7e3680, 0xf2f57470
Caught exception in looking up space point ptr... 0xf2f57470, 0xee7e3680
Caught exception in looking up space point ptr... 0xee7e3680, 0xfe540b20
Caught exception in looking up space point ptr... 0xfe540b20, 0xee7e3680
Caught exception in looking up space point ptr... 0x63cbd750, 0x3804f600
Caught exception in looking up space point ptr... 0x192b2c20, 0x63cbd750
Caught exception in looking up space point ptr... 0xd9835710, 0xde25cac0
Caught exception in looking up space point ptr... 0xde25cac0, 0xd9835710
Caught exception in looking up space point ptr... 0x3308ce60, 0x3308d0b0
Caught exception in looking up space point ptr... 0x3308d0b0, 0x3307c390
Caught exception in looking up space point ptr... 0x96010a80, 0x95f110c0
Caught exception in looking up space point ptr... 0x95f110c0, 0x9068c380
Caught exception in looking up space point ptr... 0x5948f560, 0x5948f360
Caught exception in looking up space point ptr... 0x5948f360, 0x5948f560
Caught exception in looking up space point ptr... 0xc32ea6d0, 0xc3f66f60
Caught exception in looking up space point ptr... 0xc3f66f60, 0xc32ea6d0
Caught exception in looking up space point ptr... 0x2ed5da00, 0x2ed5d6e0
Caught exception in looking up space point ptr... 0x2ed5d6e0, 0x2ed5da00
Caught exception in looking up space point ptr... 0xed194e0, 0x3de1e810
Caught exception in looking up space point ptr... 0x3de1e810, 0xed194e0
Caught exception in looking up space point ptr... 0x273684b0, 0x27368370
Caught exception in looking up space point ptr... 0x718a0fb0, 0x273684b0
Caught exception in looking up space point ptr... 0xecaf2ea0, 0xebaca5a0
Caught exception in looking up space point ptr... 0xebaca5a0, 0xecaf2ea0
Caught exception in looking up space point ptr... 0x11fba610, 0x2b214640
Caught exception in looking up space point ptr... 0x2b214640, 0x11fba610
Caught exception in looking up space point ptr... 0x11fba610, 0x2b2144b0
Caught exception in looking up space point ptr... 0x2b2144b0, 0x11fba610
Caught exception in looking up space point ptr... 0xe1bfd40, 0xe1bfed0
Caught exception in looking up space point ptr... 0xe1bfed0, 0xe1c01f0
Caught exception in looking up space point ptr... 0xb99609e0, 0xbe9e7350
Caught exception in looking up space point ptr... 0xbe9e7350, 0xb99609e0
Caught exception in looking up space point ptr... 0xbdf6a750, 0xc773b110
Caught exception in looking up space point ptr... 0xc773b110, 0xc796c840
Caught exception in looking up space point ptr... 0x66e8e050, 0x56552710
Caught exception in looking up space point ptr... 0x56552710, 0x4574fcb0
Caught exception in looking up space point ptr... 0x979a5350, 0x9710d040
Caught exception in looking up space point ptr... 0x9710d040, 0x979a5350
Caught exception in looking up space point ptr... 0xdbc78a20, 0xc2d92350
Caught exception in looking up space point ptr... 0xc2d92350, 0xda8441f0
Caught exception in looking up space point ptr... 0x2f2787f0, 0x2f2784d0
Caught exception in looking up space point ptr... 0x2f2784d0, 0x2f2787f0
Caught exception in looking up space point ptr... 0x2f2787f0, 0x2f278340
Caught exception in looking up space point ptr... 0x2f278340, 0x2f2787f0
Caught exception in looking up space point ptr... 0xe6002ab0, 0xe5464b40
Caught exception in looking up space point ptr... 0xe5464b40, 0xe6002ab0
Caught exception in looking up space point ptr... 0xd8af7710, 0xd8af7a30
Caught exception in looking up space point ptr... 0xd8af7a30, 0xd8af78a0
Caught exception in looking up space point ptr... 0x12a7a780, 0x10374560
Caught exception in looking up space point ptr... 0x10374560, 0x103747b0
Caught exception in looking up space point ptr... 0x4b3e1870, 0xdb4e220
Caught exception in looking up space point ptr... 0xdb4e220, 0x4b3e1870
Caught exception in looking up space point ptr... 0x4b3e1870, 0x4b413300
Caught exception in looking up space point ptr... 0x4b413300, 0x4b3e1870
Caught exception in looking up space point ptr... 0x5c1836e0, 0x7308ad30
Caught exception in looking up space point ptr... 0x7308ad30, 0x71a69140
Caught exception in looking up space point ptr... 0x167a7d80, 0x1188eeb0
Caught exception in looking up space point ptr... 0x1188eeb0, 0x167a89a0
Caught exception in looking up space point ptr... 0x167a72d0, 0x118900c0
Caught exception in looking up space point ptr... 0x118900c0, 0x167a72d0
Caught exception in looking up space point ptr... 0xdf803e10, 0xdecf57d0
Caught exception in looking up space point ptr... 0xdecf57d0, 0xdf803e10
Caught exception in looking up space point ptr... 0xdf803e10, 0xdfc6eac0
Caught exception in looking up space point ptr... 0xdfc6eac0, 0xdf803e10
++++>>>> total num hits: 9761838, num free: 9514064

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         113.902       265.456       2369.53       160.372       483.25         20     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.8128e-05    0.000126441   0.000274828   0.000105729   4.60338e-05      20     
produce:tpcrawdecoder:PDVDTPCReader                11.3063       11.8674       12.9999       11.7746      0.407147        20     
produce:triggerrawdecoder:PDVDTriggerReader4      0.523348      0.550824      0.592661      0.546703      0.0201023       20     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0175679     0.0190606     0.0305501     0.0180322    0.00291587       20     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0862652     0.0897679     0.105508      0.0882307    0.00506931       20     
produce:beamevent:BeamEvent                      0.000101462   0.000229474   0.00153023    0.000157878   0.000300869      20     
produce:pdvddaphne:DAPHNEReaderPDVD                3.15926       3.70453       4.18442       3.72624      0.225566        20     
produce:ophit:OpHitFinder                         0.0291466     0.0400491     0.0477099     0.0407673    0.00453738       20     
produce:wclsdatavd:WireCellToolkit                 69.9876       79.4008       92.9877       76.9959       6.92144        20     
produce:gaushit:GausHitFinder                     0.762691       1.79699       9.04502       1.36781       1.69667        20     
produce:nhitsfilter:NumberOfHitsFilter           0.000216018   0.000529437   0.00149065    0.000465875   0.000254822      20     
produce:reco3d:SpacePointSolver                    5.89149       24.3719       227.129       13.8534       46.7309        20     
produce:hitpdune:DisambigFromSpacePoints          0.0783989     0.645101       8.03158      0.258604       1.69822        20     
produce:cluster3d:Cluster3D                        1.14887       138.138       2035.6        50.4446       435.838        20     
[art]:TriggerResults:TriggerResultInserter       2.6901e-05    4.72184e-05   9.6452e-05    3.8823e-05    2.08517e-05      20     
end_path:out1:RootOutput                          5.12e-06     9.6457e-06    2.4196e-05     8.436e-06    4.84287e-06      20     
end_path:out1:RootOutput(write)                    3.66676       5.05195       7.09592        4.902       0.881685        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 18937.2 MB
  Peak resident set size usage (VmHWM): 16593.3 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 04-Feb-2026 01:09:51 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 19 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 20 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1

justIN time: 2026-02-04 04:33:51 UTC       justIN version: 01.06.00