Jobsub ID 297919.91@dunegpschedd02.fnal.gov
| Jobsub ID | 297919.91@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:54:25 |
| Site | UK_RAL-PPD |
| Entry | CMSHTPC_T2_UK_SGrid_RALPP_hep208 |
| Last heartbeat | 2026-02-04 00:13:44 |
| From worker node | Hostname | heplnc162.pp.rl.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:31:04 |
| Input files | vd-protodune:np02vd_raw_run042421_0143_df-s04-d1_dw_0_20260203T144821.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 00:13:44 |
| Saved logs | justin-logs:297919.91-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
int ptr... 0x6e096d10, 0x6f061e20
Caught exception in looking up space point ptr... 0x6f061e20, 0x6f061b10
Caught exception in looking up space point ptr... 0x6f061b10, 0x6e1745d0
Caught exception in looking up space point ptr... 0xe1d46750, 0xe1d465c0
Caught exception in looking up space point ptr... 0xe1d465c0, 0xe1d46750
Caught exception in looking up space point ptr... 0xbf3c6480, 0xbf3c69e0
Caught exception in looking up space point ptr... 0xbf3c69e0, 0xbf693ec0
Caught exception in looking up space point ptr... 0xaed4c9f0, 0xaa8c1b60
Caught exception in looking up space point ptr... 0xaa8c1b60, 0xb9969f70
Caught exception in looking up space point ptr... 0xc8786540, 0xc8786450
Caught exception in looking up space point ptr... 0xc8786450, 0xc8786540
Caught exception in looking up space point ptr... 0xab29ace0, 0xaadbe140
Caught exception in looking up space point ptr... 0xaadbe140, 0xab29ace0
Caught exception in looking up space point ptr... 0xcd40e0e0, 0xcd40de10
Caught exception in looking up space point ptr... 0xcd40dff0, 0xcd40e0e0
Caught exception in looking up space point ptr... 0x8fe33fb0, 0x8f754040
Caught exception in looking up space point ptr... 0x8f754040, 0x8fe33fb0
Caught exception in looking up space point ptr... 0xb1f88270, 0xb95d5b20
Caught exception in looking up space point ptr... 0xb95d5b20, 0xab021940
Caught exception in looking up space point ptr... 0x4401e2b0, 0x3e41f2a0
Caught exception in looking up space point ptr... 0x39fd21a0, 0x3ba5eb60
Caught exception in looking up space point ptr... 0x3ba5eb60, 0x3e295820
Caught exception in looking up space point ptr... 0x3e295820, 0x4401e2b0
Caught exception in looking up space point ptr... 0xccec880, 0x213dc470
Caught exception in looking up space point ptr... 0x213dc470, 0xcd64e40
Caught exception in looking up space point ptr... 0x95d356b0, 0x95d35320
Caught exception in looking up space point ptr... 0x95f86b20, 0x95d356b0
Caught exception in looking up space point ptr... 0xe77871f0, 0xe7786f60
Caught exception in looking up space point ptr... 0xe7786f60, 0xe7a1c2b0
Caught exception in looking up space point ptr... 0xc4644b50, 0xc4e03660
Caught exception in looking up space point ptr... 0xc4e03660, 0xc4e037f0
Caught exception in looking up space point ptr... 0x6f76d200, 0x6f77d3f0
Caught exception in looking up space point ptr... 0x6f77d3f0, 0x6f7c9400
Caught exception in looking up space point ptr... 0x8cbdc930, 0x8cb8f850
Caught exception in looking up space point ptr... 0x8cb8f850, 0x8d568a10
Caught exception in looking up space point ptr... 0xaa726e20, 0xb1c6e560
Caught exception in looking up space point ptr... 0xb1c6e560, 0xb29d7ae0
Caught exception in looking up space point ptr... 0x9ae8860, 0x9ae8d80
Caught exception in looking up space point ptr... 0x9ae8d80, 0x9ae9010
Caught exception in looking up space point ptr... 0xdb421b30, 0xe460f210
Caught exception in looking up space point ptr... 0xe460f210, 0xd5288690
Caught exception in looking up space point ptr... 0xd1ca9100, 0xd1ca8f20
Caught exception in looking up space point ptr... 0xd1ca8f20, 0xd1ca8e30
Caught exception in looking up space point ptr... 0x9129e240, 0x913de0f0
Caught exception in looking up space point ptr... 0x913de0f0, 0x91267ec0
Caught exception in looking up space point ptr... 0xe2c5ba80, 0xe1ef76c0
Caught exception in looking up space point ptr... 0xe1ef76c0, 0xe2c5ba80
Caught exception in looking up space point ptr... 0xe3c60d30, 0xe38ee4a0
Caught exception in looking up space point ptr... 0xe1d17c90, 0xe3c60d30
Caught exception in looking up space point ptr... 0x947483a0, 0x91a7d610
Caught exception in looking up space point ptr... 0x91a7d610, 0x947483a0
Caught exception in looking up space point ptr... 0x947483a0, 0x924346c0
Caught exception in looking up space point ptr... 0x924346c0, 0x947483a0
Caught exception in looking up space point ptr... 0xc1fddd0, 0x11ad2950
Caught exception in looking up space point ptr... 0x11ad2950, 0xc1fddd0
Caught exception in looking up space point ptr... 0x68518940, 0x6b0195c0
Caught exception in looking up space point ptr... 0x6b0195c0, 0x68518940
Caught exception in looking up space point ptr... 0xde91ff00, 0xe3937ad0
Caught exception in looking up space point ptr... 0xe3937ad0, 0xe6c396c0
Caught exception in looking up space point ptr... 0xe39368a0, 0xe62078c0
Caught exception in looking up space point ptr... 0xe62078c0, 0xe39368a0
Caught exception in looking up space point ptr... 0xe3904af0, 0xe620fc90
Caught exception in looking up space point ptr... 0xe620fc90, 0xe3904af0
Caught exception in looking up space point ptr... 0xa2a4cc0, 0xa2a4e50
Caught exception in looking up space point ptr... 0xa2a4e50, 0xa2a4cc0
Caught exception in looking up space point ptr... 0xa2a4cc0, 0xa2b7f30
Caught exception in looking up space point ptr... 0xa2b7f30, 0xa2a4cc0
Caught exception in looking up space point ptr... 0xa2a4cc0, 0xa2b78f0
Caught exception in looking up space point ptr... 0xa2b78f0, 0xa2a4cc0
Caught exception in looking up space point ptr... 0xe38dea60, 0xe3bfddb0
Caught exception in looking up space point ptr... 0xe3bfddb0, 0xe3bf9970
Caught exception in looking up space point ptr... 0xa97a29c0, 0xa97a2830
Caught exception in looking up space point ptr... 0xa97a2830, 0xa97a29c0
Caught exception in looking up space point ptr... 0x8f2fbb0, 0x8f2bec0
Caught exception in looking up space point ptr... 0x8f2bec0, 0x8f23960
Caught exception in looking up space point ptr... 0xa9842030, 0xa9841d10
Caught exception in looking up space point ptr... 0xa9841d10, 0xa9842030
Caught exception in looking up space point ptr... 0x12079b60, 0x12079c60
Caught exception in looking up space point ptr... 0x12079c60, 0x12079b60
Caught exception in looking up space point ptr... 0xda970f70, 0xda970ad0
Caught exception in looking up space point ptr... 0xda970ad0, 0xda970f70
++++>>>> total num hits: 8994983, num free: 8761003
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 107.498 602.334 2005.86 147.987 810.526 4
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000141746 0.000246297 0.000368601 0.00023742 8.28654e-05 4
produce:tpcrawdecoder:PDVDTPCReader 12.5292 13.6533 15.6715 13.2063 1.25589 4
produce:triggerrawdecoder:PDVDTriggerReader4 0.544299 0.640934 0.715528 0.651955 0.0738131 4
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0179746 0.0352519 0.0841484 0.0194423 0.0282464 4
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0883787 0.0888167 0.0893948 0.0887467 0.000377262 4
produce:beamevent:BeamEvent 0.000155792 0.00050029 0.00142755 0.000208912 0.000536081 4
produce:pdvddaphne:DAPHNEReaderPDVD 3.59258 3.74526 3.88396 3.75226 0.108406 4
produce:ophit:OpHitFinder 0.0422464 0.0454608 0.052471 0.0435629 0.00409955 4
produce:wclsdatavd:WireCellToolkit 70.9395 83.3792 100.652 80.9628 12.5005 4
produce:gaushit:GausHitFinder 1.16402 3.33096 8.66045 1.74969 3.08775 4
produce:nhitsfilter:NumberOfHitsFilter 0.000285505 0.00129571 0.00378024 0.000558557 0.0014409 4
produce:reco3d:SpacePointSolver 9.02621 70.6064 243.793 14.8034 100.079 4
produce:hitpdune:DisambigFromSpacePoints 0.112873 2.23908 8.27954 0.281954 3.48818 4
produce:cluster3d:Cluster3D 1.68694 420.411 1624.29 27.8317 695.343 4
[art]:TriggerResults:TriggerResultInserter 4.1137e-05 7.01842e-05 0.000116929 6.13355e-05 3.00702e-05 4
end_path:out1:RootOutput 7.605e-06 1.56623e-05 3.0889e-05 1.20775e-05 9.51735e-06 4
end_path:out1:RootOutput(write) 4.82942 5.49574 6.60241 5.05538 0.787957 3
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 18578.1 MB
Peak resident set size usage (VmHWM): 15744.5 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:13:00 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 4 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 