Jobsub ID 297920.35@dunegpschedd02.fnal.gov
| Jobsub ID | 297920.35@dunegpschedd02.fnal.gov |
| Workflow ID | 12719 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:54:27 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce02 |
| Last heartbeat | 2026-02-04 02:23:10 |
| From worker node | Hostname | wn1206080.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:36:56 |
| Input files | vd-protodune:np02vd_raw_run042433_0532_df-s04-d1_dw_0_20260203T220419.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 02:23:10 |
| Saved logs | justin-logs:297920.35-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
space point ptr... 0x91d5910, 0x40507990
Caught exception in looking up space point ptr... 0xbbf86870, 0xbbf54b60
Caught exception in looking up space point ptr... 0xbbf54b60, 0xbbf86870
Caught exception in looking up space point ptr... 0x15de71a40, 0x10317db40
Caught exception in looking up space point ptr... 0x10317db40, 0x15a936720
Caught exception in looking up space point ptr... 0x1cfec7540, 0x1cfd34030
Caught exception in looking up space point ptr... 0x1cfd34030, 0x1cfec7540
Caught exception in looking up space point ptr... 0x18e285a0, 0x18e284a0
Caught exception in looking up space point ptr... 0x18e284a0, 0x18e285a0
Caught exception in looking up space point ptr... 0x166080bd0, 0x166080940
Caught exception in looking up space point ptr... 0x166080940, 0x166080700
Caught exception in looking up space point ptr... 0x1674ee6d0, 0x4c6da250
Caught exception in looking up space point ptr... 0x4c6da250, 0x1674ee6d0
Caught exception in looking up space point ptr... 0x1661ebc60, 0x15ad03870
Caught exception in looking up space point ptr... 0x15ad03870, 0x8826d420
Caught exception in looking up space point ptr... 0x8826d420, 0x88235a00
Caught exception in looking up space point ptr... 0x88235a00, 0x8826dc10
Caught exception in looking up space point ptr... 0xbb8fcc30, 0xbb8fcaa0
Caught exception in looking up space point ptr... 0xbb8fcaa0, 0xbb8fcc30
Caught exception in looking up space point ptr... 0x9e749060, 0x9e5d2600
Caught exception in looking up space point ptr... 0x9e5d2600, 0x1d00a3b90
Caught exception in looking up space point ptr... 0x1c8c2cfe0, 0x1baabbc00
Caught exception in looking up space point ptr... 0x1baabbc00, 0x1c8c2ce50
Caught exception in looking up space point ptr... 0x167f35690, 0x615f1a10
Caught exception in looking up space point ptr... 0x615f1a10, 0x167f35790
Caught exception in looking up space point ptr... 0x81f8b710, 0x9eaff490
Caught exception in looking up space point ptr... 0x9eaff490, 0x81f8b710
Caught exception in looking up space point ptr... 0x9bb86fe0, 0x17e6edc30
Caught exception in looking up space point ptr... 0x17e6edc30, 0x9bb86fe0
Caught exception in looking up space point ptr... 0x9bb86fe0, 0x9bbd9f40
Caught exception in looking up space point ptr... 0x9bbd9f40, 0x9bb86fe0
Caught exception in looking up space point ptr... 0x2c0ce1c0, 0x2c0ce4c0
Caught exception in looking up space point ptr... 0x2c0ce4c0, 0x2c0ce2c0
Caught exception in looking up space point ptr... 0xf2cb1140, 0xf2cb1460
Caught exception in looking up space point ptr... 0xf2cb1460, 0xf2cb1140
Caught exception in looking up space point ptr... 0x8ff37430, 0x102802a20
Caught exception in looking up space point ptr... 0x102802a20, 0x8ff37430
Caught exception in looking up space point ptr... 0xd1807c10, 0xd1a0b3b0
Caught exception in looking up space point ptr... 0xd1a0b3b0, 0x98c5f150
Caught exception in looking up space point ptr... 0x58ce4aa0, 0x9dabefa0
Caught exception in looking up space point ptr... 0x9dabefa0, 0x58d74d20
Caught exception in looking up space point ptr... 0x18d8fd80, 0x1d0dec760
Caught exception in looking up space point ptr... 0x1d0dec760, 0x1d0dec8f0
Caught exception in looking up space point ptr... 0xe1c56480, 0x55a34d20
Caught exception in looking up space point ptr... 0x55a34d20, 0xe1c5ca00
Caught exception in looking up space point ptr... 0x4c539140, 0x4c5392d0
Caught exception in looking up space point ptr... 0x4c5392d0, 0x4c539140
Caught exception in looking up space point ptr... 0x3aef1ae0, 0xc318cfa0
Caught exception in looking up space point ptr... 0xc318cfa0, 0x3aef1ae0
Caught exception in looking up space point ptr... 0xa0b08280, 0x1ce9fbdd0
Caught exception in looking up space point ptr... 0x1ce9fc0f0, 0xa0b08280
Caught exception in looking up space point ptr... 0x17621010, 0x9bd60260
Caught exception in looking up space point ptr... 0x9bd60260, 0x144809f0
Caught exception in looking up space point ptr... 0x3f9adb90, 0x7afe8160
Caught exception in looking up space point ptr... 0x7afe8160, 0x3f9adb90
Caught exception in looking up space point ptr... 0x196cee820, 0x167f5b9d0
Caught exception in looking up space point ptr... 0x167f5b9d0, 0x196cee820
Caught exception in looking up space point ptr... 0xa5d5bde0, 0xa63e60e0
Caught exception in looking up space point ptr... 0xa63e60e0, 0xa5d5c420
Caught exception in looking up space point ptr... 0xa5d5c420, 0xa5d5bf70
Caught exception in looking up space point ptr... 0x17836f350, 0x9dec99a0
Caught exception in looking up space point ptr... 0x9dec99a0, 0x9dec9b30
Caught exception in looking up space point ptr... 0xb2b46dd0, 0xb2b46c40
Caught exception in looking up space point ptr... 0xb2b46c40, 0xb2b46f60
Caught exception in looking up space point ptr... 0x1a8ea40b0, 0x1a8ea3f20
Caught exception in looking up space point ptr... 0x1a8ea3f20, 0x1a8ea40b0
Caught exception in looking up space point ptr... 0x1426c7520, 0xcb683190
Caught exception in looking up space point ptr... 0xcb683190, 0xa18062d0
Caught exception in looking up space point ptr... 0x2e7a9c60, 0xf99c2c60
Caught exception in looking up space point ptr... 0xf99c2c60, 0x818fcfb0
Caught exception in looking up space point ptr... 0x818fcfb0, 0x818fd210
Caught exception in looking up space point ptr... 0x818fd470, 0x2e7a9c60
Caught exception in looking up space point ptr... 0x17f163b80, 0xe28c31f0
Caught exception in looking up space point ptr... 0xe28c31f0, 0x17f163b80
Caught exception in looking up space point ptr... 0xd22af3e0, 0x628170a0
Caught exception in looking up space point ptr... 0x628170a0, 0x1ab7d5c50
Caught exception in looking up space point ptr... 0x9584400, 0x93b3070
Caught exception in looking up space point ptr... 0x93b3070, 0x9584400
Caught exception in looking up space point ptr... 0xa2097f00, 0xa2097dc0
Caught exception in looking up space point ptr... 0xa2097dc0, 0xa2098180
++++>>>> total num hits: 7692068, num free: 7496465
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 557.526 1937.41 2854.17 2273.11 821.508 5
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.9728e-05 0.000164602 0.000395394 0.000111099 0.000115473 5
produce:tpcrawdecoder:PDVDTPCReader 207.922 309.464 398.565 296.496 74.4467 5
produce:triggerrawdecoder:PDVDTriggerReader4 1.03468 1.24889 1.51744 1.23597 0.163168 5
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0224968 0.0399013 0.0741149 0.0231034 0.021711 5
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0224712 0.0716147 0.267676 0.0225204 0.0980307 5
produce:beamevent:BeamEvent 0.000116289 0.000412855 0.00152355 0.000145975 0.000555466 5
produce:pdvddaphne:DAPHNEReaderPDVD 17.8747 25.1209 28.9033 27.2377 4.14063 5
produce:ophit:OpHitFinder 0.0429513 0.0523106 0.0583807 0.052415 0.00522156 5
produce:wclsdatavd:WireCellToolkit 187.832 198.855 215.525 191.767 10.8737 5
produce:gaushit:GausHitFinder 2.32142 14.7153 21.7489 17.8998 7.1783 5
produce:nhitsfilter:NumberOfHitsFilter 0.00135889 0.00177803 0.00205855 0.00182855 0.000242607 5
produce:reco3d:SpacePointSolver 23.173 174.01 277.015 225.751 95.915 5
produce:hitpdune:DisambigFromSpacePoints 0.330163 7.45139 11.6733 9.50044 4.08855 5
produce:cluster3d:Cluster3D 3.998 1182.04 2051.86 1557.95 727.343 5
[art]:TriggerResults:TriggerResultInserter 3.6839e-05 6.80138e-05 0.000140705 4.7629e-05 3.78061e-05 5
end_path:out1:RootOutput 4.548e-06 1.68898e-05 3.5607e-05 7.855e-06 1.32123e-05 5
end_path:out1:RootOutput(write) 9.14699 30.317 43.7238 34.1986 14.2937 4
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 20467.1 MB
Peak resident set size usage (VmHWM): 17776.9 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 02:21:35 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 5 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 