Jobsub ID 298131.0@dunegpschedd02.fnal.gov
| Jobsub ID | 298131.0@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-04 05:48:43 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce01 |
| Last heartbeat | 2026-02-04 08:20:59 |
| From worker node | Hostname | wn1907291.tier2.hep.manchester.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 05:50:00 |
| Input files | vd-protodune:np02vd_raw_run042421_0007_df-s04-d1_dw_0_20260203T133442.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 08:20:59 |
| Saved logs | justin-logs:298131.0-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
0
Caught exception in looking up space point ptr... 0x49a47670, 0xe2237db0
Caught exception in looking up space point ptr... 0xe2237db0, 0xe223b120
Caught exception in looking up space point ptr... 0xecab3c70, 0x73365490
Caught exception in looking up space point ptr... 0x73365490, 0xecab3c70
Caught exception in looking up space point ptr... 0xe9f91860, 0xe0881810
Caught exception in looking up space point ptr... 0xe0881810, 0xe10d4380
Caught exception in looking up space point ptr... 0x22e81490, 0xf73846e0
Caught exception in looking up space point ptr... 0xf73846e0, 0x22e81490
Caught exception in looking up space point ptr... 0x4ee9a1c0, 0x4ee9a4e0
Caught exception in looking up space point ptr... 0x4ee9a4e0, 0x4ee9a1c0
Caught exception in looking up space point ptr... 0x5a902820, 0x112aa7d00
Caught exception in looking up space point ptr... 0x112aa7d00, 0x4ab55480
Caught exception in looking up space point ptr... 0x64987ed0, 0x64987c40
Caught exception in looking up space point ptr... 0x649883f0, 0x64987ed0
Caught exception in looking up space point ptr... 0x53975c70, 0x538bfc00
Caught exception in looking up space point ptr... 0x538bfc00, 0x53975c70
Caught exception in looking up space point ptr... 0x51c30460, 0x606192e0
Caught exception in looking up space point ptr... 0x606192e0, 0x51c30460
Caught exception in looking up space point ptr... 0x51c30460, 0x2c2dbe00
Caught exception in looking up space point ptr... 0x2c2dbe00, 0x51c30460
Caught exception in looking up space point ptr... 0xf94e7900, 0xb07bca30
Caught exception in looking up space point ptr... 0xb07bca30, 0xf94e7900
Caught exception in looking up space point ptr... 0x87f88620, 0xc7092e50
Caught exception in looking up space point ptr... 0xc7092e50, 0x87f88620
Caught exception in looking up space point ptr... 0x87f88620, 0x3d788ae0
Caught exception in looking up space point ptr... 0x3d788ae0, 0x87f88620
Caught exception in looking up space point ptr... 0x3bfd9280, 0x4d2fd160
Caught exception in looking up space point ptr... 0x4d2fd160, 0x11590040
Caught exception in looking up space point ptr... 0x11590040, 0x3f023f80
Caught exception in looking up space point ptr... 0x3f023f80, 0xe6402740
Caught exception in looking up space point ptr... 0xe2190fd0, 0xe2047680
Caught exception in looking up space point ptr... 0xe2047680, 0xe1b20620
Caught exception in looking up space point ptr... 0x2b0d1110, 0x2b0d0bf0
Caught exception in looking up space point ptr... 0x2b0d0bf0, 0x2b0d06d0
Caught exception in looking up space point ptr... 0xef0e87f0, 0x8333c360
Caught exception in looking up space point ptr... 0x8333c360, 0x4bd76ab0
Caught exception in looking up space point ptr... 0xfad6cbc0, 0xfad6b030
Caught exception in looking up space point ptr... 0xbbbabb50, 0xfad6cbc0
Caught exception in looking up space point ptr... 0x63d5e440, 0xc9e96d00
Caught exception in looking up space point ptr... 0xc9e96d00, 0xf5e73210
Caught exception in looking up space point ptr... 0x3a5b1070, 0x696a6e50
Caught exception in looking up space point ptr... 0x696a6e50, 0x3a5b1070
Caught exception in looking up space point ptr... 0x3a5b1070, 0x14e06d60
Caught exception in looking up space point ptr... 0x14e06d60, 0x3a5b1070
Caught exception in looking up space point ptr... 0x14d104b0, 0x337001f0
Caught exception in looking up space point ptr... 0x337001f0, 0x4e332ba0
Caught exception in looking up space point ptr... 0x5261ae60, 0xf4e89e0
Caught exception in looking up space point ptr... 0xf4e89e0, 0x5297e470
Caught exception in looking up space point ptr... 0xc73f74e0, 0xd66195d0
Caught exception in looking up space point ptr... 0xc7771b50, 0xc73f74e0
Caught exception in looking up space point ptr... 0x49631d20, 0x495c5900
Caught exception in looking up space point ptr... 0x8de0a430, 0x49631d20
Caught exception in looking up space point ptr... 0xf8895680, 0xf8894870
Caught exception in looking up space point ptr... 0xf8894870, 0xf8894350
Caught exception in looking up space point ptr... 0xb7286140, 0xe01b2d90
Caught exception in looking up space point ptr... 0x49abc0d0, 0xb7286140
Caught exception in looking up space point ptr... 0xd1d60b50, 0xd1d60ce0
Caught exception in looking up space point ptr... 0xd2de5f60, 0xd1d60b50
Caught exception in looking up space point ptr... 0x84ff1600, 0x84ff1790
Caught exception in looking up space point ptr... 0x84ff1790, 0x84058c40
Caught exception in looking up space point ptr... 0xf855e700, 0xf855e570
Caught exception in looking up space point ptr... 0x3d3c4f50, 0xf855e700
Caught exception in looking up space point ptr... 0x1067112b0, 0x10607c230
Caught exception in looking up space point ptr... 0x10607c230, 0x1067112b0
Caught exception in looking up space point ptr... 0xc7cb4fb0, 0x6c8330f0
Caught exception in looking up space point ptr... 0x6ceb1740, 0xc7cb4fb0
Caught exception in looking up space point ptr... 0xd1864a60, 0x72441ae0
Caught exception in looking up space point ptr... 0x72441ae0, 0x724aa760
Caught exception in looking up space point ptr... 0x72e3b870, 0x7238ae30
Caught exception in looking up space point ptr... 0x7238ae30, 0x72e3b870
Caught exception in looking up space point ptr... 0xb3677640, 0xb3693800
Caught exception in looking up space point ptr... 0xb3693800, 0xb367d2a0
Caught exception in looking up space point ptr... 0x85593880, 0x855936f0
Caught exception in looking up space point ptr... 0x855936f0, 0x85593880
Caught exception in looking up space point ptr... 0xe62f3aa0, 0xd9ca9a0
Caught exception in looking up space point ptr... 0xd9ca9a0, 0xe62f3aa0
Caught exception in looking up space point ptr... 0x27d68a50, 0x249bfa60
Caught exception in looking up space point ptr... 0x249bfa60, 0x249bfbf0
Caught exception in looking up space point ptr... 0x105a0f070, 0x105a0f390
Caught exception in looking up space point ptr... 0x105a0f390, 0x10686f210
++++>>>> total num hits: 7998037, num free: 7806009
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 186.177 382.358 2418.98 276.158 461.01 23
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000120893 0.000249401 0.000971196 0.000199311 0.000182707 23
produce:tpcrawdecoder:PDVDTPCReader 16.5422 17.1522 19.3607 16.8426 0.700863 23
produce:triggerrawdecoder:PDVDTriggerReader4 0.661856 0.735756 1.77149 0.691247 0.221195 23
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0225081 0.0230315 0.0263353 0.0227253 0.000837698 23
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.109655 0.113894 0.134751 0.11133 0.00707056 23
produce:beamevent:BeamEvent 0.000171397 0.000306045 0.00106231 0.000263311 0.000170613 23
produce:pdvddaphne:DAPHNEReaderPDVD 7.44352 8.06424 8.86951 7.96058 0.347994 23
produce:ophit:OpHitFinder 0.054677 0.0780083 0.100574 0.0799383 0.00991406 23
produce:wclsdatavd:WireCellToolkit 134.393 146.453 169.009 146.138 8.19002 23
produce:gaushit:GausHitFinder 1.24567 3.37537 13.8563 2.78312 2.64966 23
produce:nhitsfilter:NumberOfHitsFilter 0.00045308 0.000793078 0.00188499 0.000740661 0.000328541 23
produce:reco3d:SpacePointSolver 8.25134 27.7908 239.265 13.7587 48.4008 23
produce:hitpdune:DisambigFromSpacePoints 0.104589 0.941313 7.62766 0.471676 1.69025 23
produce:cluster3d:Cluster3D 1.48242 169.18 1991.44 75.7753 410.854 23
[art]:TriggerResults:TriggerResultInserter 3.5223e-05 6.8474e-05 0.00016825 5.801e-05 3.14856e-05 23
end_path:out1:RootOutput 8.926e-06 1.47628e-05 3.6082e-05 1.3418e-05 6.5135e-06 23
end_path:out1:RootOutput(write) 4.53299 8.76501 26.213 8.38635 4.05291 22
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 18187.6 MB
Peak resident set size usage (VmHWM): 15875.1 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 08:19:35 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 22 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 23 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 