Jobsub ID 298116.0@dunegpschedd02.fnal.gov
| Jobsub ID | 298116.0@dunegpschedd02.fnal.gov |
| Workflow ID | 12719 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-04 04:46:40 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce05 |
| Last heartbeat | 2026-02-04 06:07:45 |
| From worker node | Hostname | dune001-1851175.0-lcg2710.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 9654 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 04:47:52 |
| Input files | vd-protodune:np02vd_raw_run042433_0048_df-s02-d1_dw_0_20260203T190020.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 06:07:45 |
| Saved logs | justin-logs:298116.0-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
oking up space point ptr... 0xf5137510, 0xf4d93020
Caught exception in looking up space point ptr... 0xeb697680, 0xe6770c90
Caught exception in looking up space point ptr... 0xe6770c90, 0xe5c38530
Caught exception in looking up space point ptr... 0xe99b7410, 0xeb697680
Caught exception in looking up space point ptr... 0x106dabce0, 0x1075843e0
Caught exception in looking up space point ptr... 0x1075843e0, 0x106dabce0
Caught exception in looking up space point ptr... 0x1059a9f80, 0x10e86c2e0
Caught exception in looking up space point ptr... 0x10e86c2e0, 0x1059a9f80
Caught exception in looking up space point ptr... 0x10d1659f0, 0x119724ba0
Caught exception in looking up space point ptr... 0x119724ba0, 0x10d1659f0
Caught exception in looking up space point ptr... 0xf37f5360, 0xf37f4af0
Caught exception in looking up space point ptr... 0xf37f4af0, 0xf37f4810
Caught exception in looking up space point ptr... 0xe125b510, 0xe125ba80
Caught exception in looking up space point ptr... 0xe125ba80, 0xe125b110
Caught exception in looking up space point ptr... 0x58944e10, 0x58944c70
Caught exception in looking up space point ptr... 0x58944c70, 0x58944e10
Caught exception in looking up space point ptr... 0x2e51e760, 0x67966a10
Caught exception in looking up space point ptr... 0x67966a10, 0x2e51e760
Caught exception in looking up space point ptr... 0xf3e34260, 0xf3e344f0
Caught exception in looking up space point ptr... 0xf3e344f0, 0xf3e33f80
Caught exception in looking up space point ptr... 0xcdb53de0, 0xd9a6f9b0
Caught exception in looking up space point ptr... 0xdfbb6bc0, 0xcdb53de0
Caught exception in looking up space point ptr... 0x2b92e250, 0x7c151e40
Caught exception in looking up space point ptr... 0x7c151e40, 0x7c156860
Caught exception in looking up space point ptr... 0x8f5833f0, 0x8f5834f0
Caught exception in looking up space point ptr... 0x8f5834f0, 0x8f5833f0
Caught exception in looking up space point ptr... 0x10b60d430, 0x10ea3cfb0
Caught exception in looking up space point ptr... 0x10ea3cfb0, 0x10b60d430
Caught exception in looking up space point ptr... 0x82f2e6c0, 0x82f32660
Caught exception in looking up space point ptr... 0x82f34ba0, 0x82f2e6c0
Caught exception in looking up space point ptr... 0x8dbb2450, 0x91d4b490
Caught exception in looking up space point ptr... 0x91d4b490, 0x91d4ba30
Caught exception in looking up space point ptr... 0x119176e40, 0x119b918e0
Caught exception in looking up space point ptr... 0x119b918e0, 0x119176e40
Caught exception in looking up space point ptr... 0x118cd1290, 0x117de7cd0
Caught exception in looking up space point ptr... 0x117de7cd0, 0x118cd1290
Caught exception in looking up space point ptr... 0xb91a7e30, 0xb9197490
Caught exception in looking up space point ptr... 0xb9197490, 0xb919a770
Caught exception in looking up space point ptr... 0xf3a24cf0, 0xf3a24f80
Caught exception in looking up space point ptr... 0xf3a24f80, 0xf3a25210
Caught exception in looking up space point ptr... 0xe72f93e0, 0xe944ad00
Caught exception in looking up space point ptr... 0xe944ad00, 0xe72f93e0
Caught exception in looking up space point ptr... 0xa7d8ccf0, 0xa7d8c9d0
Caught exception in looking up space point ptr... 0xa7d8c9d0, 0xa7d8c6b0
Caught exception in looking up space point ptr... 0x5c573c00, 0x4f9b1120
Caught exception in looking up space point ptr... 0x4f9b1120, 0x4f9b12f0
Caught exception in looking up space point ptr... 0xc7d99760, 0xc7cd2f50
Caught exception in looking up space point ptr... 0xc7cd2f50, 0xc7d99760
Caught exception in looking up space point ptr... 0xfa7023c0, 0xfa7022d0
Caught exception in looking up space point ptr... 0xfa7022d0, 0xfa7023c0
Caught exception in looking up space point ptr... 0x883a2400, 0x312eabb0
Caught exception in looking up space point ptr... 0x7b7adeb0, 0x729df460
Caught exception in looking up space point ptr... 0x729df460, 0x6a3f4380
Caught exception in looking up space point ptr... 0x6a3f4380, 0x7c4c8980
Caught exception in looking up space point ptr... 0x7c4c8980, 0x883a2400
Caught exception in looking up space point ptr... 0xf4fefae0, 0xf4fef630
Caught exception in looking up space point ptr... 0xf4fef630, 0xf4fefae0
Caught exception in looking up space point ptr... 0x11c9f6290, 0x11c9f5de0
Caught exception in looking up space point ptr... 0x11c9f5de0, 0x11c9f6290
Caught exception in looking up space point ptr... 0xf3e42430, 0xf3e425c0
Caught exception in looking up space point ptr... 0xf3e425c0, 0xf3e42430
Caught exception in looking up space point ptr... 0xf3e42c00, 0xf3e42d90
Caught exception in looking up space point ptr... 0xf3e42d90, 0xf3e42c00
Caught exception in looking up space point ptr... 0x58944b20, 0x589446e0
Caught exception in looking up space point ptr... 0x589446e0, 0x58944880
Caught exception in looking up space point ptr... 0xea0640f0, 0xe9ff70e0
Caught exception in looking up space point ptr... 0xe9ff70e0, 0xea0640f0
Caught exception in looking up space point ptr... 0x472ab360, 0x4713bba0
Caught exception in looking up space point ptr... 0x4713bba0, 0x473c7f10
Caught exception in looking up space point ptr... 0x473c7f10, 0xa0f79df0
Caught exception in looking up space point ptr... 0xa0f79df0, 0xa0fc6720
Caught exception in looking up space point ptr... 0xa0fc6720, 0x472ab360
Caught exception in looking up space point ptr... 0x1077a5f30, 0x1062b6640
Caught exception in looking up space point ptr... 0x1062b6640, 0x106aa9040
Caught exception in looking up space point ptr... 0xa873f880, 0xa87818c0
Caught exception in looking up space point ptr... 0xa87818c0, 0xa873f880
Caught exception in looking up space point ptr... 0xcf9ec0e0, 0xd3185840
Caught exception in looking up space point ptr... 0xd3185840, 0xd45e0490
Caught exception in looking up space point ptr... 0x7d37b010, 0x7d379a20
Caught exception in looking up space point ptr... 0x7d379a20, 0x7d37b010
++++>>>> total num hits: 9413094, num free: 9186261
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 149.789 357.3 2602.39 168.605 648.162 13
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 8.2134e-05 0.000137695 0.000310319 0.000101674 6.60963e-05 13
produce:tpcrawdecoder:PDVDTPCReader 18.9151 19.4727 19.8555 19.5259 0.288019 13
produce:triggerrawdecoder:PDVDTriggerReader4 0.680907 0.69729 0.72273 0.694388 0.0132268 13
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0228315 0.0238476 0.0340922 0.0230258 0.00295858 13
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0228131 0.0230445 0.0236509 0.0230209 0.000197445 13
produce:beamevent:BeamEvent 0.000127693 0.000235944 0.00120539 0.000151629 0.000280565 13
produce:pdvddaphne:DAPHNEReaderPDVD 3.04068 3.51426 3.75258 3.5181 0.207356 13
produce:ophit:OpHitFinder 0.0301108 0.0394924 0.0428728 0.0400249 0.00320909 13
produce:wclsdatavd:WireCellToolkit 110.658 117.472 122.173 118.285 3.55101 13
produce:gaushit:GausHitFinder 0.974646 2.33779 12.9562 1.45011 3.08142 13
produce:nhitsfilter:NumberOfHitsFilter 0.000203808 0.000395057 0.000698124 0.000407455 0.000128224 13
produce:reco3d:SpacePointSolver 8.02241 35.549 235.567 18.7094 58.1114 13
produce:hitpdune:DisambigFromSpacePoints 0.127586 0.736819 6.87708 0.240583 1.77356 13
produce:cluster3d:Cluster3D 1.21238 172.096 2200.46 3.4677 585.539 13
[art]:TriggerResults:TriggerResultInserter 2.7421e-05 4.87977e-05 9.9931e-05 4.0121e-05 1.88863e-05 13
end_path:out1:RootOutput 3.766e-06 9.99415e-06 3.3651e-05 5.428e-06 8.87849e-06 13
end_path:out1:RootOutput(write) 4.71515 5.73001 6.53345 5.90837 0.496785 12
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 19565.4 MB
Peak resident set size usage (VmHWM): 17036 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 06:06:54 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 12 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 13 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 