Jobsub ID 303650.93@dunegpschedd01.fnal.gov
| Jobsub ID | 303650.93@dunegpschedd01.fnal.gov |
| Workflow ID | 12719 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:48:28 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce02 |
| Last heartbeat | 2026-02-04 01:07:24 |
| From worker node | Hostname | dune002-1602366.0-lcg2594.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:15:44 |
| Input files | vd-protodune:np02vd_raw_run042433_0126_df-s03-d1_dw_0_20260203T193001.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:07:24 |
| Saved logs | justin-logs:303650.93-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
2e95380, 0x1539d0180
Caught exception in looking up space point ptr... 0x1539d0180, 0x18d0c96d0
Caught exception in looking up space point ptr... 0x10271ae70, 0x189467650
Caught exception in looking up space point ptr... 0x189467650, 0x10271ae70
Caught exception in looking up space point ptr... 0x5ee42e10, 0x5ee42c50
Caught exception in looking up space point ptr... 0x5ee42c50, 0x5ee42a90
Caught exception in looking up space point ptr... 0x8fc1f270, 0x9279c870
Caught exception in looking up space point ptr... 0x9279c870, 0x8fc1f270
Caught exception in looking up space point ptr... 0x11bb5a630, 0x5b73fa40
Caught exception in looking up space point ptr... 0x5b70ecb0, 0x11bb5a630
Caught exception in looking up space point ptr... 0x612f90f0, 0x12423f210
Caught exception in looking up space point ptr... 0x12423f210, 0x60787da0
Caught exception in looking up space point ptr... 0x60787da0, 0x1912777d0
Caught exception in looking up space point ptr... 0x1912777d0, 0x12423f210
Caught exception in looking up space point ptr... 0x12423f210, 0x1245f25f0
Caught exception in looking up space point ptr... 0x1245f25f0, 0x612f90f0
Caught exception in looking up space point ptr... 0x6d38f050, 0x6d38edf0
Caught exception in looking up space point ptr... 0x6d38edf0, 0x6d38f050
Caught exception in looking up space point ptr... 0x529e9730, 0x31ae6fa0
Caught exception in looking up space point ptr... 0x31ae6fa0, 0x88449c50
Caught exception in looking up space point ptr... 0x7e27afe0, 0x7e27b350
Caught exception in looking up space point ptr... 0x7e27b350, 0x7e27afe0
Caught exception in looking up space point ptr... 0x7a7324d0, 0x7a732a70
Caught exception in looking up space point ptr... 0x7a732a70, 0x7a732d00
Caught exception in looking up space point ptr... 0x6c8a6700, 0x6c42bef0
Caught exception in looking up space point ptr... 0x6c42bef0, 0x6c8a6700
Caught exception in looking up space point ptr... 0xc4233c60, 0x465829c0
Caught exception in looking up space point ptr... 0x465829c0, 0x460edbf0
Caught exception in looking up space point ptr... 0x19f558d20, 0x19f5591c0
Caught exception in looking up space point ptr... 0x19f5591c0, 0x19f558f70
Caught exception in looking up space point ptr... 0x32bcfe50, 0x32bcfd50
Caught exception in looking up space point ptr... 0x32bcfd50, 0x32bcfe50
Caught exception in looking up space point ptr... 0x32bcfe50, 0x30f74ab0
Caught exception in looking up space point ptr... 0x30f74ab0, 0x32bcfe50
Caught exception in looking up space point ptr... 0x1416d3140, 0x1416d33d0
Caught exception in looking up space point ptr... 0x1416d33d0, 0x1416d2eb0
Caught exception in looking up space point ptr... 0x18afcbd40, 0x18afaa7b0
Caught exception in looking up space point ptr... 0x18afca1f0, 0x18afcbd40
Caught exception in looking up space point ptr... 0x18e8d1d20, 0x11cb28de0
Caught exception in looking up space point ptr... 0x11cb28de0, 0x11e151740
Caught exception in looking up space point ptr... 0x6ccd5d00, 0x18e8d1d20
Caught exception in looking up space point ptr... 0x62c59010, 0x18ed6f8f0
Caught exception in looking up space point ptr... 0x18ed6f8f0, 0x48638f00
Caught exception in looking up space point ptr... 0x12a029230, 0x168579020
Caught exception in looking up space point ptr... 0x168579020, 0x12a029230
Caught exception in looking up space point ptr... 0x2fc8ecb0, 0x2fc9bb40
Caught exception in looking up space point ptr... 0x2fc9bb40, 0x2fc9c820
Caught exception in looking up space point ptr... 0x19f523820, 0x19f523380
Caught exception in looking up space point ptr... 0x19f523130, 0x19f523820
Caught exception in looking up space point ptr... 0x329a9760, 0x329a9c80
Caught exception in looking up space point ptr... 0x329a9c80, 0x329a99f0
Caught exception in looking up space point ptr... 0x5ba08140, 0x5ba07eb0
Caught exception in looking up space point ptr... 0x5ba07990, 0x5ba08140
Caught exception in looking up space point ptr... 0x179b29be0, 0x5b53cea0
Caught exception in looking up space point ptr... 0x5b53cea0, 0x11c6a4510
Caught exception in looking up space point ptr... 0x8c3baee0, 0x1577e9a00
Caught exception in looking up space point ptr... 0x1577e9a00, 0x8c3baee0
Caught exception in looking up space point ptr... 0x95460d30, 0x95461050
Caught exception in looking up space point ptr... 0x95461050, 0x95460d30
Caught exception in looking up space point ptr... 0x7a5cbd90, 0x7a558a70
Caught exception in looking up space point ptr... 0x7a558a70, 0x7a59ed90
Caught exception in looking up space point ptr... 0x3232fe90, 0x3232b090
Caught exception in looking up space point ptr... 0x3232b090, 0x323280c0
Caught exception in looking up space point ptr... 0x1882e6a70, 0x11ecd6530
Caught exception in looking up space point ptr... 0x11ecd6530, 0x7b7c01e0
Caught exception in looking up space point ptr... 0x7b7c01e0, 0x1a2701b80
Caught exception in looking up space point ptr... 0x7ea2bce0, 0x1891ceec0
Caught exception in looking up space point ptr... 0x1891ceec0, 0x7ea2bce0
Caught exception in looking up space point ptr... 0x7ea2bce0, 0x29eead00
Caught exception in looking up space point ptr... 0x29eead00, 0x7ea2bce0
Caught exception in looking up space point ptr... 0x58d790c0, 0x188acd9a0
Caught exception in looking up space point ptr... 0x188acd9a0, 0x17bfea870
Caught exception in looking up space point ptr... 0x7db95cd0, 0x7db95e20
Caught exception in looking up space point ptr... 0x7db95e20, 0x7db95cd0
Caught exception in looking up space point ptr... 0x1904fa640, 0x1904fa960
Caught exception in looking up space point ptr... 0x1904fa960, 0x1904fa4b0
Caught exception in looking up space point ptr... 0x180d4ae70, 0x5fbc2060
Caught exception in looking up space point ptr... 0x5fbc2060, 0x180d4ae70
Caught exception in looking up space point ptr... 0x5bf92500, 0x5bf904f0
Caught exception in looking up space point ptr... 0x5bf904f0, 0x5bf92500
++++>>>> total num hits: 7666557, num free: 7460221
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 214.84 719.844 2817.75 256.033 910.842 9
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000113971 0.000216121 0.000357676 0.000224033 8.22801e-05 9
produce:tpcrawdecoder:PDVDTPCReader 18.3012 18.6773 18.8841 18.717 0.175754 9
produce:triggerrawdecoder:PDVDTriggerReader4 0.698622 0.734891 0.790992 0.734766 0.0277119 9
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0232663 0.0235426 0.0241452 0.0234834 0.000229275 9
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0232331 0.0235184 0.0244215 0.0234446 0.000334831 9
produce:beamevent:BeamEvent 0.000219383 0.000453891 0.00191771 0.000263263 0.000518639 9
produce:pdvddaphne:DAPHNEReaderPDVD 3.72903 4.37675 4.75467 4.38306 0.303381 9
produce:ophit:OpHitFinder 0.0531516 0.0605848 0.0685275 0.0600044 0.00447572 9
produce:wclsdatavd:WireCellToolkit 149.413 178.277 206.962 177.959 16.5413 9
produce:gaushit:GausHitFinder 1.7129 5.37319 19.069 2.2845 5.94683 9
produce:nhitsfilter:NumberOfHitsFilter 0.000663639 0.00110436 0.00271035 0.00074373 0.00066821 9
produce:reco3d:SpacePointSolver 26.6431 84.895 316.679 37.2089 98.9375 9
produce:hitpdune:DisambigFromSpacePoints 0.220532 2.62141 10.9799 0.426429 4.25967 9
produce:cluster3d:Cluster3D 2.45065 414.722 2246.03 4.75162 788.607 9
[art]:TriggerResults:TriggerResultInserter 3.3121e-05 0.000141906 0.000620868 5.9791e-05 0.000179779 9
end_path:out1:RootOutput 1.0329e-05 0.000178567 0.00140713 2.291e-05 0.000434718 9
end_path:out1:RootOutput(write) 6.24691 11.1954 33.5807 8.07541 8.50288 8
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 19657.3 MB
Peak resident set size usage (VmHWM): 16930.7 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 01:06:15 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 8 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 9 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 