Jobsub ID 303484.0@dunegpschedd01.fnal.gov
| Jobsub ID | 303484.0@dunegpschedd01.fnal.gov |
| Workflow ID | 12689 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 15:52:05 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce05 |
| Last heartbeat | 2026-02-03 18:35:58 |
| From worker node | Hostname | dune001-1829927.0-lcg2351.gridpp.rl.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 15:54:09 |
| Input files | vd-protodune:np02vd_raw_run042421_0009_df-s02-d0_dw_0_20260203T133547.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-03 18:35:58 |
| Saved logs | justin-logs:303484.0-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ace point ptr... 0x14787d4e0, 0x1476f6050
Caught exception in looking up space point ptr... 0x1476f6050, 0x1479b8d60
Caught exception in looking up space point ptr... 0x13c34fb00, 0x10cd31330
Caught exception in looking up space point ptr... 0x10cd31330, 0x121cf6260
Caught exception in looking up space point ptr... 0x104fc5100, 0x104fc4c50
Caught exception in looking up space point ptr... 0x104fc4c50, 0x104fc5100
Caught exception in looking up space point ptr... 0x12de90cc0, 0x12de90bc0
Caught exception in looking up space point ptr... 0x12de90bc0, 0x12de90cc0
Caught exception in looking up space point ptr... 0x147886b60, 0x14776acf0
Caught exception in looking up space point ptr... 0x14776acf0, 0x1476f6640
Caught exception in looking up space point ptr... 0xc063b5b0, 0xc063b320
Caught exception in looking up space point ptr... 0xc063b320, 0xc063b840
Caught exception in looking up space point ptr... 0x15e154d70, 0x15e154680
Caught exception in looking up space point ptr... 0x15e154680, 0x15e154b20
Caught exception in looking up space point ptr... 0x104fd7530, 0x104fd73a0
Caught exception in looking up space point ptr... 0x104fd73a0, 0x104fd7530
Caught exception in looking up space point ptr... 0x104fd7530, 0x104fd7210
Caught exception in looking up space point ptr... 0x104fd7210, 0x104fd7530
Caught exception in looking up space point ptr... 0x1229f04d0, 0x13d71d730
Caught exception in looking up space point ptr... 0x13d71d730, 0x1229f04d0
Caught exception in looking up space point ptr... 0x148974670, 0x148975fe0
Caught exception in looking up space point ptr... 0x148975fe0, 0x148974670
Caught exception in looking up space point ptr... 0x15dfdf620, 0x15dfdf8b0
Caught exception in looking up space point ptr... 0x15dfdf340, 0x15dfdf620
Caught exception in looking up space point ptr... 0x126a55a30, 0x13801d980
Caught exception in looking up space point ptr... 0x13801d980, 0x126a55a30
Caught exception in looking up space point ptr... 0x12c7adce0, 0x12c7ad830
Caught exception in looking up space point ptr... 0x12c7ad830, 0x12c7adce0
Caught exception in looking up space point ptr... 0x909ce290, 0x63af4450
Caught exception in looking up space point ptr... 0x63af4450, 0x909ce290
Caught exception in looking up space point ptr... 0x10c480c30, 0x10bedb2a0
Caught exception in looking up space point ptr... 0x10bedb2a0, 0x10b69a920
Caught exception in looking up space point ptr... 0xf0c2e240, 0x13d000cf0
Caught exception in looking up space point ptr... 0x13d000cf0, 0x130a319e0
Caught exception in looking up space point ptr... 0x10c2a4180, 0x12b357500
Caught exception in looking up space point ptr... 0x12b357500, 0x10c2a4180
Caught exception in looking up space point ptr... 0x10c2a4180, 0x1217c5860
Caught exception in looking up space point ptr... 0x1217c5860, 0x10c2a4180
Caught exception in looking up space point ptr... 0x10c2a4180, 0xf1ff1850
Caught exception in looking up space point ptr... 0xf1ff1850, 0x10c2a4180
Caught exception in looking up space point ptr... 0x10e8cba90, 0xb7bd0150
Caught exception in looking up space point ptr... 0xb7bd0150, 0x149975260
Caught exception in looking up space point ptr... 0xee218040, 0xee2181d0
Caught exception in looking up space point ptr... 0xee2181d0, 0xee218360
Caught exception in looking up space point ptr... 0x12ae0d9c0, 0x12ae0d6a0
Caught exception in looking up space point ptr... 0x12ae0d6a0, 0x13820cbe0
Caught exception in looking up space point ptr... 0x13820cbe0, 0x13820ca50
Caught exception in looking up space point ptr... 0x138869040, 0x137e355f0
Caught exception in looking up space point ptr... 0x137e355f0, 0x137e35780
Caught exception in looking up space point ptr... 0xb798eb90, 0x14962de10
Caught exception in looking up space point ptr... 0x14962de10, 0x149d0d710
Caught exception in looking up space point ptr... 0x104ef1d70, 0x145e41ca0
Caught exception in looking up space point ptr... 0x145e41ca0, 0x145e413e0
Caught exception in looking up space point ptr... 0x104ef2080, 0x1446f8720
Caught exception in looking up space point ptr... 0x1446f8720, 0x104ef1d70
Caught exception in looking up space point ptr... 0x77d7d200, 0x5ed89040
Caught exception in looking up space point ptr... 0x5ed89040, 0x405e2de0
Caught exception in looking up space point ptr... 0x55824450, 0x55824770
Caught exception in looking up space point ptr... 0x55824770, 0x55824450
Caught exception in looking up space point ptr... 0xf26eea00, 0x5788af90
Caught exception in looking up space point ptr... 0x4202b330, 0xf26eea00
Caught exception in looking up space point ptr... 0xee419d20, 0xee419a00
Caught exception in looking up space point ptr... 0xee419a00, 0xee419d20
Caught exception in looking up space point ptr... 0x106b785d0, 0x13ac40f50
Caught exception in looking up space point ptr... 0x13ac40f50, 0x106b785d0
Caught exception in looking up space point ptr... 0xc1a7b800, 0x14e145ff0
Caught exception in looking up space point ptr... 0x14e145ff0, 0xfd825b00
Caught exception in looking up space point ptr... 0x114fc8730, 0x114fc85a0
Caught exception in looking up space point ptr... 0x114fc85a0, 0x114fc8730
Caught exception in looking up space point ptr... 0x755cae40, 0x6c2d0f50
Caught exception in looking up space point ptr... 0x6c2d0f50, 0x7944c130
Caught exception in looking up space point ptr... 0x12c75ce10, 0x12c75cfa0
Caught exception in looking up space point ptr... 0x12c75cfa0, 0x12c75d130
Caught exception in looking up space point ptr... 0x14d896450, 0x14d5fc7c0
Caught exception in looking up space point ptr... 0x14d5fc7c0, 0x14d896450
Caught exception in looking up space point ptr... 0x10c94b560, 0x10c94b0b0
Caught exception in looking up space point ptr... 0x10c94b0b0, 0x10c94b560
Caught exception in looking up space point ptr... 0x14d0ccbf0, 0x147915a00
Caught exception in looking up space point ptr... 0x147915a00, 0x14b6ae380
++++>>>> total num hits: 7865835, num free: 7666100
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 207.295 367.482 2180.69 269.261 409.962 26
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.8719e-05 8.86759e-05 0.000221924 7.59845e-05 3.81216e-05 26
produce:tpcrawdecoder:PDVDTPCReader 103.572 137.735 177.995 135.341 21.0616 26
produce:triggerrawdecoder:PDVDTriggerReader4 0.554719 0.839456 1.38101 0.785887 0.241151 26
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0178319 0.0216814 0.0349477 0.0185664 0.00478624 26
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0876964 0.100836 0.141847 0.0885965 0.0202059 26
produce:beamevent:BeamEvent 7.9754e-05 0.000150804 0.000910715 0.000115653 0.000154612 26
produce:pdvddaphne:DAPHNEReaderPDVD 14.3525 18.5387 23.1886 18.4501 2.22599 26
produce:ophit:OpHitFinder 0.0288244 0.0390932 0.0624363 0.0390528 0.00583414 26
produce:wclsdatavd:WireCellToolkit 65.717 79.2646 132.374 76.5191 13.6172 26
produce:gaushit:GausHitFinder 0.672504 2.04621 16.1065 1.26034 3.07645 26
produce:nhitsfilter:NumberOfHitsFilter 0.000292757 0.000598111 0.00246392 0.00045899 0.000525581 26
produce:reco3d:SpacePointSolver 6.00686 24.7983 209.192 13.2714 42.3571 26
produce:hitpdune:DisambigFromSpacePoints 0.0808904 1.23653 14.9224 0.149598 3.73973 26
produce:cluster3d:Cluster3D 0.962262 98.4334 1637.83 2.07466 346.511 26
[art]:TriggerResults:TriggerResultInserter 2.195e-05 3.40843e-05 9.0775e-05 2.85875e-05 1.57052e-05 26
end_path:out1:RootOutput 3.568e-06 7.847e-06 2.3922e-05 7.212e-06 4.20309e-06 26
end_path:out1:RootOutput(write) 3.16082 4.57032 22.5004 3.76705 3.67906 25
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 19350 MB
Peak resident set size usage (VmHWM): 16582 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 03-Feb-2026 18:35:10 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 25 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 26 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 