Jobsub ID 303734.0@dunegpschedd01.fnal.gov
| Jobsub ID | 303734.0@dunegpschedd01.fnal.gov |
| Workflow ID | 12718 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-04 01:26:30 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce01 |
| Last heartbeat | 2026-02-04 03:19:32 |
| From worker node | Hostname | wn2208251.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7513 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 01:27:18 |
| Input files | vd-protodune:np02vd_raw_run042422_0078_df-s04-d0_dw_0_20260203T155656.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 03:19:32 |
| Saved logs | justin-logs:303734.0-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
0x84e07fb0
Caught exception in looking up space point ptr... 0x84e07fb0, 0x5b91bbe0
Caught exception in looking up space point ptr... 0x269535d0, 0x263c2b90
Caught exception in looking up space point ptr... 0xb67c110, 0x269535d0
Caught exception in looking up space point ptr... 0xd3e69e60, 0xcdc51f70
Caught exception in looking up space point ptr... 0xcdc51f70, 0xd3e69e60
Caught exception in looking up space point ptr... 0x99e52840, 0x91e4d310
Caught exception in looking up space point ptr... 0x91e4d310, 0x919553c0
Caught exception in looking up space point ptr... 0x6a378c00, 0x6b480150
Caught exception in looking up space point ptr... 0x6b480150, 0x6a378c00
Caught exception in looking up space point ptr... 0x6a378c00, 0x703ce5f0
Caught exception in looking up space point ptr... 0x703ce5f0, 0x6a378c00
Caught exception in looking up space point ptr... 0x14b5ff20, 0x67a23f30
Caught exception in looking up space point ptr... 0x67a23f30, 0x14b5ff20
Caught exception in looking up space point ptr... 0xc2110060, 0xc210ff30
Caught exception in looking up space point ptr... 0xc210ff30, 0xbf7b72f0
Caught exception in looking up space point ptr... 0x91d0c870, 0x8bed2660
Caught exception in looking up space point ptr... 0x9c7d4040, 0x91d0c870
Caught exception in looking up space point ptr... 0x31c461a0, 0x3d113ca0
Caught exception in looking up space point ptr... 0x3d113ca0, 0x31c461a0
Caught exception in looking up space point ptr... 0x9434dcf0, 0x94362740
Caught exception in looking up space point ptr... 0x9e7851c0, 0x9434dcf0
Caught exception in looking up space point ptr... 0x5f571210, 0x3d03b6b0
Caught exception in looking up space point ptr... 0x3d03b6b0, 0x3d179ad0
Caught exception in looking up space point ptr... 0x85bfadd0, 0x867a4310
Caught exception in looking up space point ptr... 0x867a4310, 0x7e901b70
Caught exception in looking up space point ptr... 0xc2daf3f0, 0xc2d93d10
Caught exception in looking up space point ptr... 0xc2db03f0, 0xc30b7fa0
Caught exception in looking up space point ptr... 0xc30b7fa0, 0xc30b8040
Caught exception in looking up space point ptr... 0xc30b8040, 0xc2daf3f0
Caught exception in looking up space point ptr... 0x8e36b2d0, 0x8e36b040
Caught exception in looking up space point ptr... 0x8e36b040, 0x8e36b560
Caught exception in looking up space point ptr... 0x45c0c4c0, 0x45c0c330
Caught exception in looking up space point ptr... 0x45c0c330, 0x45c0c650
Caught exception in looking up space point ptr... 0xc1def300, 0xc16074f0
Caught exception in looking up space point ptr... 0xc16074f0, 0xc15994b0
Caught exception in looking up space point ptr... 0xc15994b0, 0xc1ccaf80
Caught exception in looking up space point ptr... 0xc1ccaf80, 0xc1cdb730
Caught exception in looking up space point ptr... 0xc1cdb730, 0xc1def300
Caught exception in looking up space point ptr... 0x87673a40, 0x87441a40
Caught exception in looking up space point ptr... 0x87441a40, 0x87673a40
Caught exception in looking up space point ptr... 0x87673a40, 0x7eea2370
Caught exception in looking up space point ptr... 0x7eea2370, 0x87673a40
Caught exception in looking up space point ptr... 0x87673a40, 0x86933680
Caught exception in looking up space point ptr... 0x86933680, 0x87673a40
Caught exception in looking up space point ptr... 0xa246de80, 0xa246e330
Caught exception in looking up space point ptr... 0xa246e330, 0xa246e010
Caught exception in looking up space point ptr... 0x4d85b880, 0xd7c94e0
Caught exception in looking up space point ptr... 0xd7c94e0, 0xd7c03c0
Caught exception in looking up space point ptr... 0xce9ebdf0, 0xcd587490
Caught exception in looking up space point ptr... 0xcd587490, 0xce9ebdf0
Caught exception in looking up space point ptr... 0xbf2fde90, 0xc0f880c0
Caught exception in looking up space point ptr... 0xc0f880c0, 0xbf7b5100
Caught exception in looking up space point ptr... 0x6b433880, 0x6caec2d0
Caught exception in looking up space point ptr... 0x6caec2d0, 0x706b40e0
Caught exception in looking up space point ptr... 0x411f4750, 0x57943560
Caught exception in looking up space point ptr... 0x57943560, 0x411f4750
Caught exception in looking up space point ptr... 0x42a7bc80, 0x3c1cde90
Caught exception in looking up space point ptr... 0x3c1cde90, 0x42a7bc80
Caught exception in looking up space point ptr... 0x42a7bc80, 0x3c1cdd00
Caught exception in looking up space point ptr... 0x3c1cdd00, 0x42a7bc80
Caught exception in looking up space point ptr... 0x5b2a8790, 0x691891d0
Caught exception in looking up space point ptr... 0x691891d0, 0x5b2a8790
Caught exception in looking up space point ptr... 0x1423d0d0, 0x1423ccb0
Caught exception in looking up space point ptr... 0x1423ccb0, 0x1423d0d0
Caught exception in looking up space point ptr... 0x770e1010, 0x7736ed00
Caught exception in looking up space point ptr... 0x7736ed00, 0x77015660
Caught exception in looking up space point ptr... 0x23731280, 0x41dd6d30
Caught exception in looking up space point ptr... 0x41dd6d30, 0x237310b0
Caught exception in looking up space point ptr... 0xc2bf1260, 0xc2bfbea0
Caught exception in looking up space point ptr... 0xc2bfbea0, 0xc202ad30
Caught exception in looking up space point ptr... 0xc202ad30, 0xc202f9a0
Caught exception in looking up space point ptr... 0xc202f9a0, 0xc2bf1260
Caught exception in looking up space point ptr... 0xc17c90a0, 0xc18e24c0
Caught exception in looking up space point ptr... 0xc18e24c0, 0xc17c90a0
Caught exception in looking up space point ptr... 0x2c8daeb0, 0x6b0b6140
Caught exception in looking up space point ptr... 0x6b0b6140, 0x2c8daeb0
Caught exception in looking up space point ptr... 0xc23d3640, 0xc21414d0
Caught exception in looking up space point ptr... 0xc21414d0, 0xc23d3640
Caught exception in looking up space point ptr... 0xc23d3640, 0xc23d3510
Caught exception in looking up space point ptr... 0xc23d3510, 0xc23d3640
++++>>>> total num hits: 8357906, num free: 8153446
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 240.614 437.04 2071.87 322.054 438.086 15
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 8.0762e-05 0.000144454 0.00023437 0.000144732 5.1637e-05 15
produce:tpcrawdecoder:PDVDTPCReader 132.344 190.885 224.943 193.949 28.4315 15
produce:triggerrawdecoder:PDVDTriggerReader4 0.767881 1.11669 1.46607 1.15652 0.166868 15
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0253308 0.0380568 0.065399 0.0354532 0.0111399 15
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.109346 0.119126 0.143145 0.111926 0.0108178 15
produce:beamevent:BeamEvent 0.000111149 0.000228065 0.00115904 0.000164179 0.000250015 15
produce:pdvddaphne:DAPHNEReaderPDVD 14.8486 23.0575 30.8382 25.0384 4.16391 15
produce:ophit:OpHitFinder 0.0342773 0.0439708 0.0522629 0.0451364 0.00488943 15
produce:wclsdatavd:WireCellToolkit 75.9412 82.3705 87.5683 81.6829 3.1565 15
produce:gaushit:GausHitFinder 0.745742 2.4859 20.8664 1.1329 4.91889 15
produce:nhitsfilter:NumberOfHitsFilter 0.000201829 0.00040552 0.00134234 0.000292239 0.000270737 15
produce:reco3d:SpacePointSolver 4.90436 28.1006 260.56 11.9125 62.2213 15
produce:hitpdune:DisambigFromSpacePoints 0.0584261 0.670929 8.24849 0.125652 2.02552 15
produce:cluster3d:Cluster3D 0.891592 103.492 1516.03 2.62186 377.518 15
[art]:TriggerResults:TriggerResultInserter 3.0107e-05 4.05918e-05 0.000105017 3.5507e-05 1.7932e-05 15
end_path:out1:RootOutput 4.429e-06 9.5226e-06 2.4627e-05 8.766e-06 4.61824e-06 15
end_path:out1:RootOutput(write) 4.30825 4.96327 5.53268 4.99089 0.388257 14
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 18257.1 MB
Peak resident set size usage (VmHWM): 15861.6 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 03:18:43 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 14 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 15 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 