Jobsub ID 303650.112@dunegpschedd01.fnal.gov
| Jobsub ID | 303650.112@dunegpschedd01.fnal.gov |
| Workflow ID | 12719 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:48:28 |
| Site | UK_QMUL |
| Entry | DUNE_UK_London_QMUL_arcce02 |
| Last heartbeat | 2026-02-04 02:05:55 |
| From worker node | Hostname | cn064.htc.esc.qmul |
| cpuinfo | Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:15:20 |
| Input files | vd-protodune:np02vd_raw_run042433_0073_df-s02-d1_dw_0_20260203T190952.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 02:05:55 |
| Saved logs | justin-logs:303650.112-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ptr... 0x14e9742d0, 0x14e974520
Caught exception in looking up space point ptr... 0x14e974520, 0x14eb69cb0
Caught exception in looking up space point ptr... 0x44aa84d0, 0x139aff670
Caught exception in looking up space point ptr... 0x139aff670, 0x82c1a880
Caught exception in looking up space point ptr... 0x147fedd40, 0x147e3eaa0
Caught exception in looking up space point ptr... 0x148a2ad20, 0x147fedd40
Caught exception in looking up space point ptr... 0xff8cb650, 0xff8cb160
Caught exception in looking up space point ptr... 0xff8cb160, 0xff8cb3b0
Caught exception in looking up space point ptr... 0x101e76ab0, 0x101e76d40
Caught exception in looking up space point ptr... 0x101e76d40, 0x101e767d0
Caught exception in looking up space point ptr... 0x101eb9940, 0x101eb9eb0
Caught exception in looking up space point ptr... 0x101eb9eb0, 0x101eb9c20
Caught exception in looking up space point ptr... 0x52fd0950, 0x52fc59b0
Caught exception in looking up space point ptr... 0x52fc59b0, 0x52fd0a90
Caught exception in looking up space point ptr... 0x101f23fc0, 0x101f23d30
Caught exception in looking up space point ptr... 0x101f24250, 0x101f23fc0
Caught exception in looking up space point ptr... 0x147d46060, 0x147d462f0
Caught exception in looking up space point ptr... 0x147d46580, 0x147d46060
Caught exception in looking up space point ptr... 0x147ebd340, 0xfc96cb60
Caught exception in looking up space point ptr... 0x147cfaca0, 0x147ebd340
Caught exception in looking up space point ptr... 0x5e87b5c0, 0x5e87b6f0
Caught exception in looking up space point ptr... 0x5e87b6f0, 0x5e87b820
Caught exception in looking up space point ptr... 0x3d297590, 0xd41b4ae0
Caught exception in looking up space point ptr... 0xd41b4ae0, 0xef89a2e0
Caught exception in looking up space point ptr... 0xd3c2c460, 0xd4029b00
Caught exception in looking up space point ptr... 0xd4029b00, 0xd3c2c460
Caught exception in looking up space point ptr... 0xd3c2c460, 0xd3fc4d70
Caught exception in looking up space point ptr... 0xd3fc4d70, 0xd3c2c460
Caught exception in looking up space point ptr... 0xff8cddf0, 0xff8ce2e0
Caught exception in looking up space point ptr... 0xff8ce2e0, 0xff8ce090
Caught exception in looking up space point ptr... 0x1020bac90, 0x101ca1ca0
Caught exception in looking up space point ptr... 0x101ca1ca0, 0x101d6d9d0
Caught exception in looking up space point ptr... 0x14d0b70c0, 0x88adaaa0
Caught exception in looking up space point ptr... 0x88adaaa0, 0x13060e5a0
Caught exception in looking up space point ptr... 0x7d062320, 0x1576ea770
Caught exception in looking up space point ptr... 0x1576ea770, 0x7d062320
Caught exception in looking up space point ptr... 0x101ea6690, 0x101ea6920
Caught exception in looking up space point ptr... 0x101ea6920, 0x101ea63b0
Caught exception in looking up space point ptr... 0x13cf9b3c0, 0x13cf9b650
Caught exception in looking up space point ptr... 0x13cf9b650, 0x13cf9b0e0
Caught exception in looking up space point ptr... 0x4746e0f0, 0x4746de60
Caught exception in looking up space point ptr... 0x4746e380, 0x4746e0f0
Caught exception in looking up space point ptr... 0xf0682a60, 0xf0682f80
Caught exception in looking up space point ptr... 0xf0682cf0, 0xf0682a60
Caught exception in looking up space point ptr... 0xfebf650, 0xfebf970
Caught exception in looking up space point ptr... 0xfebf970, 0xfebf650
Caught exception in looking up space point ptr... 0x6a38f6b0, 0x6a3f3140
Caught exception in looking up space point ptr... 0x6a3f3140, 0x6a3aaf10
Caught exception in looking up space point ptr... 0x157cb750, 0xee6ea180
Caught exception in looking up space point ptr... 0xee6ea180, 0x157cb750
Caught exception in looking up space point ptr... 0xf084db20, 0xf0c2e620
Caught exception in looking up space point ptr... 0xf0a65bc0, 0xf084db20
Caught exception in looking up space point ptr... 0x2ca7b3d0, 0x2c93d400
Caught exception in looking up space point ptr... 0x2c92be30, 0x2ca7b3d0
Caught exception in looking up space point ptr... 0x96cb7b20, 0x96ca97c0
Caught exception in looking up space point ptr... 0x96ca95a0, 0x96cb7b20
Caught exception in looking up space point ptr... 0x1148a590, 0x1148a400
Caught exception in looking up space point ptr... 0x1148a400, 0x1148a590
Caught exception in looking up space point ptr... 0x1148a590, 0x6e75d760
Caught exception in looking up space point ptr... 0x6e75d760, 0x1148a590
Caught exception in looking up space point ptr... 0x1148a590, 0x6e75da80
Caught exception in looking up space point ptr... 0x6e75da80, 0x1148a590
Caught exception in looking up space point ptr... 0x6fc494d0, 0x6fc499f0
Caught exception in looking up space point ptr... 0x6fc49760, 0x6fc494d0
Caught exception in looking up space point ptr... 0x7d82f1f0, 0x3b77c120
Caught exception in looking up space point ptr... 0x3b77c120, 0x7d82f1f0
Caught exception in looking up space point ptr... 0xfe551c10, 0x165ad39c0
Caught exception in looking up space point ptr... 0x165ad39c0, 0xfe551c10
Caught exception in looking up space point ptr... 0x437bb710, 0x4c41ddc0
Caught exception in looking up space point ptr... 0x4c41ddc0, 0x12dd75990
Caught exception in looking up space point ptr... 0x114dc70c0, 0x14d2b2a30
Caught exception in looking up space point ptr... 0x14d2b2a30, 0x14c5eda70
Caught exception in looking up space point ptr... 0x87f65010, 0x87f65330
Caught exception in looking up space point ptr... 0x87f65330, 0x87f651a0
Caught exception in looking up space point ptr... 0x1660928b0, 0x95cd9110
Caught exception in looking up space point ptr... 0x956016b0, 0x1660928b0
Caught exception in looking up space point ptr... 0x66748150, 0x3dd5cea0
Caught exception in looking up space point ptr... 0x3dd5cea0, 0x66748150
Caught exception in looking up space point ptr... 0x39338300, 0xf054ec70
Caught exception in looking up space point ptr... 0xf054ec70, 0x39338300
++++>>>> total num hits: 6897573, num free: 6712173
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 351.02 526.31 1770.66 414.268 365.009 19
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.2808e-05 0.000128276 0.00024732 0.000116285 3.68235e-05 19
produce:tpcrawdecoder:PDVDTPCReader 98.5095 152.853 240.624 137.929 38.4986 19
produce:triggerrawdecoder:PDVDTriggerReader4 0.56882 0.793436 1.17284 0.746485 0.158435 19
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0165316 0.0236182 0.0709502 0.0173533 0.0141265 19
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0165884 0.01766 0.032916 0.0167352 0.00360968 19
produce:beamevent:BeamEvent 0.00013417 0.000213532 0.0012143 0.00015395 0.00023675 19
produce:pdvddaphne:DAPHNEReaderPDVD 12.7002 17.5512 23.3876 17.0381 2.6072 19
produce:ophit:OpHitFinder 0.0344007 0.0650544 0.0818703 0.0653895 0.00919137 19
produce:wclsdatavd:WireCellToolkit 140.365 193.39 211.545 197.217 15.6054 19
produce:gaushit:GausHitFinder 1.38047 3.22668 18.2176 2.22886 3.708 19
produce:nhitsfilter:NumberOfHitsFilter 0.000561843 0.000979742 0.00283863 0.000754388 0.000603783 19
produce:reco3d:SpacePointSolver 12.7735 35.3816 168.369 21.8306 40.0171 19
produce:hitpdune:DisambigFromSpacePoints 0.19472 1.4684 12.0439 0.368853 3.24952 19
produce:cluster3d:Cluster3D 1.79006 113.632 1289.32 3.43517 329.939 19
[art]:TriggerResults:TriggerResultInserter 2.8955e-05 4.29145e-05 0.000106668 3.9826e-05 1.56499e-05 19
end_path:out1:RootOutput 5.164e-06 1.07597e-05 3.3758e-05 9.946e-06 6.00415e-06 19
end_path:out1:RootOutput(write) 5.29253 8.2877 26.8239 7.53071 4.57124 18
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 17597.4 MB
Peak resident set size usage (VmHWM): 14945.4 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 02:04:38 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 18 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 19 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 