Jobsub ID 303656.88@dunegpschedd01.fnal.gov
| Jobsub ID | 303656.88@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:56:25 |
| Site | UK_Oxford |
| Entry | DUNE_UK_SGrid_Oxford_arc01 |
| Last heartbeat | 2026-02-04 00:57:40 |
| From worker node | Hostname | t2wn068.physics.ox.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 00:13:29 |
| Input files | vd-protodune:np02vd_raw_run042421_0143_df-s04-d1_dw_0_20260203T144821.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 00:57:40 |
| Saved logs | justin-logs:303656.88-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
0xc85e8260, 0xc84368a0
Caught exception in looking up space point ptr... 0x78586fe0, 0x696f35e0
Caught exception in looking up space point ptr... 0x696f35e0, 0x7dffc8d0
Caught exception in looking up space point ptr... 0xefbd630, 0xeef9f40
Caught exception in looking up space point ptr... 0xeef9f40, 0x3905c620
Caught exception in looking up space point ptr... 0x73608e70, 0x7eec74a0
Caught exception in looking up space point ptr... 0x7eec74a0, 0x73608e70
Caught exception in looking up space point ptr... 0x81ac5a80, 0x73837db0
Caught exception in looking up space point ptr... 0x73837db0, 0x81c62250
Caught exception in looking up space point ptr... 0x50aee650, 0x444bb750
Caught exception in looking up space point ptr... 0x444bb750, 0x50aee650
Caught exception in looking up space point ptr... 0x3194ec80, 0x31a53510
Caught exception in looking up space point ptr... 0x31a53510, 0x558ab780
Caught exception in looking up space point ptr... 0x62b20150, 0x63362140
Caught exception in looking up space point ptr... 0x63362140, 0x62b20150
Caught exception in looking up space point ptr... 0x7ed44b30, 0x7ede5880
Caught exception in looking up space point ptr... 0x7ecdb830, 0x7ed44b30
Caught exception in looking up space point ptr... 0x62b9a080, 0x4558b1a0
Caught exception in looking up space point ptr... 0x4558b1a0, 0x4c598840
Caught exception in looking up space point ptr... 0xc7148500, 0xc7148320
Caught exception in looking up space point ptr... 0xc7148320, 0xc7148500
Caught exception in looking up space point ptr... 0x3e2ea760, 0x4eadb670
Caught exception in looking up space point ptr... 0x4eadb670, 0x3e2ea760
Caught exception in looking up space point ptr... 0x3e2ea760, 0x3e2ea5d0
Caught exception in looking up space point ptr... 0x3e2ea5d0, 0x3e2ea760
Caught exception in looking up space point ptr... 0x3e2ea760, 0x3e2ea440
Caught exception in looking up space point ptr... 0x3e2ea440, 0x3e2ea760
Caught exception in looking up space point ptr... 0x7d013760, 0x7d013af0
Caught exception in looking up space point ptr... 0x7d013af0, 0x7d018a30
Caught exception in looking up space point ptr... 0x14fead90, 0x84e0640
Caught exception in looking up space point ptr... 0x84e0640, 0x14fead90
Caught exception in looking up space point ptr... 0x69e39d10, 0x6f233650
Caught exception in looking up space point ptr... 0x6f233650, 0x632cac30
Caught exception in looking up space point ptr... 0xe6846d0, 0x10dc7520
Caught exception in looking up space point ptr... 0x10dc7520, 0x98f1110
Caught exception in looking up space point ptr... 0x931004c0, 0x9445ece0
Caught exception in looking up space point ptr... 0x9445ece0, 0x931004c0
Caught exception in looking up space point ptr... 0xb11a6fc0, 0xaefff830
Caught exception in looking up space point ptr... 0xaefff830, 0xbe3f3b50
Caught exception in looking up space point ptr... 0x14130840, 0x14150aa0
Caught exception in looking up space point ptr... 0x71d193b0, 0x14130840
Caught exception in looking up space point ptr... 0x3e2e5170, 0x87d96e0
Caught exception in looking up space point ptr... 0x87d96e0, 0x3e2e5170
Caught exception in looking up space point ptr... 0xcf69e810, 0xcfa697b0
Caught exception in looking up space point ptr... 0xcfa697b0, 0xcfa698a0
Caught exception in looking up space point ptr... 0xc7ad0910, 0xc7acad30
Caught exception in looking up space point ptr... 0xc7acad30, 0xc7ad0910
Caught exception in looking up space point ptr... 0xc7802a30, 0xc429c230
Caught exception in looking up space point ptr... 0xc429c230, 0xc7802a30
Caught exception in looking up space point ptr... 0x22bb3d60, 0x22bb3ef0
Caught exception in looking up space point ptr... 0x22bb3ef0, 0x22bb3d60
Caught exception in looking up space point ptr... 0x22bb3d60, 0x22bb4210
Caught exception in looking up space point ptr... 0x22bb4210, 0x22bb3d60
Caught exception in looking up space point ptr... 0xc85f1210, 0xc85f10e0
Caught exception in looking up space point ptr... 0xc85f10e0, 0xc86a74d0
Caught exception in looking up space point ptr... 0x1083c870, 0x10fcfae0
Caught exception in looking up space point ptr... 0x10fcfae0, 0x10fcfbe0
Caught exception in looking up space point ptr... 0xd9ab64e0, 0xd97fdb40
Caught exception in looking up space point ptr... 0xd97fdb40, 0xd9ab64e0
Caught exception in looking up space point ptr... 0x2c2203d0, 0x2c2200b0
Caught exception in looking up space point ptr... 0x4675d4c0, 0x2c2203d0
Caught exception in looking up space point ptr... 0xca91f270, 0xca516f60
Caught exception in looking up space point ptr... 0xca516f60, 0xca91f270
Caught exception in looking up space point ptr... 0x7e3e7600, 0x7e3e7470
Caught exception in looking up space point ptr... 0x7e3e7470, 0x7ddb0920
Caught exception in looking up space point ptr... 0x2a038f40, 0x691fd650
Caught exception in looking up space point ptr... 0x691fd650, 0x2a038f40
Caught exception in looking up space point ptr... 0x99986330, 0x920b4510
Caught exception in looking up space point ptr... 0x920b4510, 0x99986330
Caught exception in looking up space point ptr... 0x99986330, 0x9cb01b90
Caught exception in looking up space point ptr... 0x9cb01b90, 0x99986330
Caught exception in looking up space point ptr... 0x84bdcd0, 0x84bd9b0
Caught exception in looking up space point ptr... 0x84bd9b0, 0x84bdcd0
Caught exception in looking up space point ptr... 0x39d1b8f0, 0x39d1b440
Caught exception in looking up space point ptr... 0x39d1b440, 0x39d1b5d0
Caught exception in looking up space point ptr... 0x9674c8a0, 0x967bb6b0
Caught exception in looking up space point ptr... 0x967bb6b0, 0x9636c080
Caught exception in looking up space point ptr... 0xcf483330, 0xcf483240
Caught exception in looking up space point ptr... 0xcf483240, 0xcf483330
Caught exception in looking up space point ptr... 0xc5a26d30, 0xc761fec0
Caught exception in looking up space point ptr... 0xc761fec0, 0xc5a26d30
++++>>>> total num hits: 8151713, num free: 7947795
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 131.903 617.579 2024.71 156.852 812.613 4
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000119122 0.000182476 0.000327492 0.000141645 8.4448e-05 4
produce:tpcrawdecoder:PDVDTPCReader 12.1409 12.3174 12.4579 12.3353 0.117972 4
produce:triggerrawdecoder:PDVDTriggerReader4 0.584564 0.600118 0.607953 0.603977 0.00919789 4
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0195012 0.0197205 0.0201143 0.0196332 0.000233947 4
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0962783 0.0973116 0.0982859 0.0973411 0.000918863 4
produce:beamevent:BeamEvent 0.000106268 0.000422827 0.00121552 0.00018476 0.000459135 4
produce:pdvddaphne:DAPHNEReaderPDVD 3.35856 3.56166 3.70334 3.59238 0.13002 4
produce:ophit:OpHitFinder 0.0434883 0.0457539 0.049315 0.0451061 0.00228788 4
produce:wclsdatavd:WireCellToolkit 89.6396 93.4144 98.5619 92.728 3.24263 4
produce:gaushit:GausHitFinder 0.860543 2.64866 6.33659 1.69876 2.16281 4
produce:nhitsfilter:NumberOfHitsFilter 0.000348752 0.000744632 0.00155564 0.000537068 0.000484096 4
produce:reco3d:SpacePointSolver 8.21796 58.3241 199.113 12.9829 81.3647 4
produce:hitpdune:DisambigFromSpacePoints 0.116957 1.59541 5.91045 0.177127 2.49146 4
produce:cluster3d:Cluster3D 1.75864 440.743 1704.52 28.3467 729.923 4
[art]:TriggerResults:TriggerResultInserter 4.308e-05 7.52675e-05 0.000159277 4.93565e-05 4.87366e-05 4
end_path:out1:RootOutput 1.0009e-05 2.0894e-05 4.1057e-05 1.6255e-05 1.26441e-05 4
end_path:out1:RootOutput(write) 5.13042 5.56851 5.86763 5.70748 0.316599 3
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 16746.4 MB
Peak resident set size usage (VmHWM): 14256.4 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:56:23 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 4 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 