Jobsub ID 303718.6@dunegpschedd01.fnal.gov
| Jobsub ID | 303718.6@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-04 00:52:28 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce01 |
| Last heartbeat | 2026-02-04 01:22:31 |
| From worker node | Hostname | wn1904250.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7513 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 00:53:06 |
| Input files | vd-protodune:np02vd_raw_run042421_0010_df-s04-d1_dw_0_20260203T133622.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:22:31 |
| Saved logs | justin-logs:303718.6-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
up space point ptr... 0x965d9130, 0x982fe940
Caught exception in looking up space point ptr... 0x982fe940, 0x982fead0
Caught exception in looking up space point ptr... 0xc4906720, 0xc3d71e00
Caught exception in looking up space point ptr... 0xc3d71e00, 0xc4906720
Caught exception in looking up space point ptr... 0xc4906720, 0xc3d7c1e0
Caught exception in looking up space point ptr... 0xc3d7c1e0, 0xc4906720
Caught exception in looking up space point ptr... 0x481a15c0, 0x481a16c0
Caught exception in looking up space point ptr... 0x481a16c0, 0x481a15c0
Caught exception in looking up space point ptr... 0xc55af030, 0xc55aef40
Caught exception in looking up space point ptr... 0xc55aef40, 0xc55af030
Caught exception in looking up space point ptr... 0xaa21b930, 0xaa21bdd0
Caught exception in looking up space point ptr... 0xaa21bdd0, 0xaa229520
Caught exception in looking up space point ptr... 0x20e6e880, 0x20e6e5f0
Caught exception in looking up space point ptr... 0x20e6e5f0, 0x20e6e360
Caught exception in looking up space point ptr... 0xa8425950, 0xa8421f00
Caught exception in looking up space point ptr... 0xa8377410, 0xa8425950
Caught exception in looking up space point ptr... 0x25935230, 0x2305bc10
Caught exception in looking up space point ptr... 0x2305bc10, 0x52773360
Caught exception in looking up space point ptr... 0x820f79a0, 0x820acfe0
Caught exception in looking up space point ptr... 0x828a6990, 0x820f79a0
Caught exception in looking up space point ptr... 0x5f8eb590, 0x5f8eb720
Caught exception in looking up space point ptr... 0x5f8eb720, 0x5f8eb400
Caught exception in looking up space point ptr... 0xacfdd780, 0xacf24e70
Caught exception in looking up space point ptr... 0xacf24e70, 0xacf61f30
Caught exception in looking up space point ptr... 0x210276c0, 0x192bd8a0
Caught exception in looking up space point ptr... 0x192bd8a0, 0x210276c0
Caught exception in looking up space point ptr... 0x210276c0, 0x2008a370
Caught exception in looking up space point ptr... 0x2008a370, 0x210276c0
Caught exception in looking up space point ptr... 0xac2f05e0, 0xac2f04f0
Caught exception in looking up space point ptr... 0xac2f04f0, 0xabe66910
Caught exception in looking up space point ptr... 0xac2f09a0, 0xac2f05e0
Caught exception in looking up space point ptr... 0xacf12d60, 0xacf12b80
Caught exception in looking up space point ptr... 0xacf12b80, 0xacf12d60
Caught exception in looking up space point ptr... 0xc22571c0, 0xc234e340
Caught exception in looking up space point ptr... 0xc234e340, 0xc321b240
Caught exception in looking up space point ptr... 0x81eb4070, 0x81ece100
Caught exception in looking up space point ptr... 0x81ece100, 0x81eb4070
Caught exception in looking up space point ptr... 0x2af91290, 0x2af91000
Caught exception in looking up space point ptr... 0x2af90a90, 0x2af91290
Caught exception in looking up space point ptr... 0xba5f99e0, 0xba9b0e30
Caught exception in looking up space point ptr... 0xba9b0e30, 0xba5f9b20
Caught exception in looking up space point ptr... 0x210b8430, 0x210586f0
Caught exception in looking up space point ptr... 0x210c7000, 0x210b8430
Caught exception in looking up space point ptr... 0x1f5f1610, 0x1f60b2c0
Caught exception in looking up space point ptr... 0x1f60b2c0, 0x1f60b070
Caught exception in looking up space point ptr... 0x582566f0, 0x58256980
Caught exception in looking up space point ptr... 0x58256980, 0x58256460
Caught exception in looking up space point ptr... 0x7faec4f0, 0x7fa5ea60
Caught exception in looking up space point ptr... 0x7fa5ea60, 0x7faec590
Caught exception in looking up space point ptr... 0xc7c54bb0, 0xc7c411b0
Caught exception in looking up space point ptr... 0xc7c411b0, 0xc7c54bb0
Caught exception in looking up space point ptr... 0x7216f7e0, 0x72134c80
Caught exception in looking up space point ptr... 0x72134c80, 0x721349f0
Caught exception in looking up space point ptr... 0xbc0a5150, 0xbc0a4ec0
Caught exception in looking up space point ptr... 0xbc0a4ec0, 0xbc0a53e0
Caught exception in looking up space point ptr... 0x2770af00, 0x27707650
Caught exception in looking up space point ptr... 0x27707650, 0x2770af00
Caught exception in looking up space point ptr... 0x2770af00, 0x276facc0
Caught exception in looking up space point ptr... 0x276facc0, 0x2770af00
Caught exception in looking up space point ptr... 0xbdbd57e0, 0xbdbd5fe0
Caught exception in looking up space point ptr... 0xbdbd5fe0, 0xbdbd5ac0
Caught exception in looking up space point ptr... 0x6975e080, 0x6ad24380
Caught exception in looking up space point ptr... 0x698f8780, 0x6975e080
Caught exception in looking up space point ptr... 0x7dbe35f0, 0x7dbae530
Caught exception in looking up space point ptr... 0x7dbae800, 0x7dbe35f0
Caught exception in looking up space point ptr... 0xa818a740, 0xa8206140
Caught exception in looking up space point ptr... 0xa8206140, 0xa818a740
Caught exception in looking up space point ptr... 0x8ad46d90, 0x7fbdbc50
Caught exception in looking up space point ptr... 0x7fbdbc50, 0x8ad46d90
Caught exception in looking up space point ptr... 0x4326df20, 0x40df5f70
Caught exception in looking up space point ptr... 0x40df5f70, 0x4326df20
Caught exception in looking up space point ptr... 0x17812740, 0x40b40470
Caught exception in looking up space point ptr... 0x40b40470, 0x17812740
Caught exception in looking up space point ptr... 0x4273e7c0, 0x3cd973a0
Caught exception in looking up space point ptr... 0x3cd973a0, 0x438c2aa0
Caught exception in looking up space point ptr... 0xb4f289a0, 0xb451d380
Caught exception in looking up space point ptr... 0xb451d560, 0xb4f289a0
Caught exception in looking up space point ptr... 0xa301c800, 0x96b7e450
Caught exception in looking up space point ptr... 0x96b7e450, 0x9a26a3e0
Caught exception in looking up space point ptr... 0xae226340, 0xae226520
Caught exception in looking up space point ptr... 0xae226520, 0xae226430
++++>>>> total num hits: 8426156, num free: 8211965
04-Feb-2026 01:21:32 GMT Opened output file with pattern "%ifb_reco_stage0_%tc_offline.root"
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 1633.69 1633.69 1633.69 1633.69 0 1
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.00027162 0.00027162 0.00027162 0.00027162 0 1
produce:tpcrawdecoder:PDVDTPCReader 17.6996 17.6996 17.6996 17.6996 0 1
produce:triggerrawdecoder:PDVDTriggerReader4 0.658764 0.658764 0.658764 0.658764 0 1
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0225276 0.0225276 0.0225276 0.0225276 0 1
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.11041 0.11041 0.11041 0.11041 0 1
produce:beamevent:BeamEvent 0.00115212 0.00115212 0.00115212 0.00115212 0 1
produce:pdvddaphne:DAPHNEReaderPDVD 3.96229 3.96229 3.96229 3.96229 0 1
produce:ophit:OpHitFinder 0.0448048 0.0448048 0.0448048 0.0448048 0 1
produce:wclsdatavd:WireCellToolkit 76.5568 76.5568 76.5568 76.5568 0 1
produce:gaushit:GausHitFinder 8.91861 8.91861 8.91861 8.91861 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.00128754 0.00128754 0.00128754 0.00128754 0 1
produce:reco3d:SpacePointSolver 178.589 178.589 178.589 178.589 0 1
produce:hitpdune:DisambigFromSpacePoints 8.09074 8.09074 8.09074 8.09074 0 1
produce:cluster3d:Cluster3D 1339.01 1339.01 1339.01 1339.01 0 1
[art]:TriggerResults:TriggerResultInserter 8.6733e-05 8.6733e-05 8.6733e-05 8.6733e-05 0 1
end_path:out1:RootOutput 2.2973e-05 2.2973e-05 2.2973e-05 2.2973e-05 0 1
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 17868.9 MB
Peak resident set size usage (VmHWM): 15327.4 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 01:21:55 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 1 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 