Jobsub ID 297917.153@dunegpschedd02.fnal.gov
| Jobsub ID | 297917.153@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:48:25 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce02 |
| Last heartbeat | 2026-02-04 01:28:51 |
| From worker node | Hostname | wn5916091.tier2.hep.manchester.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:24:24 |
| Input files | vd-protodune:np02vd_raw_run042421_0009_df-s03-d0_dw_0_20260203T133549.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:28:51 |
| Saved logs | justin-logs:297917.153-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
0, 0x4eaca1b0
Caught exception in looking up space point ptr... 0x60bd3490, 0x45a8fba0
Caught exception in looking up space point ptr... 0x45a8fba0, 0x60bd3490
Caught exception in looking up space point ptr... 0x2d2a48a0, 0x2d257b40
Caught exception in looking up space point ptr... 0x2d257b40, 0x2d2a48a0
Caught exception in looking up space point ptr... 0x2d2a48a0, 0x586d2570
Caught exception in looking up space point ptr... 0x586d2570, 0x2d2a48a0
Caught exception in looking up space point ptr... 0x2d2a48a0, 0xe0c5c20
Caught exception in looking up space point ptr... 0xe0c5c20, 0x2d2a48a0
Caught exception in looking up space point ptr... 0xac4fcaf0, 0xac77eeb0
Caught exception in looking up space point ptr... 0xac77eeb0, 0xac4fcaf0
Caught exception in looking up space point ptr... 0x58f13230, 0x21010ff0
Caught exception in looking up space point ptr... 0x21010ff0, 0x58f13230
Caught exception in looking up space point ptr... 0xaf7cb400, 0xaeb98110
Caught exception in looking up space point ptr... 0xaeb98110, 0xaf7cb400
Caught exception in looking up space point ptr... 0xf54bbf0, 0xf537ee0
Caught exception in looking up space point ptr... 0xf537ee0, 0x15f756f0
Caught exception in looking up space point ptr... 0x5663d7c0, 0x5c025830
Caught exception in looking up space point ptr... 0x5c025830, 0xf54bbf0
Caught exception in looking up space point ptr... 0xaf2c9cf0, 0xaf2c9a90
Caught exception in looking up space point ptr... 0xaf2c9a90, 0xaf2c9cf0
Caught exception in looking up space point ptr... 0x70165240, 0x6f375930
Caught exception in looking up space point ptr... 0x6f375930, 0x70165240
Caught exception in looking up space point ptr... 0x70165240, 0x6f3757a0
Caught exception in looking up space point ptr... 0x6f3757a0, 0x70165240
Caught exception in looking up space point ptr... 0xc8de4a50, 0xc9f05fc0
Caught exception in looking up space point ptr... 0xc9f05fc0, 0xc8d011b0
Caught exception in looking up space point ptr... 0x74920b80, 0x7580cac0
Caught exception in looking up space point ptr... 0x7580cac0, 0x6d749280
Caught exception in looking up space point ptr... 0xae88df40, 0xaf668d40
Caught exception in looking up space point ptr... 0xaf668d40, 0xaf668e70
Caught exception in looking up space point ptr... 0xa7cdf1a0, 0xa4a3aec0
Caught exception in looking up space point ptr... 0xa4a3aec0, 0xa5148220
Caught exception in looking up space point ptr... 0x2c0ecf40, 0x2c0ecfe0
Caught exception in looking up space point ptr... 0x2c0ecfe0, 0x2c0f0640
Caught exception in looking up space point ptr... 0x41b4ea60, 0x2df23440
Caught exception in looking up space point ptr... 0x2df23440, 0x41b4ea60
Caught exception in looking up space point ptr... 0xbe4177d0, 0xb5787b00
Caught exception in looking up space point ptr... 0xb5787b00, 0xbe4177d0
Caught exception in looking up space point ptr... 0xbe4177d0, 0xbe6a6730
Caught exception in looking up space point ptr... 0xbe6a6730, 0xbe4177d0
Caught exception in looking up space point ptr... 0xbe4177d0, 0xbe475cf0
Caught exception in looking up space point ptr... 0xbe475cf0, 0xbe4177d0
Caught exception in looking up space point ptr... 0xa3fb7410, 0xa33d0d00
Caught exception in looking up space point ptr... 0xa33d0d00, 0x9ce56090
Caught exception in looking up space point ptr... 0xcf1ec330, 0xce879d40
Caught exception in looking up space point ptr... 0xcf32e7d0, 0xcf1ec330
Caught exception in looking up space point ptr... 0xa5cb7ac0, 0xa4793450
Caught exception in looking up space point ptr... 0xa4793450, 0xa4082660
Caught exception in looking up space point ptr... 0xb36d30e0, 0xc1451d70
Caught exception in looking up space point ptr... 0xb313da50, 0xb36d30e0
Caught exception in looking up space point ptr... 0xcab42270, 0xcb5cd1c0
Caught exception in looking up space point ptr... 0xcb5cd1c0, 0xcab42270
Caught exception in looking up space point ptr... 0x41a61570, 0x4e74b140
Caught exception in looking up space point ptr... 0x4e74b140, 0x487e2b70
Caught exception in looking up space point ptr... 0x396fa010, 0x3f968a20
Caught exception in looking up space point ptr... 0x3f968a20, 0x35b79340
Caught exception in looking up space point ptr... 0x105a8470, 0x2d4a3140
Caught exception in looking up space point ptr... 0x2d4a3140, 0x105a8470
Caught exception in looking up space point ptr... 0x105a8470, 0x105a82e0
Caught exception in looking up space point ptr... 0x105a82e0, 0x105a8470
Caught exception in looking up space point ptr... 0x105a8470, 0x2d4a2ff0
Caught exception in looking up space point ptr... 0x2d4a2ff0, 0x105a8470
Caught exception in looking up space point ptr... 0xa2bb02c0, 0xa2b9dca0
Caught exception in looking up space point ptr... 0xa2b9dca0, 0xa2bb02c0
Caught exception in looking up space point ptr... 0x7f22a5c0, 0x7f401d80
Caught exception in looking up space point ptr... 0x7e55b7d0, 0x7f22a5c0
Caught exception in looking up space point ptr... 0xaeca38e0, 0xaed27bf0
Caught exception in looking up space point ptr... 0xaed27bf0, 0xaeca38e0
Caught exception in looking up space point ptr... 0x327ed640, 0x327edb60
Caught exception in looking up space point ptr... 0x327ed3b0, 0x327ed640
Caught exception in looking up space point ptr... 0x14c11f40, 0x14c15cf0
Caught exception in looking up space point ptr... 0x14c15cf0, 0x14c11f40
Caught exception in looking up space point ptr... 0x14daca30, 0x2db8c610
Caught exception in looking up space point ptr... 0x2db8c610, 0x22435610
Caught exception in looking up space point ptr... 0x4e5d5c10, 0x4e5d6fc0
Caught exception in looking up space point ptr... 0x4e5d6fc0, 0x4e5d5870
Caught exception in looking up space point ptr... 0xa01ee0b0, 0xa06130a0
Caught exception in looking up space point ptr... 0xa06130a0, 0xa01ee0b0
Caught exception in looking up space point ptr... 0xc630afb0, 0xba5c8b10
Caught exception in looking up space point ptr... 0xba5c8b10, 0xb20bb430
++++>>>> total num hits: 7337941, num free: 7154943
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 169.79 288.049 2570.54 196.802 466.048 25
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.3157e-05 0.000195304 0.000364739 0.000177976 7.04718e-05 25
produce:tpcrawdecoder:PDVDTPCReader 14.5579 15.3753 15.9818 15.3176 0.323414 25
produce:triggerrawdecoder:PDVDTriggerReader4 0.653733 0.677744 0.724405 0.67808 0.0170998 25
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0219262 0.0225354 0.0274949 0.0222115 0.00114601 25
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.108655 0.114578 0.135183 0.10962 0.00926881 25
produce:beamevent:BeamEvent 0.000175966 0.000294208 0.00136856 0.000238995 0.000225131 25
produce:pdvddaphne:DAPHNEReaderPDVD 7.20981 7.94236 8.93252 7.88819 0.37996 25
produce:ophit:OpHitFinder 0.0385391 0.0745414 0.0947249 0.0765541 0.0119056 25
produce:wclsdatavd:WireCellToolkit 122.367 138.042 152.422 137.175 7.50946 25
produce:gaushit:GausHitFinder 1.39329 2.382 10.3851 1.98986 1.67571 25
produce:nhitsfilter:NumberOfHitsFilter 0.000408487 0.00071377 0.00269852 0.000627905 0.000429574 25
produce:reco3d:SpacePointSolver 6.82247 29.9623 311.107 18.0154 57.6077 25
produce:hitpdune:DisambigFromSpacePoints 0.131936 0.830886 14.7421 0.238472 2.84047 25
produce:cluster3d:Cluster3D 1.41992 85.8159 2057.06 3.48867 402.379 25
[art]:TriggerResults:TriggerResultInserter 4.3446e-05 6.38288e-05 0.000158354 5.6881e-05 2.5305e-05 25
end_path:out1:RootOutput 1.0767e-05 1.58539e-05 4.1029e-05 1.3632e-05 6.51732e-06 25
end_path:out1:RootOutput(write) 5.85459 7.03667 7.83655 7.05843 0.489983 24
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 15889.6 MB
Peak resident set size usage (VmHWM): 13265.5 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 01:27:53 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 24 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 25 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 