Jobsub ID 297922.23@dunegpschedd02.fnal.gov
| Jobsub ID | 297922.23@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 23:00:24 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce04 |
| Last heartbeat | 2026-02-04 01:50:43 |
| From worker node | Hostname | dune001-1520390.0-lcg2708.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 9654 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:57:27 |
| Input files | vd-protodune:np02vd_raw_run042421_0007_df-s04-d1_dw_0_20260203T133442.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:50:43 |
| Saved logs | justin-logs:297922.23-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ng up space point ptr... 0xe0c98f00, 0x7d0e5260
Caught exception in looking up space point ptr... 0x7d0e5260, 0x12b4a7330
Caught exception in looking up space point ptr... 0x1198b4da0, 0xbd02d8d0
Caught exception in looking up space point ptr... 0xbd02d8d0, 0x1198b4da0
Caught exception in looking up space point ptr... 0x44e5f720, 0x13772dc50
Caught exception in looking up space point ptr... 0x68806f50, 0x44e5f720
Caught exception in looking up space point ptr... 0xcc7515e0, 0xcc7fb670
Caught exception in looking up space point ptr... 0xcc7fb670, 0xcc7515e0
Caught exception in looking up space point ptr... 0x763a3d80, 0x588327d0
Caught exception in looking up space point ptr... 0x588327d0, 0x763a3d80
Caught exception in looking up space point ptr... 0x7fc6d8e0, 0x5a247480
Caught exception in looking up space point ptr... 0x5a24a040, 0x5a124de0
Caught exception in looking up space point ptr... 0x5a124de0, 0x5a124f70
Caught exception in looking up space point ptr... 0x5a124f70, 0x7fc6d8e0
Caught exception in looking up space point ptr... 0xfb126f00, 0xfb126d70
Caught exception in looking up space point ptr... 0xfb126d70, 0x999f84a0
Caught exception in looking up space point ptr... 0x13629cff0, 0x66909350
Caught exception in looking up space point ptr... 0x66909350, 0x12c360970
Caught exception in looking up space point ptr... 0xb6349350, 0xea82f3d0
Caught exception in looking up space point ptr... 0xea82f3d0, 0xdf2f4c30
Caught exception in looking up space point ptr... 0x73fc4b30, 0x449d9b20
Caught exception in looking up space point ptr... 0x449d9b20, 0x73fc4b30
Caught exception in looking up space point ptr... 0x73fc4b30, 0x79f140b0
Caught exception in looking up space point ptr... 0x79f140b0, 0x73fc4b30
Caught exception in looking up space point ptr... 0xa4d7b330, 0xa4eb65b0
Caught exception in looking up space point ptr... 0xd5fe3b90, 0xa4d7b330
Caught exception in looking up space point ptr... 0xc8511bb0, 0xd4ecf5c0
Caught exception in looking up space point ptr... 0xd4ecf5c0, 0xc8511bb0
Caught exception in looking up space point ptr... 0x15fc54240, 0xdd42b510
Caught exception in looking up space point ptr... 0xdd42b510, 0x15fc54240
Caught exception in looking up space point ptr... 0x136ec55a0, 0x136cdf390
Caught exception in looking up space point ptr... 0x136cdf390, 0x136ec55a0
Caught exception in looking up space point ptr... 0x15fc2acc0, 0x96780e40
Caught exception in looking up space point ptr... 0x96780e40, 0xbf096160
Caught exception in looking up space point ptr... 0x13b7325b0, 0xb7d75b90
Caught exception in looking up space point ptr... 0xb7d75b90, 0xa2fc7500
Caught exception in looking up space point ptr... 0x13b5565c0, 0xe86f6a30
Caught exception in looking up space point ptr... 0xe86f6a30, 0x13b5565c0
Caught exception in looking up space point ptr... 0x97f7cb80, 0x97fdb8c0
Caught exception in looking up space point ptr... 0x97fdb8c0, 0x97f7cb80
Caught exception in looking up space point ptr... 0xc896de00, 0xc896dd60
Caught exception in looking up space point ptr... 0xc896dd60, 0xc896de00
Caught exception in looking up space point ptr... 0x8ce37190, 0x3f5c9470
Caught exception in looking up space point ptr... 0x3f5c9470, 0xa0ef4fe0
Caught exception in looking up space point ptr... 0x13b395c90, 0x169933ec0
Caught exception in looking up space point ptr... 0x6f80f600, 0x13b395c90
Caught exception in looking up space point ptr... 0x4b42a530, 0xecf8fc50
Caught exception in looking up space point ptr... 0xe88515e0, 0x4b42a530
Caught exception in looking up space point ptr... 0x9e956400, 0x5d01d300
Caught exception in looking up space point ptr... 0x5d01d300, 0x9e956400
Caught exception in looking up space point ptr... 0x3fb12360, 0xe52443f0
Caught exception in looking up space point ptr... 0x6ad8a4f0, 0x3fb12360
Caught exception in looking up space point ptr... 0x12a3aeb60, 0x12a3b6930
Caught exception in looking up space point ptr... 0x12a3b6930, 0x12a3b6b50
Caught exception in looking up space point ptr... 0x9f673470, 0x14165fe20
Caught exception in looking up space point ptr... 0x14165fe20, 0x1416567f0
Caught exception in looking up space point ptr... 0x9f677480, 0x141608920
Caught exception in looking up space point ptr... 0x141608920, 0x141657a20
Caught exception in looking up space point ptr... 0x98d12bc0, 0x769ec990
Caught exception in looking up space point ptr... 0x769ec990, 0x4b8096b0
Caught exception in looking up space point ptr... 0xf654cfb0, 0x937159a0
Caught exception in looking up space point ptr... 0x937159a0, 0xf654cfb0
Caught exception in looking up space point ptr... 0xdc90f850, 0xfbc6b810
Caught exception in looking up space point ptr... 0xfbc6b810, 0x6a9a5b90
Caught exception in looking up space point ptr... 0xc6fde270, 0xb7946730
Caught exception in looking up space point ptr... 0xb7946730, 0x4c681510
Caught exception in looking up space point ptr... 0x119046190, 0xc3204600
Caught exception in looking up space point ptr... 0xc3204600, 0x119046190
Caught exception in looking up space point ptr... 0x119046190, 0xbd67c0c0
Caught exception in looking up space point ptr... 0xbd67c0c0, 0x119046190
Caught exception in looking up space point ptr... 0xd9cb8ce0, 0xd9cb8ba0
Caught exception in looking up space point ptr... 0xd9cb8ba0, 0xd9cb8ce0
Caught exception in looking up space point ptr... 0x949b4cc0, 0xe89316e0
Caught exception in looking up space point ptr... 0xe89316e0, 0x949b4cc0
Caught exception in looking up space point ptr... 0x949b4cc0, 0x8a3ccd70
Caught exception in looking up space point ptr... 0x8a3ccd70, 0x949b4cc0
Caught exception in looking up space point ptr... 0x1198ed960, 0xb8bf9740
Caught exception in looking up space point ptr... 0xb8bf9740, 0x1198ed960
Caught exception in looking up space point ptr... 0xc5b26bb0, 0xc5abe530
Caught exception in looking up space point ptr... 0xc5abe530, 0xc5b26a40
++++>>>> total num hits: 7022863, num free: 6846998
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 83.0783 255.138 1431.8 146.755 295.153 26
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 8.1262e-05 0.000141246 0.000397118 0.000112514 7.36216e-05 26
produce:tpcrawdecoder:PDVDTPCReader 13.1302 13.782 17.0917 13.5053 0.812748 26
produce:triggerrawdecoder:PDVDTriggerReader4 0.688928 0.714346 0.768305 0.710246 0.0165295 26
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0230259 0.0232389 0.023639 0.023227 0.000125537 26
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.113979 0.116327 0.13901 0.114549 0.00623369 26
produce:beamevent:BeamEvent 0.000125088 0.000225555 0.00115915 0.000188959 0.000190672 26
produce:pdvddaphne:DAPHNEReaderPDVD 3.23459 3.67306 4.37767 3.63372 0.232885 26
produce:ophit:OpHitFinder 0.034583 0.0423235 0.0527461 0.0422193 0.00362169 26
produce:wclsdatavd:WireCellToolkit 55.8317 72.7448 86.0052 73.6951 6.27322 26
produce:gaushit:GausHitFinder 0.522535 2.26037 10.6373 1.45687 2.19052 26
produce:nhitsfilter:NumberOfHitsFilter 0.000159931 0.000441974 0.00111764 0.000355584 0.000286745 26
produce:reco3d:SpacePointSolver 4.46124 24.0618 133.557 9.44441 34.8643 26
produce:hitpdune:DisambigFromSpacePoints 0.0440621 1.15285 6.1547 0.237847 1.98548 26
produce:cluster3d:Cluster3D 0.809648 130.365 1193.56 43.0309 254.837 26
[art]:TriggerResults:TriggerResultInserter 2.3997e-05 5.64634e-05 0.000235083 4.4582e-05 4.34632e-05 26
end_path:out1:RootOutput 4.487e-06 2.70685e-05 0.000404389 1.10015e-05 7.57308e-05 26
end_path:out1:RootOutput(write) 2.78278 6.38953 21.3358 5.16363 4.32365 25
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 17390 MB
Peak resident set size usage (VmHWM): 14718.7 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 01:49:33 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 25 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 26 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 