Jobsub ID 303647.154@dunegpschedd01.fnal.gov
| Jobsub ID | 303647.154@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:46:24 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce504 |
| Last heartbeat | 2026-02-03 23:30:42 |
| From worker node | Hostname | b9p02p3882.cern.ch |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 22:47:15 |
| Input files | vd-protodune:np02vd_raw_run042421_0143_df-s04-d1_dw_0_20260203T144821.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-03 23:30:42 |
| Saved logs | justin-logs:303647.154-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
n looking up space point ptr... 0x9d7671e0, 0xc1318b50
Caught exception in looking up space point ptr... 0xc1318b50, 0x9d7671e0
Caught exception in looking up space point ptr... 0x3c8aa260, 0x42fba430
Caught exception in looking up space point ptr... 0x42fba430, 0x42fa9c00
Caught exception in looking up space point ptr... 0xf2de6260, 0xb8224480
Caught exception in looking up space point ptr... 0xb8224480, 0xf2de6260
Caught exception in looking up space point ptr... 0xe34ec2e0, 0xe34a69a0
Caught exception in looking up space point ptr... 0xe34a69a0, 0xe34ec2e0
Caught exception in looking up space point ptr... 0x51247d70, 0xf0a098f0
Caught exception in looking up space point ptr... 0xf0a098f0, 0x51247be0
Caught exception in looking up space point ptr... 0x76431860, 0x764316d0
Caught exception in looking up space point ptr... 0x764316d0, 0x764313b0
Caught exception in looking up space point ptr... 0xf8261dc0, 0xf8261c30
Caught exception in looking up space point ptr... 0xf8261c30, 0xf8261780
Caught exception in looking up space point ptr... 0xf8261780, 0xf8261910
Caught exception in looking up space point ptr... 0xf8261910, 0xf8261dc0
Caught exception in looking up space point ptr... 0xacae4ed0, 0xacae4d40
Caught exception in looking up space point ptr... 0xacae4d40, 0xacae4ed0
Caught exception in looking up space point ptr... 0xacae4ed0, 0x9849ca0
Caught exception in looking up space point ptr... 0x9849ca0, 0xacae4ed0
Caught exception in looking up space point ptr... 0xaa9c52a0, 0xd302deb0
Caught exception in looking up space point ptr... 0xd302deb0, 0x42765460
Caught exception in looking up space point ptr... 0xfde3ed40, 0xdb9a0730
Caught exception in looking up space point ptr... 0xdb9a0730, 0xfde3ed40
Caught exception in looking up space point ptr... 0xd77e7620, 0x652a1980
Caught exception in looking up space point ptr... 0x652a1980, 0xc49bd1b0
Caught exception in looking up space point ptr... 0x107715480, 0x1077152f0
Caught exception in looking up space point ptr... 0x1077152f0, 0x107715480
Caught exception in looking up space point ptr... 0x107715480, 0x1077157a0
Caught exception in looking up space point ptr... 0x1077157a0, 0x107715480
Caught exception in looking up space point ptr... 0x107715480, 0x107f95210
Caught exception in looking up space point ptr... 0x107f95210, 0x107715480
Caught exception in looking up space point ptr... 0xcd864580, 0xcd2fe700
Caught exception in looking up space point ptr... 0xcd2fe700, 0xcd864580
Caught exception in looking up space point ptr... 0xc6bfc540, 0xc78278a0
Caught exception in looking up space point ptr... 0xc78278a0, 0xc6bfc540
Caught exception in looking up space point ptr... 0xa20e9680, 0xa20bd1a0
Caught exception in looking up space point ptr... 0xa20bd1a0, 0xa20e9680
Caught exception in looking up space point ptr... 0xa20e9680, 0x9f0bad00
Caught exception in looking up space point ptr... 0x9f0bad00, 0xa20e9680
Caught exception in looking up space point ptr... 0xc68aed70, 0xc68aeb20
Caught exception in looking up space point ptr... 0xc68aeb20, 0xc68aed70
Caught exception in looking up space point ptr... 0xb46d19d0, 0xb46cbbe0
Caught exception in looking up space point ptr... 0xb46cbbe0, 0xb46d19d0
Caught exception in looking up space point ptr... 0xda6e4c00, 0xbda214c0
Caught exception in looking up space point ptr... 0xbda214c0, 0xda6e4c00
Caught exception in looking up space point ptr... 0xd4d4c570, 0xbd0fadb0
Caught exception in looking up space point ptr... 0xbd0fadb0, 0xd4d4c570
Caught exception in looking up space point ptr... 0xd4d4c570, 0xbc025750
Caught exception in looking up space point ptr... 0xbc025750, 0xd4d4c570
Caught exception in looking up space point ptr... 0xe763a8a0, 0xe8466650
Caught exception in looking up space point ptr... 0xe8466650, 0xe763a8a0
Caught exception in looking up space point ptr... 0x3e991fe0, 0x3edd05f0
Caught exception in looking up space point ptr... 0x3edd05f0, 0x3e991fe0
Caught exception in looking up space point ptr... 0x107ff8720, 0x107ff8400
Caught exception in looking up space point ptr... 0x107ff8400, 0x107ff8720
Caught exception in looking up space point ptr... 0xd5786f70, 0xd5232760
Caught exception in looking up space point ptr... 0xd5232760, 0xdb316100
Caught exception in looking up space point ptr... 0xa2b9c180, 0xa2055e00
Caught exception in looking up space point ptr... 0xa2055e00, 0xa1e91fe0
Caught exception in looking up space point ptr... 0x10ccbf3d0, 0x10ccbf0b0
Caught exception in looking up space point ptr... 0x10ccbf0b0, 0x10ccbf3d0
Caught exception in looking up space point ptr... 0x62768c50, 0x63368870
Caught exception in looking up space point ptr... 0x63368870, 0x63368a00
Caught exception in looking up space point ptr... 0x626d8280, 0x63368eb0
Caught exception in looking up space point ptr... 0x63368eb0, 0x63369040
Caught exception in looking up space point ptr... 0x104d14370, 0x104d039e0
Caught exception in looking up space point ptr... 0x104d039e0, 0x104d14370
Caught exception in looking up space point ptr... 0xa8a86470, 0xf5e0d170
Caught exception in looking up space point ptr... 0xf5e0d170, 0xa8a86470
Caught exception in looking up space point ptr... 0xf8b38590, 0xf8b38400
Caught exception in looking up space point ptr... 0xf8b38400, 0xf8b38590
Caught exception in looking up space point ptr... 0x102e1a710, 0x102e1a260
Caught exception in looking up space point ptr... 0x102e1a260, 0x102e1a3f0
Caught exception in looking up space point ptr... 0xf8b3e1c0, 0xf8b3dd10
Caught exception in looking up space point ptr... 0xf8b3dd10, 0xf8b3e1c0
Caught exception in looking up space point ptr... 0x10d08b3c0, 0x26ff8500
Caught exception in looking up space point ptr... 0x26ff8500, 0x10d08b3c0
Caught exception in looking up space point ptr... 0xa5a3c200, 0xa5a3c390
Caught exception in looking up space point ptr... 0xa5a3c390, 0xa62bc1e0
++++>>>> total num hits: 7805566, num free: 7625175
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 87.5662 495.427 1488.75 141.559 540.23 5
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.9111e-05 0.000150646 0.000214992 0.000155731 5.68978e-05 5
produce:tpcrawdecoder:PDVDTPCReader 4.07587 4.15268 4.30995 4.09555 0.0898054 5
produce:triggerrawdecoder:PDVDTriggerReader4 0.0141762 0.0195322 0.0328331 0.0172474 0.00688958 5
produce:timingrawdecoder:PDHDTimingRawDecoder 0.000469895 0.000732114 0.00113647 0.000678337 0.000235667 5
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.00178391 0.00216986 0.00237124 0.00221937 0.000203873 5
produce:beamevent:BeamEvent 0.000102111 0.00033454 0.00113359 0.000128382 0.000400581 5
produce:pdvddaphne:DAPHNEReaderPDVD 2.24266 2.81579 3.27654 3.06426 0.418457 5
produce:ophit:OpHitFinder 0.0307379 0.0373849 0.0428188 0.0394579 0.0050984 5
produce:wclsdatavd:WireCellToolkit 67.6605 74.7852 82.3455 73.4994 5.50961 5
produce:gaushit:GausHitFinder 0.882467 3.56133 7.99904 1.83199 2.72306 5
produce:nhitsfilter:NumberOfHitsFilter 0.000236382 0.000766198 0.00158857 0.000700147 0.000481377 5
produce:reco3d:SpacePointSolver 6.66321 59.2825 173.584 15.1016 65.6157 5
produce:hitpdune:DisambigFromSpacePoints 0.0977116 2.85081 9.08431 0.146999 3.60509 5
produce:cluster3d:Cluster3D 1.26004 341.508 1223.87 37.9249 471.44 5
[art]:TriggerResults:TriggerResultInserter 2.156e-05 4.68486e-05 8.4681e-05 4.5941e-05 2.24461e-05 5
end_path:out1:RootOutput 6.96e-06 1.2846e-05 2.806e-05 1.036e-05 7.78501e-06 5
end_path:out1:RootOutput(write) 3.56635 7.9619 19.2776 4.50183 6.54705 4
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 17468.2 MB
Peak resident set size usage (VmHWM): 15457.6 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:29:51 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 5 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 