Jobsub ID 303650.70@dunegpschedd01.fnal.gov
| Jobsub ID | 303650.70@dunegpschedd01.fnal.gov |
| Workflow ID | 12719 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:48:28 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2026-02-04 00:35:14 |
| From worker node | Hostname | wn-f01-16-01-07.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:15:44 |
| Input files | vd-protodune:np02vd_raw_run042433_0498_df-s04-d0_dw_0_20260203T215123.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 00:35:14 |
| Saved logs | justin-logs:303650.70-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
0x498cebd0, 0x76597990
Caught exception in looking up space point ptr... 0x76597990, 0x6faf9070
Caught exception in looking up space point ptr... 0x6faf9070, 0x76597990
Caught exception in looking up space point ptr... 0x9c1f7970, 0x96e55d20
Caught exception in looking up space point ptr... 0x96e55d20, 0x970586a0
Caught exception in looking up space point ptr... 0xc4790400, 0xc939ec80
Caught exception in looking up space point ptr... 0xc939ec80, 0xc939edb0
Caught exception in looking up space point ptr... 0x9bf7db30, 0x6801c040
Caught exception in looking up space point ptr... 0xac04adf0, 0x9bf7db30
Caught exception in looking up space point ptr... 0x93a1370, 0x37cb3180
Caught exception in looking up space point ptr... 0x37cb3180, 0x93a1370
Caught exception in looking up space point ptr... 0x6c3c6320, 0x428c77e0
Caught exception in looking up space point ptr... 0x428c77e0, 0x6c3c6320
Caught exception in looking up space point ptr... 0x9adb8b30, 0x9c2a5b80
Caught exception in looking up space point ptr... 0x9c2a5b80, 0x9adb8b30
Caught exception in looking up space point ptr... 0x542dff90, 0x88cb8550
Caught exception in looking up space point ptr... 0x88cb8550, 0x36ad1ef0
Caught exception in looking up space point ptr... 0x9b0835e0, 0x9b311bb0
Caught exception in looking up space point ptr... 0x9b311bb0, 0x9b0835e0
Caught exception in looking up space point ptr... 0x9a83f870, 0x9b36bf40
Caught exception in looking up space point ptr... 0x9b36bf40, 0x9a83f870
Caught exception in looking up space point ptr... 0x74aded60, 0x41953d90
Caught exception in looking up space point ptr... 0x41953d90, 0x74adf080
Caught exception in looking up space point ptr... 0x87efa420, 0x56e6c090
Caught exception in looking up space point ptr... 0x56e6c090, 0x59807d80
Caught exception in looking up space point ptr... 0x9c9255b0, 0x9b374780
Caught exception in looking up space point ptr... 0x9b374780, 0x9c9255b0
Caught exception in looking up space point ptr... 0xc80086b0, 0xc46cd9b0
Caught exception in looking up space point ptr... 0xc46cd9b0, 0xc490da20
Caught exception in looking up space point ptr... 0xc937f9b0, 0xc7317ed0
Caught exception in looking up space point ptr... 0xc9363990, 0xc937f9b0
Caught exception in looking up space point ptr... 0x607a4490, 0xf6a9730
Caught exception in looking up space point ptr... 0xf6a9730, 0x607a4490
Caught exception in looking up space point ptr... 0x67a8fcc0, 0x484017c0
Caught exception in looking up space point ptr... 0x484017c0, 0x2fdd55f0
Caught exception in looking up space point ptr... 0x6d75cce0, 0x756d8f40
Caught exception in looking up space point ptr... 0x756d8f40, 0x6d75cce0
Caught exception in looking up space point ptr... 0x77499d40, 0x4527fc80
Caught exception in looking up space point ptr... 0x4527fc80, 0x77499d40
Caught exception in looking up space point ptr... 0x852c4990, 0x6c02cd90
Caught exception in looking up space point ptr... 0x6c02cd90, 0x852c4990
Caught exception in looking up space point ptr... 0x5383bb50, 0x77120180
Caught exception in looking up space point ptr... 0x77120180, 0x5f19cfb0
Caught exception in looking up space point ptr... 0x772f1e90, 0x7f3293e0
Caught exception in looking up space point ptr... 0x7f3293e0, 0x772f1e90
Caught exception in looking up space point ptr... 0xf0c60b0, 0xf0f85c0
Caught exception in looking up space point ptr... 0xf0f85c0, 0x1add9540
Caught exception in looking up space point ptr... 0xaa586030, 0xab1126c0
Caught exception in looking up space point ptr... 0xab1126c0, 0xaa586030
Caught exception in looking up space point ptr... 0xaa381470, 0xa915a370
Caught exception in looking up space point ptr... 0xa915a370, 0xa892a610
Caught exception in looking up space point ptr... 0x1add8e60, 0x6a3a3250
Caught exception in looking up space point ptr... 0x6a3a3250, 0xf0ead20
Caught exception in looking up space point ptr... 0x99c833d0, 0x99d38f90
Caught exception in looking up space point ptr... 0x99d38f90, 0x999eb990
Caught exception in looking up space point ptr... 0xa8276b70, 0xa7db45a0
Caught exception in looking up space point ptr... 0xa7db45a0, 0xa8276b70
Caught exception in looking up space point ptr... 0x966062f0, 0x99a1f320
Caught exception in looking up space point ptr... 0x99a1f320, 0x40c612b0
Caught exception in looking up space point ptr... 0x578fd720, 0x578fb1e0
Caught exception in looking up space point ptr... 0x578fb1e0, 0x578fd9c0
Caught exception in looking up space point ptr... 0x6c160f50, 0x41b86280
Caught exception in looking up space point ptr... 0x41b86280, 0x6c160f50
Caught exception in looking up space point ptr... 0xd3650e80, 0xd2bad980
Caught exception in looking up space point ptr... 0xd2bad980, 0xd2bad660
Caught exception in looking up space point ptr... 0x5a825d90, 0x5a825e90
Caught exception in looking up space point ptr... 0x5a825e90, 0x5a825d90
Caught exception in looking up space point ptr... 0xaad7cd20, 0xa9362c80
Caught exception in looking up space point ptr... 0xa9362c80, 0xaad7cd20
Caught exception in looking up space point ptr... 0xfdd2080, 0xfdd2210
Caught exception in looking up space point ptr... 0xfdd2210, 0xfdd2080
Caught exception in looking up space point ptr... 0x57a630d0, 0x57a6a380
Caught exception in looking up space point ptr... 0x57a6a380, 0x57a75880
Caught exception in looking up space point ptr... 0xc4842c70, 0xc482da50
Caught exception in looking up space point ptr... 0xc482da50, 0xc4489fe0
Caught exception in looking up space point ptr... 0xc4489fe0, 0xc4489cd0
Caught exception in looking up space point ptr... 0xc4489cd0, 0xc4842c70
Caught exception in looking up space point ptr... 0xc796c5e0, 0xc84aa8e0
Caught exception in looking up space point ptr... 0xc84aa8e0, 0xc848f2c0
Caught exception in looking up space point ptr... 0xc848f2c0, 0xc7c30da0
Caught exception in looking up space point ptr... 0xc7c30da0, 0xc796c5e0
++++>>>> total num hits: 7272110, num free: 7094171
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 331.767 865.925 2682.09 440.177 909.031 5
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.6655e-05 0.000185407 0.000288014 0.000176459 6.09869e-05 5
produce:tpcrawdecoder:PDVDTPCReader 24.325 28.5108 32.7532 28.5338 2.88787 5
produce:triggerrawdecoder:PDVDTriggerReader4 0.311502 0.439217 0.831023 0.345226 0.197351 5
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0105747 0.0108701 0.011366 0.0106377 0.000328671 5
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0104175 0.0106509 0.0112283 0.0105261 0.000300571 5
produce:beamevent:BeamEvent 0.000133349 0.00046043 0.00164525 0.000171388 0.000592851 5
produce:pdvddaphne:DAPHNEReaderPDVD 6.64708 7.48059 7.85404 7.78876 0.468638 5
produce:ophit:OpHitFinder 0.0559637 0.0578408 0.0594516 0.0579061 0.00142802 5
produce:wclsdatavd:WireCellToolkit 235.732 242.97 251.234 243.545 5.84008 5
produce:gaushit:GausHitFinder 3.95875 5.74239 11.9701 4.25452 3.11639 5
produce:nhitsfilter:NumberOfHitsFilter 0.000680651 0.00104732 0.00202487 0.000707245 0.000517204 5
produce:reco3d:SpacePointSolver 22.8492 63.6849 207.239 24.8709 72.0544 5
produce:hitpdune:DisambigFromSpacePoints 0.557112 2.62615 10.7168 0.627817 4.04547 5
produce:cluster3d:Cluster3D 6.03098 504.754 2181.79 110.811 839.531 5
[art]:TriggerResults:TriggerResultInserter 3.0505e-05 6.27162e-05 0.000130326 4.4441e-05 3.55431e-05 5
end_path:out1:RootOutput 8.189e-06 1.41008e-05 3.3288e-05 1.0029e-05 9.6235e-06 5
end_path:out1:RootOutput(write) 10.5907 12.0108 12.6284 12.412 0.824915 4
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 15371 MB
Peak resident set size usage (VmHWM): 12747.9 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 01:34:09 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 5 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 