Jobsub ID 303651.187@dunegpschedd01.fnal.gov
| Jobsub ID | 303651.187@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:50:24 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce509 |
| Last heartbeat | 2026-02-03 23:55:29 |
| From worker node | Hostname | b9p34p1022.cern.ch |
| cpuinfo | AMD EPYC 9655 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:23:36 |
| Input files | vd-protodune:np02vd_raw_run042421_0095_df-s04-d1_dw_0_20260203T142223.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-03 23:55:29 |
| Saved logs | justin-logs:303651.187-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
.. 0xcdeca1c0, 0xcdeca4d0
Caught exception in looking up space point ptr... 0xe02ab90, 0xe02af80
Caught exception in looking up space point ptr... 0xe02af80, 0xe02ab90
Caught exception in looking up space point ptr... 0xe02ab90, 0xe02ae30
Caught exception in looking up space point ptr... 0xe02ae30, 0xe02ab90
Caught exception in looking up space point ptr... 0xe02ab90, 0xe033d80
Caught exception in looking up space point ptr... 0xe033d80, 0xe02ab90
Caught exception in looking up space point ptr... 0x97697a10, 0x97122770
Caught exception in looking up space point ptr... 0x97122770, 0x97697a10
Caught exception in looking up space point ptr... 0x97697a10, 0x9ad12e70
Caught exception in looking up space point ptr... 0x9ad12e70, 0x97697a10
Caught exception in looking up space point ptr... 0xee663340, 0xee6631b0
Caught exception in looking up space point ptr... 0xee6631b0, 0xee6634d0
Caught exception in looking up space point ptr... 0xae5090f0, 0xb2209650
Caught exception in looking up space point ptr... 0xb2209650, 0xa1462f30
Caught exception in looking up space point ptr... 0x95a501d0, 0x9556d3c0
Caught exception in looking up space point ptr... 0x9556d3c0, 0x95a501d0
Caught exception in looking up space point ptr... 0x996a08d0, 0x982cb430
Caught exception in looking up space point ptr... 0x982cb430, 0x996a08d0
Caught exception in looking up space point ptr... 0x9e904ce0, 0x98beeea0
Caught exception in looking up space point ptr... 0x98beeea0, 0x9c07b9a0
Caught exception in looking up space point ptr... 0xedd533b0, 0xedb45dd0
Caught exception in looking up space point ptr... 0xedb45dd0, 0xedd533b0
Caught exception in looking up space point ptr... 0x8c9d2db0, 0x5d5bbaa0
Caught exception in looking up space point ptr... 0x8b0689a0, 0x8c9d2db0
Caught exception in looking up space point ptr... 0xd64e32e0, 0xd64e4a50
Caught exception in looking up space point ptr... 0xd64e9ba0, 0xd64e32e0
Caught exception in looking up space point ptr... 0xd78971e0, 0xd7897b80
Caught exception in looking up space point ptr... 0xd7897b80, 0xd7897350
Caught exception in looking up space point ptr... 0xe845dac0, 0xe845d090
Caught exception in looking up space point ptr... 0xe845d090, 0xe845dac0
Caught exception in looking up space point ptr... 0xcff06210, 0xac94bab0
Caught exception in looking up space point ptr... 0xac94bab0, 0xa81cb200
Caught exception in looking up space point ptr... 0x8c52a530, 0x8abaa340
Caught exception in looking up space point ptr... 0x8c4b7b10, 0x8c52a530
Caught exception in looking up space point ptr... 0x21428f50, 0x229b93e0
Caught exception in looking up space point ptr... 0x229b93e0, 0x21428f50
Caught exception in looking up space point ptr... 0x497369b0, 0x336a4ee0
Caught exception in looking up space point ptr... 0x336a4ee0, 0x497369b0
Caught exception in looking up space point ptr... 0x497369b0, 0x51e36ac0
Caught exception in looking up space point ptr... 0x51e36ac0, 0x497369b0
Caught exception in looking up space point ptr... 0xa37abe60, 0xa37abbd0
Caught exception in looking up space point ptr... 0xa37abbd0, 0xa37ab940
Caught exception in looking up space point ptr... 0xeffec520, 0xeffec320
Caught exception in looking up space point ptr... 0xeffec320, 0xeffec520
Caught exception in looking up space point ptr... 0x1144b650, 0x34f2de60
Caught exception in looking up space point ptr... 0x34f2de60, 0x1144b650
Caught exception in looking up space point ptr... 0x4761a550, 0x4765d6c0
Caught exception in looking up space point ptr... 0x4765d6c0, 0x59032ec0
Caught exception in looking up space point ptr... 0x928871d0, 0x94179ab0
Caught exception in looking up space point ptr... 0x94179ab0, 0x941e3190
Caught exception in looking up space point ptr... 0x4ac22270, 0x3da5a660
Caught exception in looking up space point ptr... 0x3da5a660, 0x80b846b0
Caught exception in looking up space point ptr... 0xef2b9990, 0xef2b98a0
Caught exception in looking up space point ptr... 0xef2b98a0, 0xef2b9990
Caught exception in looking up space point ptr... 0x36dda820, 0x36ddadf0
Caught exception in looking up space point ptr... 0x36ddadf0, 0x36dda820
Caught exception in looking up space point ptr... 0x9e7b6440, 0x9dedf100
Caught exception in looking up space point ptr... 0x9dedf100, 0x9e7b6440
Caught exception in looking up space point ptr... 0xcb130fe0, 0xbcbc5490
Caught exception in looking up space point ptr... 0xbcbc5490, 0xc27e3a10
Caught exception in looking up space point ptr... 0xc500ce00, 0xc500cc70
Caught exception in looking up space point ptr... 0xc500cc70, 0xc500cf90
Caught exception in looking up space point ptr... 0xa5482e40, 0xa62e3710
Caught exception in looking up space point ptr... 0xa62e3710, 0xa5482e40
Caught exception in looking up space point ptr... 0xd98d1310, 0xd99c8df0
Caught exception in looking up space point ptr... 0xd99c8df0, 0xe3898c10
Caught exception in looking up space point ptr... 0xe3898c10, 0xe555f080
Caught exception in looking up space point ptr... 0xe555f080, 0xd98d1310
Caught exception in looking up space point ptr... 0xf002e7f0, 0xf002e610
Caught exception in looking up space point ptr... 0xf002e610, 0xf002e7f0
Caught exception in looking up space point ptr... 0x927f5e40, 0x927f8ed0
Caught exception in looking up space point ptr... 0x927f8ed0, 0x991f6a70
Caught exception in looking up space point ptr... 0xdc6db1e0, 0xdc6e3160
Caught exception in looking up space point ptr... 0xdc6e3160, 0xe3c8fe00
Caught exception in looking up space point ptr... 0x36dd6200, 0x36dd5d10
Caught exception in looking up space point ptr... 0x36dd5d10, 0x36dd6550
Caught exception in looking up space point ptr... 0x928cd7a0, 0x9119fff0
Caught exception in looking up space point ptr... 0x91539580, 0x928cd7a0
Caught exception in looking up space point ptr... 0xe040050, 0xe03c140
Caught exception in looking up space point ptr... 0xe03c140, 0xe040050
++++>>>> total num hits: 8887350, num free: 8670329
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 63.1722 302.399 1422.22 74.1215 501.016 6
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.8262e-05 0.000117427 0.000248132 8.44715e-05 6.3747e-05 6
produce:tpcrawdecoder:PDVDTPCReader 3.6282 3.67915 3.7301 3.67481 0.0341973 6
produce:triggerrawdecoder:PDVDTriggerReader4 0.00880421 0.0115468 0.0151757 0.0112497 0.00190131 6
produce:timingrawdecoder:PDHDTimingRawDecoder 0.000386458 0.000576756 0.000845395 0.000505 0.000169557 6
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.00123651 0.001541 0.00187601 0.00147558 0.000252661 6
produce:beamevent:BeamEvent 0.000105738 0.00028337 0.00110013 0.000118918 0.000365505 6
produce:pdvddaphne:DAPHNEReaderPDVD 1.93459 2.12747 2.38684 2.04533 0.183707 6
produce:ophit:OpHitFinder 0.0265387 0.0347579 0.0402649 0.0355656 0.00468782 6
produce:wclsdatavd:WireCellToolkit 47.1819 53.6985 57.6397 54.1756 3.56161 6
produce:gaushit:GausHitFinder 0.673497 3.60929 8.68887 2.77455 3.01999 6
produce:nhitsfilter:NumberOfHitsFilter 0.000177295 0.000554984 0.000926366 0.000567444 0.000286847 6
produce:reco3d:SpacePointSolver 5.69004 43.8177 156.969 6.30619 65.3294 4
produce:hitpdune:DisambigFromSpacePoints 0.0418526 1.54111 5.87957 0.12152 2.50502 4
produce:cluster3d:Cluster3D 0.810561 308.343 1191.54 20.5122 510.141 4
[art]:TriggerResults:TriggerResultInserter 3.0255e-05 6.73155e-05 0.000137325 6.05705e-05 3.42997e-05 6
end_path:out1:RootOutput 8.263e-06 1.39508e-05 3.332e-05 1.06505e-05 8.78893e-06 6
end_path:out1:RootOutput(write) 2.62705 4.09617 5.54131 4.37099 1.05044 5
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 17296.8 MB
Peak resident set size usage (VmHWM): 14698.1 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:54:57 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 5 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 6 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 