Jobsub ID 303711.0@dunegpschedd01.fnal.gov
| Jobsub ID | 303711.0@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-04 00:44:28 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce504 |
| Last heartbeat | 2026-02-04 01:13:32 |
| From worker node | Hostname | b9p12p0289.cern.ch |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 00:44:54 |
| Input files | vd-protodune:np02vd_raw_run042421_0095_df-s02-d0_dw_0_20260203T142223.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 01:13:32 |
| Saved logs | justin-logs:303711.0-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
0x4888d6f0
Caught exception in looking up space point ptr... 0x4888d6f0, 0x13517d60
Caught exception in looking up space point ptr... 0x688b7550, 0xa0991360
Caught exception in looking up space point ptr... 0xa0991360, 0xa089f780
Caught exception in looking up space point ptr... 0xa089f780, 0x9fdb4ee0
Caught exception in looking up space point ptr... 0x9fdb4ee0, 0x9ed7d2b0
Caught exception in looking up space point ptr... 0xa095e140, 0x688b7550
Caught exception in looking up space point ptr... 0xa9d0fa60, 0x9b7e77b0
Caught exception in looking up space point ptr... 0x9b7e77b0, 0x9b8e9060
Caught exception in looking up space point ptr... 0x75fc1650, 0xb1103ef0
Caught exception in looking up space point ptr... 0xb1103ef0, 0x75fc1650
Caught exception in looking up space point ptr... 0x75fc1650, 0x791b51f0
Caught exception in looking up space point ptr... 0x791b51f0, 0x75fc1650
Caught exception in looking up space point ptr... 0x651f9250, 0x59364070
Caught exception in looking up space point ptr... 0x59364070, 0x651f9250
Caught exception in looking up space point ptr... 0x651f9250, 0x65215a70
Caught exception in looking up space point ptr... 0x65215a70, 0x651f9250
Caught exception in looking up space point ptr... 0x65d428b0, 0x641392f0
Caught exception in looking up space point ptr... 0x641392f0, 0x64139160
Caught exception in looking up space point ptr... 0xcc75b120, 0xcc75b270
Caught exception in looking up space point ptr... 0xcc75b270, 0xccb15790
Caught exception in looking up space point ptr... 0x33f22170, 0x23aee1e0
Caught exception in looking up space point ptr... 0xe4e4270, 0x33f22170
Caught exception in looking up space point ptr... 0xb6b621f0, 0xb6b3d220
Caught exception in looking up space point ptr... 0xb6b3d220, 0xb6b621f0
Caught exception in looking up space point ptr... 0xc3d72430, 0xc72422e0
Caught exception in looking up space point ptr... 0xc72422e0, 0xc3d72430
Caught exception in looking up space point ptr... 0x9ad0ed00, 0x99b4b5c0
Caught exception in looking up space point ptr... 0x99b4b5c0, 0x9ad0ed00
Caught exception in looking up space point ptr... 0xb636bda0, 0xb636bc20
Caught exception in looking up space point ptr... 0xb636bc20, 0xb636bda0
Caught exception in looking up space point ptr... 0xb636bda0, 0xb5c477e0
Caught exception in looking up space point ptr... 0xb5c477e0, 0xb636bda0
Caught exception in looking up space point ptr... 0xb5ec2d40, 0xb5fa7fe0
Caught exception in looking up space point ptr... 0xb5fa7fe0, 0xb5ec2d40
Caught exception in looking up space point ptr... 0xb5ec2d40, 0xb3c69040
Caught exception in looking up space point ptr... 0xb3c69040, 0xb5ec2d40
Caught exception in looking up space point ptr... 0xbfdcd730, 0xc0215a00
Caught exception in looking up space point ptr... 0xc0215a00, 0xc6e383e0
Caught exception in looking up space point ptr... 0x5fafc2a0, 0x5fafc4a0
Caught exception in looking up space point ptr... 0x5fafc4a0, 0x5fafc3a0
Caught exception in looking up space point ptr... 0x13b6c140, 0x49a94ae0
Caught exception in looking up space point ptr... 0x49a94ae0, 0x13b6b940
Caught exception in looking up space point ptr... 0xb4ffe320, 0xb4ecb430
Caught exception in looking up space point ptr... 0xb4ecb430, 0xb4ffe320
Caught exception in looking up space point ptr... 0x7f7fd290, 0x7f51a780
Caught exception in looking up space point ptr... 0x7f51a780, 0x7f7fd290
Caught exception in looking up space point ptr... 0xcc72bf10, 0xccb24920
Caught exception in looking up space point ptr... 0xccb24920, 0xcc72ba60
Caught exception in looking up space point ptr... 0xca557cd0, 0xca4f1690
Caught exception in looking up space point ptr... 0xca4f1690, 0xca557cd0
Caught exception in looking up space point ptr... 0x97cc4070, 0xadfd90f0
Caught exception in looking up space point ptr... 0xadfd90f0, 0x97cc4070
Caught exception in looking up space point ptr... 0x97cc4070, 0xb2348dd0
Caught exception in looking up space point ptr... 0xb2348dd0, 0x97cc4070
Caught exception in looking up space point ptr... 0x97cc4070, 0x75ba7170
Caught exception in looking up space point ptr... 0x75ba7170, 0x97cc4070
Caught exception in looking up space point ptr... 0x97cc4070, 0x974f40d0
Caught exception in looking up space point ptr... 0x974f40d0, 0x97cc4070
Caught exception in looking up space point ptr... 0xb63d0f40, 0xb3fcdbe0
Caught exception in looking up space point ptr... 0xb63ad1a0, 0xb63d0f40
Caught exception in looking up space point ptr... 0xc3ff9250, 0xc3178f00
Caught exception in looking up space point ptr... 0xc3178f00, 0xc3ff9250
Caught exception in looking up space point ptr... 0x8f7521d0, 0x6a100470
Caught exception in looking up space point ptr... 0x6a100470, 0x8f7521d0
Caught exception in looking up space point ptr... 0x8f7521d0, 0x2b97d4f0
Caught exception in looking up space point ptr... 0x2b97d4f0, 0x8f7521d0
Caught exception in looking up space point ptr... 0x99109af0, 0x9bb047f0
Caught exception in looking up space point ptr... 0x9bb047f0, 0x9bb04980
Caught exception in looking up space point ptr... 0xc18efa00, 0xbec954d0
Caught exception in looking up space point ptr... 0xbec954d0, 0xc219e5e0
Caught exception in looking up space point ptr... 0xb6ac53d0, 0xb6ad02a0
Caught exception in looking up space point ptr... 0xb6ad02a0, 0xb6ac53d0
Caught exception in looking up space point ptr... 0x4f1ad080, 0x4f1b8620
Caught exception in looking up space point ptr... 0x4f1b8620, 0x4f1ad080
Caught exception in looking up space point ptr... 0x3b2c42e0, 0x146debf0
Caught exception in looking up space point ptr... 0x146debf0, 0x3b2c42e0
Caught exception in looking up space point ptr... 0x9eac6ea0, 0x9dabf9a0
Caught exception in looking up space point ptr... 0x9dabf9a0, 0x9eac6ea0
Caught exception in looking up space point ptr... 0x80c33cc0, 0x8f17d800
Caught exception in looking up space point ptr... 0x8f17d800, 0x80c33cc0
++++>>>> total num hits: 7246943, num free: 7065415
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 72.7499 323.44 1118.07 73.7249 405.286 5
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 4.466e-05 9.36522e-05 0.000230271 5.571e-05 6.95784e-05 5
produce:tpcrawdecoder:PDVDTPCReader 4.25691 4.55164 5.1363 4.40654 0.328837 5
produce:triggerrawdecoder:PDVDTriggerReader4 0.0117907 0.0148655 0.0195528 0.014036 0.00259713 5
produce:timingrawdecoder:PDHDTimingRawDecoder 0.000458041 0.00064924 0.000927853 0.000544322 0.000180456 5
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.00163478 0.00216932 0.00264745 0.00216876 0.000348612 5
produce:beamevent:BeamEvent 7.798e-05 0.000296795 0.00101716 0.00013011 0.000360759 5
produce:pdvddaphne:DAPHNEReaderPDVD 1.94534 2.32607 2.82759 2.37581 0.301521 5
produce:ophit:OpHitFinder 0.0307137 0.0349374 0.0429355 0.0341237 0.00433838 5
produce:wclsdatavd:WireCellToolkit 52.2109 53.7455 55.9844 52.7633 1.52191 5
produce:gaushit:GausHitFinder 0.566393 2.8302 6.45849 0.940624 2.59125 5
produce:nhitsfilter:NumberOfHitsFilter 0.000190471 0.000465015 0.00106596 0.000387571 0.00031444 5
produce:reco3d:SpacePointSolver 5.32555 34.4087 129.91 7.75995 48.1013 5
produce:hitpdune:DisambigFromSpacePoints 0.0569208 1.09427 4.41772 0.0681348 1.69047 5
produce:cluster3d:Cluster3D 1.07018 220.668 916.369 1.48554 354.909 5
[art]:TriggerResults:TriggerResultInserter 2.559e-05 4.26702e-05 8.3841e-05 3.238e-05 2.11347e-05 5
end_path:out1:RootOutput 7.56e-06 1.3132e-05 2.519e-05 1.044e-05 6.20885e-06 5
end_path:out1:RootOutput(write) 3.44128 4.67396 7.9509 3.65183 1.895 4
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 15511.1 MB
Peak resident set size usage (VmHWM): 13216.8 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 02:12:55 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 5 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 