Jobsub ID 298035.6@dunegpschedd02.fnal.gov
| Jobsub ID | 298035.6@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-04 02:12:32 |
| Site | ES_PIC |
| Entry | DUNE_T1_ES_PIC_ce16-multicore |
| Last heartbeat | 2026-02-04 05:41:50 |
| From worker node | Hostname | hnode66.pic.es |
| cpuinfo | AMD EPYC 7402P 24-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 02:13:37 |
| Input files | vd-protodune:np02vd_raw_run042421_0054_df-s04-d1_dw_0_20260203T140015.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 05:41:50 |
| Saved logs | justin-logs:298035.6-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ng up space point ptr... 0x1539c35d0, 0x154032b90
Caught exception in looking up space point ptr... 0x1880fde10, 0x187b7a260
Caught exception in looking up space point ptr... 0x187b7a260, 0x1880fde10
Caught exception in looking up space point ptr... 0x5df48dd0, 0x419f3530
Caught exception in looking up space point ptr... 0x419f3530, 0xf01d5f20
Caught exception in looking up space point ptr... 0x126e00770, 0x152e1b2a0
Caught exception in looking up space point ptr... 0x152e1b2a0, 0x126e00770
Caught exception in looking up space point ptr... 0x157b3b080, 0x156e033f0
Caught exception in looking up space point ptr... 0x156e033f0, 0x157b3b080
Caught exception in looking up space point ptr... 0x102452a40, 0x102452f60
Caught exception in looking up space point ptr... 0x102452f60, 0x10259c970
Caught exception in looking up space point ptr... 0x154b41970, 0x153122820
Caught exception in looking up space point ptr... 0x153122820, 0x1533963b0
Caught exception in looking up space point ptr... 0x15f5e4840, 0x15f5e4a90
Caught exception in looking up space point ptr... 0x15f5e45f0, 0x15f5e4840
Caught exception in looking up space point ptr... 0xf3dc5c60, 0x102899f60
Caught exception in looking up space point ptr... 0x102899f60, 0xf352a110
Caught exception in looking up space point ptr... 0x1241ca3e0, 0x107302c00
Caught exception in looking up space point ptr... 0x107302c00, 0x1471bcfe0
Caught exception in looking up space point ptr... 0x29f56a70, 0x5ffe1970
Caught exception in looking up space point ptr... 0x5ffe1970, 0x29f56a70
Caught exception in looking up space point ptr... 0x63d5ce70, 0x46320b00
Caught exception in looking up space point ptr... 0x4f723220, 0x63d5ce70
Caught exception in looking up space point ptr... 0x6e736e90, 0x6e6c43a0
Caught exception in looking up space point ptr... 0x6e6c43a0, 0x6e719f00
Caught exception in looking up space point ptr... 0x1abba5d50, 0x1abba7d10
Caught exception in looking up space point ptr... 0x1abba7d10, 0x1abba5290
Caught exception in looking up space point ptr... 0x147ca2e60, 0xf61e3350
Caught exception in looking up space point ptr... 0xf61e3350, 0x147b5e7a0
Caught exception in looking up space point ptr... 0x18d553380, 0x18d6a7bc0
Caught exception in looking up space point ptr... 0x18d49acb0, 0x18d553380
Caught exception in looking up space point ptr... 0x193c313a0, 0x18e57ebc0
Caught exception in looking up space point ptr... 0x18e57ebc0, 0x193c313a0
Caught exception in looking up space point ptr... 0x3669dd60, 0x3669e2d0
Caught exception in looking up space point ptr... 0x3669e2d0, 0x3669dff0
Caught exception in looking up space point ptr... 0x1a989ab50, 0x1a98a5a40
Caught exception in looking up space point ptr... 0x1a98a5a40, 0x1a97ffb10
Caught exception in looking up space point ptr... 0x1562458c0, 0x153f20980
Caught exception in looking up space point ptr... 0x153f20980, 0x156417e70
Caught exception in looking up space point ptr... 0x6e3e4910, 0x6e240c90
Caught exception in looking up space point ptr... 0x6e240c90, 0x6e3e4910
Caught exception in looking up space point ptr... 0x1000d7c00, 0xff550130
Caught exception in looking up space point ptr... 0xff550130, 0xf6773aa0
Caught exception in looking up space point ptr... 0x190f5a580, 0x190fa6600
Caught exception in looking up space point ptr... 0x190fa6600, 0x190f60430
Caught exception in looking up space point ptr... 0x19b538da0, 0x19b3844d0
Caught exception in looking up space point ptr... 0x19b523230, 0x19b538da0
Caught exception in looking up space point ptr... 0x1502171b0, 0x150217670
Caught exception in looking up space point ptr... 0x1501fee10, 0x1502171b0
Caught exception in looking up space point ptr... 0x35a63040, 0x35a632d0
Caught exception in looking up space point ptr... 0x35a632d0, 0x35a62d60
Caught exception in looking up space point ptr... 0x123af4930, 0xf75f9dd0
Caught exception in looking up space point ptr... 0xf75f9dd0, 0x123af4930
Caught exception in looking up space point ptr... 0x190d57350, 0x190c56ce0
Caught exception in looking up space point ptr... 0x12df875d0, 0x190d57350
Caught exception in looking up space point ptr... 0x123ff6720, 0x12320c1f0
Caught exception in looking up space point ptr... 0x12320c1f0, 0x12320c0b0
Caught exception in looking up space point ptr... 0x100dd7240, 0x3c06f050
Caught exception in looking up space point ptr... 0x3c06f050, 0x100dd7240
Caught exception in looking up space point ptr... 0x14fe63470, 0x141dd9c90
Caught exception in looking up space point ptr... 0x141dd9c90, 0x141f50960
Caught exception in looking up space point ptr... 0x141f50960, 0x14fbda820
Caught exception in looking up space point ptr... 0x14fbda820, 0x14fe4e000
Caught exception in looking up space point ptr... 0x2b7cadb0, 0x3f7d5c10
Caught exception in looking up space point ptr... 0x3f7d5c10, 0x3f7dc430
Caught exception in looking up space point ptr... 0x3f7dc430, 0x49865990
Caught exception in looking up space point ptr... 0x49865990, 0x2b7cadb0
Caught exception in looking up space point ptr... 0x14786b530, 0x146af3d90
Caught exception in looking up space point ptr... 0x146af3d90, 0x147d45c10
Caught exception in looking up space point ptr... 0x157311a60, 0x14c3d2570
Caught exception in looking up space point ptr... 0x14c3d2570, 0x157311a60
Caught exception in looking up space point ptr... 0x4b663970, 0x4af58ca0
Caught exception in looking up space point ptr... 0x4af58ca0, 0x4b663970
Caught exception in looking up space point ptr... 0x19db0a610, 0x194a12bf0
Caught exception in looking up space point ptr... 0x194a12bf0, 0x19db0a610
Caught exception in looking up space point ptr... 0x19db0a610, 0x174dd3170
Caught exception in looking up space point ptr... 0x174dd3170, 0x19db0a610
Caught exception in looking up space point ptr... 0x10141c660, 0x100ca9600
Caught exception in looking up space point ptr... 0x100ca9600, 0x3bb8cce0
++++>>>> total num hits: 7485247, num free: 7274365
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 254.146 513.392 2028.24 409.189 425.261 24
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.6152e-05 9.65368e-05 0.00028809 8.8758e-05 4.11016e-05 24
produce:tpcrawdecoder:PDVDTPCReader 152.72 262.379 335.246 282.395 57.178 24
produce:triggerrawdecoder:PDVDTriggerReader4 1.27442 1.60064 2.17273 1.58289 0.221979 24
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0363114 0.0509698 0.122168 0.0456974 0.0202829 24
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.180345 0.192541 0.264032 0.181 0.0219952 24
produce:beamevent:BeamEvent 8.6893e-05 0.000165377 0.00120559 0.000116544 0.000218392 24
produce:pdvddaphne:DAPHNEReaderPDVD 17.023 30.4787 40.8037 32.1178 6.23323 24
produce:ophit:OpHitFinder 0.029238 0.0393246 0.0480622 0.0393809 0.00476639 24
produce:wclsdatavd:WireCellToolkit 62.6787 75.0273 86.6668 75.2229 8.05479 24
produce:gaushit:GausHitFinder 0.576811 1.7504 11.0597 1.00343 2.42169 24
produce:nhitsfilter:NumberOfHitsFilter 0.000192027 0.000449882 0.00180894 0.000314396 0.000386114 24
produce:reco3d:SpacePointSolver 4.21519 23.6635 195.512 7.45588 51.7167 24
produce:hitpdune:DisambigFromSpacePoints 0.0463831 0.880766 10.8346 0.0903384 2.61532 24
produce:cluster3d:Cluster3D 0.72432 112.593 1380.9 1.7737 367.325 24
[art]:TriggerResults:TriggerResultInserter 2.0341e-05 3.11866e-05 7.9653e-05 2.9306e-05 1.26413e-05 24
end_path:out1:RootOutput 3.28e-06 4.94967e-06 2.1301e-05 4.2805e-06 3.45191e-06 24
end_path:out1:RootOutput(write) 3.22349 4.91492 23.5892 4.05227 4.03454 23
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 19368.8 MB
Peak resident set size usage (VmHWM): 16896.7 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 06:40:53 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 23 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 24 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 