Jobsub ID 298112.1@dunegpschedd02.fnal.gov
| Jobsub ID | 298112.1@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-04 04:20:38 |
| Site | UK_Manchester |
| Entry | UBoone_T2_UK_Manchester_ce01 |
| Last heartbeat | 2026-02-04 06:14:05 |
| From worker node | Hostname | wn2209241.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7513 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 04:21:46 |
| Input files | vd-protodune:np02vd_raw_run042421_0050_df-s04-d0_dw_0_20260203T135802.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 06:14:05 |
| Saved logs | justin-logs:298112.1-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
tr... 0xfc2fb150, 0xc060bad0
Caught exception in looking up space point ptr... 0x1e8d171d0, 0x8cd6b620
Caught exception in looking up space point ptr... 0x8cd6b620, 0x1e8d171d0
Caught exception in looking up space point ptr... 0x16a0a15f0, 0xbadd6b00
Caught exception in looking up space point ptr... 0xc0713a70, 0x16a0a15f0
Caught exception in looking up space point ptr... 0x17cc4c640, 0x1ea0286c0
Caught exception in looking up space point ptr... 0x1ea0286c0, 0x17cc4c640
Caught exception in looking up space point ptr... 0x1b09c7610, 0x1b09c7480
Caught exception in looking up space point ptr... 0x1b09c7480, 0x1b09c7610
Caught exception in looking up space point ptr... 0x194b3cea0, 0x1946770b0
Caught exception in looking up space point ptr... 0x1946770b0, 0x194b3cfa0
Caught exception in looking up space point ptr... 0x256a13a0, 0x2c253cd0
Caught exception in looking up space point ptr... 0x203642730, 0x256a13a0
Caught exception in looking up space point ptr... 0x7675b360, 0x17ae2dce0
Caught exception in looking up space point ptr... 0x17ae2dce0, 0x10fe45660
Caught exception in looking up space point ptr... 0xde360f0, 0xde35f60
Caught exception in looking up space point ptr... 0xde35f60, 0xde360f0
Caught exception in looking up space point ptr... 0xde360f0, 0xd8ce540
Caught exception in looking up space point ptr... 0xd8ce540, 0xde360f0
Caught exception in looking up space point ptr... 0xde360f0, 0xd8ce860
Caught exception in looking up space point ptr... 0xd8ce860, 0xde360f0
Caught exception in looking up space point ptr... 0x1db3fd6d0, 0x496837e0
Caught exception in looking up space point ptr... 0xd955750, 0x1db3fd6d0
Caught exception in looking up space point ptr... 0x132c6630, 0x13b28cc80
Caught exception in looking up space point ptr... 0x44b567e0, 0x132c6630
Caught exception in looking up space point ptr... 0x1ea274130, 0x55ff35a0
Caught exception in looking up space point ptr... 0x55ff35a0, 0x97365840
Caught exception in looking up space point ptr... 0x19094dc20, 0x19094db20
Caught exception in looking up space point ptr... 0x19094db20, 0x19085d230
Caught exception in looking up space point ptr... 0x175753750, 0x175f26cf0
Caught exception in looking up space point ptr... 0x16a3dc9a0, 0x175753750
Caught exception in looking up space point ptr... 0x566e2020, 0x44778da0
Caught exception in looking up space point ptr... 0x44778da0, 0x1ccae19a0
Caught exception in looking up space point ptr... 0x1ccae19a0, 0x335b85c0
Caught exception in looking up space point ptr... 0x3abc6790, 0x16bebb030
Caught exception in looking up space point ptr... 0x16bebb030, 0x3abc6790
Caught exception in looking up space point ptr... 0x39565c30, 0x39566150
Caught exception in looking up space point ptr... 0x39566150, 0x39565c30
Caught exception in looking up space point ptr... 0x2540af60, 0x446c2460
Caught exception in looking up space point ptr... 0x446c2460, 0xbc0f0d80
Caught exception in looking up space point ptr... 0x2fd8ba00, 0x2013cdaa0
Caught exception in looking up space point ptr... 0x2013cdaa0, 0x2540af60
Caught exception in looking up space point ptr... 0x39c7f3e0, 0x10fdb2af0
Caught exception in looking up space point ptr... 0x10fdb2af0, 0x39c7f3e0
Caught exception in looking up space point ptr... 0x39c7f3e0, 0x14a196fc0
Caught exception in looking up space point ptr... 0x14a196fc0, 0x39c7f3e0
Caught exception in looking up space point ptr... 0x1cfdfde30, 0x1c6897690
Caught exception in looking up space point ptr... 0x1c6897690, 0x1cfdfde30
Caught exception in looking up space point ptr... 0x169edc320, 0xba6150a0
Caught exception in looking up space point ptr... 0xba6150a0, 0x169edc320
Caught exception in looking up space point ptr... 0x9eaa7180, 0xc0834130
Caught exception in looking up space point ptr... 0xc0834130, 0x9eaa7180
Caught exception in looking up space point ptr... 0xbbd3c7d0, 0x44bc69e0
Caught exception in looking up space point ptr... 0x44bc69e0, 0x3e1a7d70
Caught exception in looking up space point ptr... 0x6c9aad70, 0x4bcd4650
Caught exception in looking up space point ptr... 0x4bcd4650, 0x6c9aad70
Caught exception in looking up space point ptr... 0x1f7c68780, 0x1f7c685f0
Caught exception in looking up space point ptr... 0x1f7c685f0, 0x1f7c68780
Caught exception in looking up space point ptr... 0x1a9ac12a0, 0x38ee27f0
Caught exception in looking up space point ptr... 0x38ee27f0, 0x1a9ac12a0
Caught exception in looking up space point ptr... 0x9d412400, 0x9d449fa0
Caught exception in looking up space point ptr... 0x9d449fa0, 0x9d412400
Caught exception in looking up space point ptr... 0x1a7058170, 0x1a6f93510
Caught exception in looking up space point ptr... 0x1a6f93510, 0x1a7058170
Caught exception in looking up space point ptr... 0x3533f560, 0x8955fec0
Caught exception in looking up space point ptr... 0x3533c610, 0x3533f560
Caught exception in looking up space point ptr... 0xbc889d50, 0xbc889bc0
Caught exception in looking up space point ptr... 0xbc889bc0, 0xbc889d50
Caught exception in looking up space point ptr... 0x148a7b480, 0x14ab42b90
Caught exception in looking up space point ptr... 0x14ab42b90, 0x148a7b480
Caught exception in looking up space point ptr... 0x38e6a9b0, 0x564037d0
Caught exception in looking up space point ptr... 0x564037d0, 0x56404000
Caught exception in looking up space point ptr... 0x12e69c850, 0x10f7e1be0
Caught exception in looking up space point ptr... 0x10f7e1be0, 0x12e69c850
Caught exception in looking up space point ptr... 0x1105fa260, 0x463611c0
Caught exception in looking up space point ptr... 0x463611c0, 0x46361530
Caught exception in looking up space point ptr... 0x1bfd077d0, 0x1bfd07640
Caught exception in looking up space point ptr... 0x1bfd07640, 0x1bfd077d0
Caught exception in looking up space point ptr... 0x13802190, 0x13807b10
Caught exception in looking up space point ptr... 0x13807b10, 0x137cf420
++++>>>> total num hits: 8369485, num free: 8143834
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 219.213 1608.38 2535.79 1839.27 877.787 4
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.6672e-05 0.00030025 0.000783555 0.000160386 0.00028038 4
produce:tpcrawdecoder:PDVDTPCReader 113.017 153.883 213.321 144.598 42.16 4
produce:triggerrawdecoder:PDVDTriggerReader4 0.892502 1.04766 1.24489 1.02662 0.135014 4
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0221571 0.023707 0.0279449 0.0223629 0.00244824 4
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.108617 0.108981 0.109393 0.108958 0.000287098 4
produce:beamevent:BeamEvent 0.000142107 0.00054081 0.00172182 0.000149656 0.000681872 4
produce:pdvddaphne:DAPHNEReaderPDVD 13.8179 20.3139 26.4356 20.5011 4.62379 4
produce:ophit:OpHitFinder 0.0377955 0.0454728 0.0492056 0.0474451 0.00455611 4
produce:wclsdatavd:WireCellToolkit 70.5228 87.8837 102.886 89.0629 11.7985 4
produce:gaushit:GausHitFinder 0.721911 13.5461 20.7902 16.3361 7.93477 4
produce:nhitsfilter:NumberOfHitsFilter 0.000671914 0.00135176 0.00171187 0.00151162 0.000405281 4
produce:reco3d:SpacePointSolver 8.59445 161.251 236.87 199.769 89.7776 4
produce:hitpdune:DisambigFromSpacePoints 0.106075 7.14385 11.0134 8.72798 4.17181 4
produce:cluster3d:Cluster3D 1.48047 1142.96 1962.41 1303.98 733.749 4
[art]:TriggerResults:TriggerResultInserter 3.9604e-05 6.07118e-05 9.7584e-05 5.28295e-05 2.21018e-05 4
end_path:out1:RootOutput 6.392e-06 1.28842e-05 2.8123e-05 8.511e-06 8.84958e-06 4
end_path:out1:RootOutput(write) 4.87502 26.8538 42.7695 32.9167 16.0534 3
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 21624 MB
Peak resident set size usage (VmHWM): 19127 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 06:11:00 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 4 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 