Jobsub ID 303653.156@dunegpschedd01.fnal.gov
| Jobsub ID | 303653.156@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:52:24 |
| Site | UK_RAL-PPD |
| Entry | CMSHTPC_T2_UK_SGrid_RALPP_hep207 |
| Last heartbeat | 2026-02-04 00:22:34 |
| From worker node | Hostname | heplnc158.pp.rl.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 23:46:01 |
| Input files | vd-protodune:np02vd_raw_run042421_0010_df-s03-d1_dw_0_20260203T133622.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 00:22:34 |
| Saved logs | justin-logs:303653.156-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
up space point ptr... 0x96907c20, 0x96ab7200
Caught exception in looking up space point ptr... 0xa4891ea0, 0xac652630
Caught exception in looking up space point ptr... 0xac652630, 0xa4891ea0
Caught exception in looking up space point ptr... 0x5ea563d0, 0x5ef5ca00
Caught exception in looking up space point ptr... 0x5ef5ca00, 0x5a57bf30
Caught exception in looking up space point ptr... 0x6c2eb170, 0x6c7ea900
Caught exception in looking up space point ptr... 0x6c7ea900, 0x6c7ea860
Caught exception in looking up space point ptr... 0x7a0aca40, 0x710d98b0
Caught exception in looking up space point ptr... 0x710d98b0, 0x7a1bc7a0
Caught exception in looking up space point ptr... 0x2603d190, 0x269ddf50
Caught exception in looking up space point ptr... 0x269ddf50, 0x2603d060
Caught exception in looking up space point ptr... 0x66d01850, 0x66759380
Caught exception in looking up space point ptr... 0x66759380, 0x534def40
Caught exception in looking up space point ptr... 0x68f929d0, 0x69cc18e0
Caught exception in looking up space point ptr... 0x69cc18e0, 0x69daf2b0
Caught exception in looking up space point ptr... 0x485816c0, 0x482d7bb0
Caught exception in looking up space point ptr... 0x482d7bb0, 0x42aff9c0
Caught exception in looking up space point ptr... 0x36d703c0, 0x36d69550
Caught exception in looking up space point ptr... 0x36d69550, 0x36d5f5e0
Caught exception in looking up space point ptr... 0xa84e3d10, 0xa48d9010
Caught exception in looking up space point ptr... 0xa48d9010, 0xa3d54ec0
Caught exception in looking up space point ptr... 0x8a0bce50, 0x88dd5f10
Caught exception in looking up space point ptr... 0x88dd5f10, 0x8a0bce50
Caught exception in looking up space point ptr... 0x2d4bfbc0, 0x2d49d7d0
Caught exception in looking up space point ptr... 0x2d49d7d0, 0x2d4c6440
Caught exception in looking up space point ptr... 0x636cf6a0, 0x6247dad0
Caught exception in looking up space point ptr... 0x6247dad0, 0x650c37b0
Caught exception in looking up space point ptr... 0x8b02bfd0, 0x8c538d40
Caught exception in looking up space point ptr... 0x8c538d40, 0x8b02bfd0
Caught exception in looking up space point ptr... 0x8b02bfd0, 0x8c453080
Caught exception in looking up space point ptr... 0x8c453080, 0x8b02bfd0
Caught exception in looking up space point ptr... 0x53eddf20, 0x53e60be0
Caught exception in looking up space point ptr... 0x53ed7180, 0x53eddf20
Caught exception in looking up space point ptr... 0xb4302860, 0xb43020b0
Caught exception in looking up space point ptr... 0xb43020b0, 0xb4302340
Caught exception in looking up space point ptr... 0x68261800, 0x5c995120
Caught exception in looking up space point ptr... 0x5c995120, 0x5922ecf0
Caught exception in looking up space point ptr... 0xab06a9f0, 0xaa7650a0
Caught exception in looking up space point ptr... 0xaa7650a0, 0xab06a9f0
Caught exception in looking up space point ptr... 0x9a0708d0, 0x9a03edc0
Caught exception in looking up space point ptr... 0x9a03edc0, 0x9a0708d0
Caught exception in looking up space point ptr... 0x36d60700, 0x36d94760
Caught exception in looking up space point ptr... 0x36d852e0, 0x36d60700
Caught exception in looking up space point ptr... 0x171d3c40, 0x227f5f40
Caught exception in looking up space point ptr... 0x227f5f40, 0x171d3c40
Caught exception in looking up space point ptr... 0x99fb9500, 0x99f6ad90
Caught exception in looking up space point ptr... 0x99f6ad90, 0x99fc7cf0
Caught exception in looking up space point ptr... 0x99fc7cf0, 0x99faebb0
Caught exception in looking up space point ptr... 0x9a6475d0, 0x9a647530
Caught exception in looking up space point ptr... 0x9a7b4cb0, 0x9a6475d0
Caught exception in looking up space point ptr... 0x692d3d40, 0x692d4060
Caught exception in looking up space point ptr... 0x692d4060, 0x692d3d40
Caught exception in looking up space point ptr... 0xacf7b5a0, 0xacf7b410
Caught exception in looking up space point ptr... 0xacf7b410, 0xacf7b5a0
Caught exception in looking up space point ptr... 0x758600a0, 0x6973cea0
Caught exception in looking up space point ptr... 0x6973cea0, 0x758600a0
Caught exception in looking up space point ptr... 0x9e3e56e0, 0x9e3d20e0
Caught exception in looking up space point ptr... 0x9e3d20e0, 0xab713340
Caught exception in looking up space point ptr... 0xa7a8cc90, 0xb472bef0
Caught exception in looking up space point ptr... 0xb472bef0, 0xa6b6eec0
Caught exception in looking up space point ptr... 0xad43fdc0, 0xa8907140
Caught exception in looking up space point ptr... 0xa8907140, 0xa8640b80
Caught exception in looking up space point ptr... 0x2225d7b0, 0x4a1640c0
Caught exception in looking up space point ptr... 0x4a1640c0, 0x20596ac0
Caught exception in looking up space point ptr... 0x20596ac0, 0x28982a20
Caught exception in looking up space point ptr... 0x25acee50, 0x25215dc0
Caught exception in looking up space point ptr... 0x25215dc0, 0x251df0f0
Caught exception in looking up space point ptr... 0x251df0f0, 0x251ea800
Caught exception in looking up space point ptr... 0x251ea800, 0x2516fb60
Caught exception in looking up space point ptr... 0x9de30c50, 0x9e7171f0
Caught exception in looking up space point ptr... 0x9e7171f0, 0x9e77a350
Caught exception in looking up space point ptr... 0x9d452fa0, 0x9d524850
Caught exception in looking up space point ptr... 0x9d524850, 0x9d621920
Caught exception in looking up space point ptr... 0x8d382460, 0x8b9b31f0
Caught exception in looking up space point ptr... 0x8b9b31f0, 0x8b9b6450
Caught exception in looking up space point ptr... 0x40e12b70, 0x28dfd0f0
Caught exception in looking up space point ptr... 0x28dfd0f0, 0x40e12b70
Caught exception in looking up space point ptr... 0x9a0ef2f0, 0x9a11da30
Caught exception in looking up space point ptr... 0x9a11da30, 0x9a0e7010
Caught exception in looking up space point ptr... 0x5c532ef0, 0x5aef71b0
Caught exception in looking up space point ptr... 0x5aef71b0, 0x5c532ef0
++++>>>> total num hits: 7299158, num free: 7116744
04-Feb-2026 00:21:20 GMT Opened output file with pattern "%ifb_reco_stage0_%tc_offline.root"
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 2041.64 2041.64 2041.64 2041.64 0 1
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.000309252 0.000309252 0.000309252 0.000309252 0 1
produce:tpcrawdecoder:PDVDTPCReader 12.178 12.178 12.178 12.178 0 1
produce:triggerrawdecoder:PDVDTriggerReader4 0.534031 0.534031 0.534031 0.534031 0 1
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0187255 0.0187255 0.0187255 0.0187255 0 1
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0894665 0.0894665 0.0894665 0.0894665 0 1
produce:beamevent:BeamEvent 0.00153791 0.00153791 0.00153791 0.00153791 0 1
produce:pdvddaphne:DAPHNEReaderPDVD 3.59062 3.59062 3.59062 3.59062 0 1
produce:ophit:OpHitFinder 0.0471462 0.0471462 0.0471462 0.0471462 0 1
produce:wclsdatavd:WireCellToolkit 93.4707 93.4707 93.4707 93.4707 0 1
produce:gaushit:GausHitFinder 21.5008 21.5008 21.5008 21.5008 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.00181646 0.00181646 0.00181646 0.00181646 0 1
produce:reco3d:SpacePointSolver 194.986 194.986 194.986 194.986 0 1
produce:hitpdune:DisambigFromSpacePoints 7.09542 7.09542 7.09542 7.09542 0 1
produce:cluster3d:Cluster3D 1708.04 1708.04 1708.04 1708.04 0 1
[art]:TriggerResults:TriggerResultInserter 0.000151966 0.000151966 0.000151966 0.000151966 0 1
end_path:out1:RootOutput 5.0806e-05 5.0806e-05 5.0806e-05 5.0806e-05 0 1
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 16628.3 MB
Peak resident set size usage (VmHWM): 14578.9 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:21:44 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 1 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 