Jobsub ID 298054.2@dunegpschedd02.fnal.gov
| Jobsub ID | 298054.2@dunegpschedd02.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-04 02:44:34 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce01 |
| Last heartbeat | 2026-02-04 03:21:06 |
| From worker node | Hostname | dune001-1526419.0-lcg2398.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-04 02:45:18 |
| Input files | vd-protodune:np02vd_raw_run042421_0010_df-s02-d0_dw_0_20260203T133622.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-04 03:21:06 |
| Saved logs | justin-logs:298054.2-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
up space point ptr... 0x74054a90, 0x74054940
Caught exception in looking up space point ptr... 0x74054940, 0x740546a0
Caught exception in looking up space point ptr... 0x83cf1ba0, 0x848570f0
Caught exception in looking up space point ptr... 0x84856ae0, 0x83cf1ba0
Caught exception in looking up space point ptr... 0x743407f0, 0x7b3f9cd0
Caught exception in looking up space point ptr... 0x7b3f9cd0, 0x7b59b470
Caught exception in looking up space point ptr... 0xc1396b40, 0xc21e46b0
Caught exception in looking up space point ptr... 0xc21e46b0, 0xc21f30f0
Caught exception in looking up space point ptr... 0xc0c69550, 0xc1396b40
Caught exception in looking up space point ptr... 0x6c3f3250, 0x5df24000
Caught exception in looking up space point ptr... 0x6c628890, 0x6c3f3250
Caught exception in looking up space point ptr... 0xcb59b8a0, 0xbdafb700
Caught exception in looking up space point ptr... 0xbdafb700, 0xc2352710
Caught exception in looking up space point ptr... 0xc9383200, 0xc941c9d0
Caught exception in looking up space point ptr... 0xc941c9d0, 0xc9383200
Caught exception in looking up space point ptr... 0x41d99330, 0x41d99590
Caught exception in looking up space point ptr... 0x41d99460, 0x41d99330
Caught exception in looking up space point ptr... 0xc1f90230, 0xc0ce3fd0
Caught exception in looking up space point ptr... 0xc0ce3fd0, 0xcb768370
Caught exception in looking up space point ptr... 0xc9daa0d0, 0xcaa7db10
Caught exception in looking up space point ptr... 0xcaa7db10, 0xc9daa0d0
Caught exception in looking up space point ptr... 0x76466aa0, 0x7bd345e0
Caught exception in looking up space point ptr... 0x7bd345e0, 0x7b2d1660
Caught exception in looking up space point ptr... 0x5fd18760, 0x58facc60
Caught exception in looking up space point ptr... 0x58facc60, 0x5fd185d0
Caught exception in looking up space point ptr... 0xd1b07830, 0xd1ab1350
Caught exception in looking up space point ptr... 0xd1ab0e30, 0xd1b07830
Caught exception in looking up space point ptr... 0xd1aac850, 0xd1b18450
Caught exception in looking up space point ptr... 0xd195a550, 0xd1aac850
Caught exception in looking up space point ptr... 0x76a312e0, 0x81a02600
Caught exception in looking up space point ptr... 0x81a02600, 0x76f77860
Caught exception in looking up space point ptr... 0x7d7fe3d0, 0x3e459cf0
Caught exception in looking up space point ptr... 0x3e459cf0, 0x7dde7300
Caught exception in looking up space point ptr... 0x53949c60, 0x3b02f510
Caught exception in looking up space point ptr... 0x3b02f510, 0x3ab594a0
Caught exception in looking up space point ptr... 0x3baf9300, 0x5d91fbd0
Caught exception in looking up space point ptr... 0x5d91fbd0, 0x65a1e690
Caught exception in looking up space point ptr... 0x74ed7960, 0x748fcf50
Caught exception in looking up space point ptr... 0x748fcf50, 0x74ed7960
Caught exception in looking up space point ptr... 0xb49454f0, 0xb48a0340
Caught exception in looking up space point ptr... 0xb48a0340, 0xb49454f0
Caught exception in looking up space point ptr... 0xb49454f0, 0xb48a04e0
Caught exception in looking up space point ptr... 0xb48a04e0, 0xb49454f0
Caught exception in looking up space point ptr... 0x7c277890, 0x41675e00
Caught exception in looking up space point ptr... 0x41675e00, 0x7c277890
Caught exception in looking up space point ptr... 0x716e21e0, 0x71d51370
Caught exception in looking up space point ptr... 0x71d51370, 0x71d51500
Caught exception in looking up space point ptr... 0xb49c96a0, 0xb49cdb70
Caught exception in looking up space point ptr... 0xb49cdb70, 0xb49c96a0
Caught exception in looking up space point ptr... 0x4bbc8620, 0x4bbc87b0
Caught exception in looking up space point ptr... 0x4bbc87b0, 0x4bbc8940
Caught exception in looking up space point ptr... 0xb48802b0, 0xb4883970
Caught exception in looking up space point ptr... 0xb4883970, 0xb48802b0
Caught exception in looking up space point ptr... 0xb2afe6e0, 0xb24d4980
Caught exception in looking up space point ptr... 0xb24d4980, 0xb2afe6e0
Caught exception in looking up space point ptr... 0xbf5c4e80, 0xc8cf8dc0
Caught exception in looking up space point ptr... 0xc8cf8dc0, 0xbf5c4e80
Caught exception in looking up space point ptr... 0x74cf2ba0, 0x73eb1c50
Caught exception in looking up space point ptr... 0x73eb1c50, 0x74cf2ba0
Caught exception in looking up space point ptr... 0xbcaa7f30, 0xbd246f60
Caught exception in looking up space point ptr... 0xbd246f60, 0xbd215e00
Caught exception in looking up space point ptr... 0xc74ac250, 0xc7481f20
Caught exception in looking up space point ptr... 0xc7481f20, 0xc74ac250
Caught exception in looking up space point ptr... 0x4bbc6e60, 0x4bbc6cd0
Caught exception in looking up space point ptr... 0x4bbc6cd0, 0x4bbc6960
Caught exception in looking up space point ptr... 0xc9c5d760, 0xc9fc4940
Caught exception in looking up space point ptr... 0xc9fc4940, 0xc9c5d760
Caught exception in looking up space point ptr... 0x9dc81640, 0xa46218a0
Caught exception in looking up space point ptr... 0xa46218a0, 0xa758db10
Caught exception in looking up space point ptr... 0x4bbc7e50, 0x4bbc8170
Caught exception in looking up space point ptr... 0x4bbc8170, 0x4bbc7e50
Caught exception in looking up space point ptr... 0x63e17c50, 0x63e17ac0
Caught exception in looking up space point ptr... 0x63e17ac0, 0x63e17c50
Caught exception in looking up space point ptr... 0xa92f8a50, 0xa43b38e0
Caught exception in looking up space point ptr... 0xa43b38e0, 0xa3d6c680
Caught exception in looking up space point ptr... 0x6164b4f0, 0x6164b810
Caught exception in looking up space point ptr... 0x6164b810, 0x6164b4f0
Caught exception in looking up space point ptr... 0xabe56df0, 0xb07fd790
Caught exception in looking up space point ptr... 0xb07fd790, 0xabe56df0
Caught exception in looking up space point ptr... 0x7dbfdc60, 0x7a7635e0
Caught exception in looking up space point ptr... 0x7a7635e0, 0x7dbfdc60
++++>>>> total num hits: 7289782, num free: 7102555
04-Feb-2026 03:19:53 UTC Opened output file with pattern "%ifb_reco_stage0_%tc_offline.root"
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 1990.7 1990.7 1990.7 1990.7 0 1
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 0.00028134 0.00028134 0.00028134 0.00028134 0 1
produce:tpcrawdecoder:PDVDTPCReader 14.6327 14.6327 14.6327 14.6327 0 1
produce:triggerrawdecoder:PDVDTriggerReader4 0.71738 0.71738 0.71738 0.71738 0 1
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0240983 0.0240983 0.0240983 0.0240983 0 1
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.117845 0.117845 0.117845 0.117845 0 1
produce:beamevent:BeamEvent 0.00232962 0.00232962 0.00232962 0.00232962 0 1
produce:pdvddaphne:DAPHNEReaderPDVD 3.67206 3.67206 3.67206 3.67206 0 1
produce:ophit:OpHitFinder 0.0433041 0.0433041 0.0433041 0.0433041 0 1
produce:wclsdatavd:WireCellToolkit 85.5333 85.5333 85.5333 85.5333 0 1
produce:gaushit:GausHitFinder 12.9573 12.9573 12.9573 12.9573 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.00218361 0.00218361 0.00218361 0.00218361 0 1
produce:reco3d:SpacePointSolver 222.653 222.653 222.653 222.653 0 1
produce:hitpdune:DisambigFromSpacePoints 8.4003 8.4003 8.4003 8.4003 0 1
produce:cluster3d:Cluster3D 1641.85 1641.85 1641.85 1641.85 0 1
[art]:TriggerResults:TriggerResultInserter 0.000142379 0.000142379 0.000142379 0.000142379 0 1
end_path:out1:RootOutput 5.0194e-05 5.0194e-05 5.0194e-05 5.0194e-05 0 1
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 16921.6 MB
Peak resident set size usage (VmHWM): 14594.5 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 03:20:13 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 1 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 