Jobsub ID 303647.94@dunegpschedd01.fnal.gov
| Jobsub ID | 303647.94@dunegpschedd01.fnal.gov |
| Workflow ID | 12717 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 22:46:24 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce513 |
| Last heartbeat | 2026-02-03 23:48:01 |
| From worker node | Hostname | b9p12p8471.cern.ch |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 22:47:36 |
| Input files | vd-protodune:np02vd_raw_run042421_0009_df-s02-d1_dw_0_20260203T133547.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-03 23:48:01 |
| Saved logs | justin-logs:303647.94-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
0x7ec17a80
Caught exception in looking up space point ptr... 0x7ec17a80, 0x7ef94060
Caught exception in looking up space point ptr... 0xd1ef87a0, 0xd1f31cb0
Caught exception in looking up space point ptr... 0xd1f31cb0, 0xd1ef87a0
Caught exception in looking up space point ptr... 0x5a4b3da0, 0x52a66500
Caught exception in looking up space point ptr... 0x52a66500, 0x5110b420
Caught exception in looking up space point ptr... 0xddf808a0, 0xddf7f090
Caught exception in looking up space point ptr... 0xddf7eb70, 0xddf808a0
Caught exception in looking up space point ptr... 0xb71308f0, 0xb7135d40
Caught exception in looking up space point ptr... 0xb71a6e40, 0xb71308f0
Caught exception in looking up space point ptr... 0x2ee0e690, 0x424b5010
Caught exception in looking up space point ptr... 0x424b5010, 0x20e72fc0
Caught exception in looking up space point ptr... 0xcfa658a0, 0xd13905a0
Caught exception in looking up space point ptr... 0xd018d140, 0xcfa658a0
Caught exception in looking up space point ptr... 0x3b8f88b0, 0x3b8f8590
Caught exception in looking up space point ptr... 0x3b8f8590, 0x3b8f88b0
Caught exception in looking up space point ptr... 0xbcab82a0, 0xbcab8040
Caught exception in looking up space point ptr... 0xbcab8040, 0xbcab8170
Caught exception in looking up space point ptr... 0x66fe47c0, 0x66fe4ce0
Caught exception in looking up space point ptr... 0x66fe4a50, 0x66fe47c0
Caught exception in looking up space point ptr... 0xbcbeea80, 0xbcbeebb0
Caught exception in looking up space point ptr... 0xbcbeebb0, 0xbcbbbf10
Caught exception in looking up space point ptr... 0xb60927b0, 0xb60ae0f0
Caught exception in looking up space point ptr... 0xb6096000, 0xb60927b0
Caught exception in looking up space point ptr... 0x40eec560, 0x7118f000
Caught exception in looking up space point ptr... 0x40f2fed0, 0x40eec560
Caught exception in looking up space point ptr... 0x8e14eaf0, 0x829c0580
Caught exception in looking up space point ptr... 0x829c0580, 0x8e14eaf0
Caught exception in looking up space point ptr... 0x10460750, 0x43c22140
Caught exception in looking up space point ptr... 0x43c26460, 0x10460750
Caught exception in looking up space point ptr... 0x68e67490, 0x62d9d780
Caught exception in looking up space point ptr... 0x62d9d4f0, 0x68e67490
Caught exception in looking up space point ptr... 0x48138610, 0x481382f0
Caught exception in looking up space point ptr... 0x481382f0, 0x48138610
Caught exception in looking up space point ptr... 0x14f20640, 0x571743f0
Caught exception in looking up space point ptr... 0x571743f0, 0x14f1f4d0
Caught exception in looking up space point ptr... 0x9fc9f20, 0x14f67140
Caught exception in looking up space point ptr... 0x14f67140, 0x14f25d00
Caught exception in looking up space point ptr... 0x91f06610, 0xa2991e70
Caught exception in looking up space point ptr... 0xa2991e70, 0xa56e1620
Caught exception in looking up space point ptr... 0xa0b7f8d0, 0xa3a7e120
Caught exception in looking up space point ptr... 0xa3a7e120, 0xa212ea00
Caught exception in looking up space point ptr... 0x7ad9000, 0x4e537230
Caught exception in looking up space point ptr... 0x4e537230, 0x7ad9000
Caught exception in looking up space point ptr... 0xb80df3e0, 0xac1f8ee0
Caught exception in looking up space point ptr... 0xac1f8ee0, 0xb80df3e0
Caught exception in looking up space point ptr... 0x8f38fb10, 0x77575bc0
Caught exception in looking up space point ptr... 0xa2a31ad0, 0x8f38fb10
Caught exception in looking up space point ptr... 0xa3aa89b0, 0xa106e850
Caught exception in looking up space point ptr... 0xa106e850, 0xa3aa89b0
Caught exception in looking up space point ptr... 0x10395ec0, 0x4d811690
Caught exception in looking up space point ptr... 0x4d811690, 0x45323950
Caught exception in looking up space point ptr... 0x45323950, 0x21d39c40
Caught exception in looking up space point ptr... 0x21d39c40, 0x4d811690
Caught exception in looking up space point ptr... 0x4d811690, 0x718222e0
Caught exception in looking up space point ptr... 0x718222e0, 0x4d811690
Caught exception in looking up space point ptr... 0x4d811690, 0x2df2f4e0
Caught exception in looking up space point ptr... 0x71b62a20, 0x5bdd7ac0
Caught exception in looking up space point ptr... 0x5bdd7ac0, 0x21be4140
Caught exception in looking up space point ptr... 0xbf1be620, 0xbe2ad870
Caught exception in looking up space point ptr... 0xbe2ad870, 0xbe2ad780
Caught exception in looking up space point ptr... 0xb5543940, 0xb4427ff0
Caught exception in looking up space point ptr... 0xb4427ff0, 0xb56d7dd0
Caught exception in looking up space point ptr... 0xb6a398d0, 0xb7b20600
Caught exception in looking up space point ptr... 0xb7b20600, 0xb6a398d0
Caught exception in looking up space point ptr... 0x31498c20, 0x51296960
Caught exception in looking up space point ptr... 0x51296960, 0x31498c20
Caught exception in looking up space point ptr... 0x31498c20, 0x61201300
Caught exception in looking up space point ptr... 0x61201300, 0x31498c20
Caught exception in looking up space point ptr... 0xcb062980, 0xca2c2fd0
Caught exception in looking up space point ptr... 0xca2c2fd0, 0xcb062980
Caught exception in looking up space point ptr... 0xbae58f80, 0xbae59250
Caught exception in looking up space point ptr... 0xbae59250, 0xbae59070
Caught exception in looking up space point ptr... 0x54a96b50, 0x30cd9d60
Caught exception in looking up space point ptr... 0x30cd9d60, 0x54a96b50
Caught exception in looking up space point ptr... 0x31ee58c0, 0x394be420
Caught exception in looking up space point ptr... 0x394be420, 0x3953c3e0
Caught exception in looking up space point ptr... 0x63cba2f0, 0x63cb9fd0
Caught exception in looking up space point ptr... 0x63cb9fd0, 0x63cba2f0
Caught exception in looking up space point ptr... 0x3263a030, 0x25cb6a40
Caught exception in looking up space point ptr... 0x25cb6a40, 0x3263a030
++++>>>> total num hits: 7804279, num free: 7609435
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 69.8689 134.988 1415.16 85.3313 256.182 26
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 4.252e-05 8.88522e-05 0.000264383 7.7536e-05 4.88125e-05 26
produce:tpcrawdecoder:PDVDTPCReader 3.79834 4.66431 6.81672 4.43887 0.729655 26
produce:triggerrawdecoder:PDVDTriggerReader4 0.0106176 0.0166316 0.0578439 0.0125934 0.00976849 26
produce:timingrawdecoder:PDHDTimingRawDecoder 0.000402685 0.00115236 0.0171659 0.00046626 0.00320604 26
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0014241 0.00184154 0.00421616 0.00172292 0.000499876 26
produce:beamevent:BeamEvent 6.0051e-05 0.000115335 0.00082255 8.4351e-05 0.000142814 26
produce:pdvddaphne:DAPHNEReaderPDVD 2.09491 2.9072 3.43111 2.92669 0.290129 26
produce:ophit:OpHitFinder 0.0257968 0.0372835 0.0463308 0.0375062 0.0054018 26
produce:wclsdatavd:WireCellToolkit 50.8786 59.3485 66.4134 59.081 4.18401 26
produce:gaushit:GausHitFinder 0.566183 1.38461 14.0086 0.824667 2.54921 26
produce:nhitsfilter:NumberOfHitsFilter 0.000149741 0.000287742 0.00107213 0.000242843 0.000175879 26
produce:reco3d:SpacePointSolver 5.00593 16.6096 161.306 10.0205 29.3351 26
produce:hitpdune:DisambigFromSpacePoints 0.0519252 0.24944 4.3225 0.0786538 0.815382 26
produce:cluster3d:Cluster3D 0.809505 46.3703 1162.17 1.84699 223.161 26
[art]:TriggerResults:TriggerResultInserter 1.5411e-05 3.21009e-05 0.000100791 2.9175e-05 1.63063e-05 26
end_path:out1:RootOutput 4.27e-06 7.44985e-06 3.0691e-05 6.2455e-06 4.96348e-06 26
end_path:out1:RootOutput(write) 2.8493 3.51321 4.31993 3.41125 0.471202 25
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 16952.3 MB
Peak resident set size usage (VmHWM): 14825.9 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 04-Feb-2026 00:47:16 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 25 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 26 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 