Jobsub ID 303484.18@dunegpschedd01.fnal.gov
| Jobsub ID | 303484.18@dunegpschedd01.fnal.gov |
| Workflow ID | 12689 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-02-03 15:52:05 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2026-02-03 16:58:06 |
| From worker node | Hostname | wn-s03-04-13-03.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Gold 6438Y+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-02-03 15:56:46 |
| Input files | vd-protodune:np02vd_raw_run042421_0011_df-s02-d0_dw_0_20260203T133654.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-02-03 16:58:06 |
| Saved logs | justin-logs:303484.18-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
0xb62b6bc0
Caught exception in looking up space point ptr... 0xaf878fc0, 0xaf8792e0
Caught exception in looking up space point ptr... 0xaf8792e0, 0xaf879470
Caught exception in looking up space point ptr... 0x9ba62180, 0x9ba615d0
Caught exception in looking up space point ptr... 0x9ba62410, 0x9ba62180
Caught exception in looking up space point ptr... 0xad8aee90, 0xad8af120
Caught exception in looking up space point ptr... 0xad8af120, 0xad8aec00
Caught exception in looking up space point ptr... 0xb1ca2280, 0xb1ca23b0
Caught exception in looking up space point ptr... 0xb1ca23b0, 0xb1ca2150
Caught exception in looking up space point ptr... 0xb1ca2150, 0xb1ca2610
Caught exception in looking up space point ptr... 0xa819ed00, 0xa818a820
Caught exception in looking up space point ptr... 0xa818a820, 0xa8184f70
Caught exception in looking up space point ptr... 0x84e49390, 0x849a3a60
Caught exception in looking up space point ptr... 0x849a3a60, 0x84e49390
Caught exception in looking up space point ptr... 0x5c4aae20, 0x65202bb0
Caught exception in looking up space point ptr... 0x65202bb0, 0x5c4aae20
Caught exception in looking up space point ptr... 0x7750e4f0, 0x77d94b20
Caught exception in looking up space point ptr... 0x77d94b20, 0x7750e4f0
Caught exception in looking up space point ptr... 0x96475b20, 0x96476040
Caught exception in looking up space point ptr... 0x96476040, 0x971abee0
Caught exception in looking up space point ptr... 0xc459bac0, 0xc90d0b90
Caught exception in looking up space point ptr... 0xc90d0b90, 0xc99f78a0
Caught exception in looking up space point ptr... 0x63f928d0, 0xf0a39b0
Caught exception in looking up space point ptr... 0xf0a39b0, 0x46a85330
Caught exception in looking up space point ptr... 0x9069c8d0, 0x9006dad0
Caught exception in looking up space point ptr... 0x9006dad0, 0x9069ca60
Caught exception in looking up space point ptr... 0x5854f2d0, 0x5854c380
Caught exception in looking up space point ptr... 0x5b0111d0, 0x5854f2d0
Caught exception in looking up space point ptr... 0x4ab9fe70, 0x40ff7ac0
Caught exception in looking up space point ptr... 0x40ff7ac0, 0x40ff7830
Caught exception in looking up space point ptr... 0xc2dc5060, 0xc10fa4a0
Caught exception in looking up space point ptr... 0xc10fa4a0, 0xc135c250
Caught exception in looking up space point ptr... 0x48e795e0, 0x48e797e0
Caught exception in looking up space point ptr... 0x48e797e0, 0x48e794e0
Caught exception in looking up space point ptr... 0x48e794e0, 0x48825d50
Caught exception in looking up space point ptr... 0x48825d50, 0x48825ee0
Caught exception in looking up space point ptr... 0x48825ee0, 0x48e795e0
Caught exception in looking up space point ptr... 0x5ae6c4a0, 0x5ae6c630
Caught exception in looking up space point ptr... 0x5ae6c630, 0x5ae6c950
Caught exception in looking up space point ptr... 0xa6e15d40, 0xa73cfbe0
Caught exception in looking up space point ptr... 0xa73cfbe0, 0xa6e15d40
Caught exception in looking up space point ptr... 0x758b6ee0, 0x76e5cce0
Caught exception in looking up space point ptr... 0x76e5cce0, 0x758b6ee0
Caught exception in looking up space point ptr... 0x758b6ee0, 0x8317cfe0
Caught exception in looking up space point ptr... 0x8317cfe0, 0x758b6ee0
Caught exception in looking up space point ptr... 0x79cd0330, 0x79b009c0
Caught exception in looking up space point ptr... 0x79b009c0, 0x79cd0330
Caught exception in looking up space point ptr... 0xafe7b2e0, 0xafe7b240
Caught exception in looking up space point ptr... 0xaffc3570, 0xafe7b2e0
Caught exception in looking up space point ptr... 0xcb0788a0, 0xcb06a390
Caught exception in looking up space point ptr... 0xcb06a390, 0xcb0788a0
Caught exception in looking up space point ptr... 0x987d8e30, 0x9f4215d0
Caught exception in looking up space point ptr... 0x9f4215d0, 0x9f27e1f0
Caught exception in looking up space point ptr... 0xa14095a0, 0xa1409ac0
Caught exception in looking up space point ptr... 0xa1409ac0, 0xa14095a0
Caught exception in looking up space point ptr... 0xc69e3fc0, 0xc69dfb80
Caught exception in looking up space point ptr... 0xc69dfb80, 0xc69e3fc0
Caught exception in looking up space point ptr... 0xb208d4b0, 0xb208c090
Caught exception in looking up space point ptr... 0xb208c090, 0xb208d4b0
Caught exception in looking up space point ptr... 0xcba39ed0, 0xcb905e20
Caught exception in looking up space point ptr... 0xcba39bb0, 0xcba39ed0
Caught exception in looking up space point ptr... 0x7921baf0, 0x7921bfa0
Caught exception in looking up space point ptr... 0x7921bfa0, 0x7921bc80
Caught exception in looking up space point ptr... 0x736443f0, 0x70ab2ca0
Caught exception in looking up space point ptr... 0x70ab2ca0, 0x7080a3a0
Caught exception in looking up space point ptr... 0x8a9fadd0, 0x8a9fb060
Caught exception in looking up space point ptr... 0x8a9fb060, 0x8a9fadd0
Caught exception in looking up space point ptr... 0xa8188090, 0xa81893b0
Caught exception in looking up space point ptr... 0xa81893b0, 0xa8188090
Caught exception in looking up space point ptr... 0x587fe320, 0x587fe000
Caught exception in looking up space point ptr... 0x587fe000, 0x587fe320
Caught exception in looking up space point ptr... 0xa00a4720, 0xa0cdd3e0
Caught exception in looking up space point ptr... 0xa0cdd3e0, 0xa122a7e0
Caught exception in looking up space point ptr... 0xb16b8ea0, 0xb0fa9ea0
Caught exception in looking up space point ptr... 0xb0fa9ea0, 0xb1934dd0
Caught exception in looking up space point ptr... 0x790565f0, 0x789e49a0
Caught exception in looking up space point ptr... 0x789e49a0, 0x7de75450
Caught exception in looking up space point ptr... 0xc53ee790, 0xc52133a0
Caught exception in looking up space point ptr... 0xc52133a0, 0xc53ee790
Caught exception in looking up space point ptr... 0xc5578940, 0xc557cdc0
Caught exception in looking up space point ptr... 0xc557cdc0, 0xc5578940
++++>>>> total num hits: 7066627, num free: 6871894
===================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
===================================================================================================================================
Full event 137.502 453.254 2313.88 143.873 759.602 7
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.1888e-05 0.000130034 0.000291749 9.7077e-05 6.75467e-05 7
produce:tpcrawdecoder:PDVDTPCReader 8.24273 8.67797 8.98489 8.71201 0.207589 7
produce:triggerrawdecoder:PDVDTriggerReader4 0.30584 0.316271 0.327626 0.314908 0.00643395 7
produce:timingrawdecoder:PDHDTimingRawDecoder 0.0105222 0.0107545 0.0109345 0.0107459 0.000124336 7
produce:ctbrawdecoder:PDHDCTBRawDecoder 0.0516036 0.0569905 0.064211 0.0547607 0.00518319 7
produce:beamevent:BeamEvent 0.000152149 0.000297833 0.00110024 0.000159285 0.000328027 7
produce:pdvddaphne:DAPHNEReaderPDVD 4.51184 4.79337 5.04839 4.81384 0.200099 7
produce:ophit:OpHitFinder 0.0461063 0.0499488 0.0552055 0.049776 0.00258656 7
produce:wclsdatavd:WireCellToolkit 93.879 100.304 109.534 99.25 5.22817 7
produce:gaushit:GausHitFinder 1.44898 5.24996 18.3827 1.79047 6.05156 7
produce:nhitsfilter:NumberOfHitsFilter 0.000598187 0.00138214 0.00316488 0.00085712 0.00105213 7
produce:reco3d:SpacePointSolver 17.9988 62.1553 265.843 21.4189 91.1429 6
produce:hitpdune:DisambigFromSpacePoints 0.159925 2.17359 12.0857 0.198846 4.4329 6
produce:cluster3d:Cluster3D 1.96153 318.989 1898.88 3.34274 706.548 6
[art]:TriggerResults:TriggerResultInserter 3.5185e-05 6.49379e-05 0.000130132 4.0118e-05 3.33735e-05 7
end_path:out1:RootOutput 8.859e-06 1.41461e-05 3.4791e-05 1.077e-05 8.55932e-06 7
end_path:out1:RootOutput(write) 5.27794 6.064 8.02865 5.70884 0.91638 6
===================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 15882.7 MB
Peak resident set size usage (VmHWM): 13953.8 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 03-Feb-2026 17:55:25 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 6 entries while recob::SpacePoints_cluster3d_Vertex_pdvdofflinestage0. has 7 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Error in reco1 