Jobsub ID 40398.45@dunegpschedd02.fnal.gov
| Jobsub ID | 40398.45@dunegpschedd02.fnal.gov |
| Workflow ID | 2644 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-16 12:44:56 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_juk |
| Last heartbeat | 2025-09-16 14:07:47 |
| From worker node | Hostname | wn-snel-030.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7H12 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-09-16 13:53:51 |
| Input files | vd-protodune:np02vd_raw_run039353_0986_df-s05-d3_dw_0_20250916T114453.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-09-16 14:07:47 |
| Saved logs | justin-logs:40398.45-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
channel 9013
wclsFrameSaver: no samples within desired window for channel 9013
wclsFrameSaver: no samples within desired window for channel 9015
wclsFrameSaver: no samples within desired window for channel 9016
wclsFrameSaver: no samples within desired window for channel 9016
wclsFrameSaver: no samples within desired window for channel 9017
wclsFrameSaver: no samples within desired window for channel 9017
wclsFrameSaver: no samples within desired window for channel 9601
wclsFrameSaver: no samples within desired window for channel 9604
wclsFrameSaver: no samples within desired window for channel 9605
wclsFrameSaver: no samples within desired window for channel 9705
wclsFrameSaver: no samples within desired window for channel 9838
wclsFrameSaver: no samples within desired window for channel 9850
wclsFrameSaver: no samples within desired window for channel 10146
wclsFrameSaver: no samples within desired window for channel 10147
wclsFrameSaver: no samples within desired window for channel 10157
wclsFrameSaver: no samples within desired window for channel 10228
wclsFrameSaver: no samples within desired window for channel 10286
wclsFrameSaver: no samples within desired window for channel 10307
wclsFrameSaver: no samples within desired window for channel 10336
wclsFrameSaver: no samples within desired window for channel 10499
wclsFrameSaver: no samples within desired window for channel 10500
wclsFrameSaver: no samples within desired window for channel 11020
wclsFrameSaver: no samples within desired window for channel 11640
wclsFrameSaver: no samples within desired window for channel 11640
wclsFrameSaver: no samples within desired window for channel 11641
wclsFrameSaver: no samples within desired window for channel 11758
wclsFrameSaver: no samples within desired window for channel 11759
wclsFrameSaver: no samples within desired window for channel 11760
wclsFrameSaver: no samples within desired window for channel 11761
wclsFrameSaver: no samples within desired window for channel 12274
wclsFrameSaver: no samples within desired window for channel 12275
wclsFrameSaver: no samples within desired window for channel 12276
wclsFrameSaver: no samples within desired window for channel 12276
wclsFrameSaver: no samples within desired window for channel 12278
FrameSaver: q=9.37509e+06 n=1108623 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 2070 XUs and 1863 XVs -> 90 XUVs
C:0 T:1 225 XUs and 290 XVs -> 20 XUVs
C:0 T:2 2184 XUs and 2302 XVs -> 83 XUVs
C:0 T:3 178 XUs and 234 XVs -> 8 XUVs
C:0 T:4 362 XUs and 560 XVs -> 16 XUVs
C:0 T:5 2534 XUs and 2581 XVs -> 103 XUVs
C:0 T:6 1269 XUs and 1187 XVs -> 37 XUVs
C:0 T:7 374 XUs and 387 XVs -> 32 XUVs
C:0 T:8 1376 XUs and 2166 XVs -> 92 XUVs
C:0 T:9 3136 XUs and 4103 XVs -> 185 XUVs
C:0 T:10 1860 XUs and 2324 XVs -> 131 XUVs
C:0 T:11 2903 XUs and 2441 XVs -> 135 XUVs
C:0 T:12 674 XUs and 1188 XVs -> 44 XUVs
C:0 T:13 1567 XUs and 2194 XVs -> 112 XUVs
C:0 T:14 3090 XUs and 2739 XVs -> 155 XUVs
C:0 T:15 1848 XUs and 1985 XVs -> 74 XUVs
1317 XUVs total
1163 collection wire objects
1317 potential space points
Neighbour search...
6393 tests to find 3368 neighbours
Iterating with no regularization...
Begin: 4.24088e+08
0 4.1633e+08
1 4.16227e+08
Now with regularization...
Begin: 4.10622e+08
0 4.1062e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=10000 keeping as is
[16:06:41.661] D [ main ] executing 1 apps, thread limit 0:
[16:06:41.661] D [ main ] executing 1 apps, thread limit 0:
[16:06:41.661] D [ main ] executing app: "Pgrapher"
[16:06:41.661] D [ pgraph ] <Pgrapher:> executing graph
[16:06:41.661] D [ pgraph ] executing with 26 nodes
[16:06:41.662] D [ glue ] <FrameFanout:nfsp> call=6: input: frame: ident=75911 time=12 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=75911 time=12 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=75911 time=12 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=75911 time=12 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=75911 time=12 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=75911 time=12 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=75911 time=12 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=75911 time=12 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=75911 time=12 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[16:06:41.663] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[16:06:41.663] D [ glue ] <ChannelSelector:chsel7> input frame: ident=75911 time=12 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=75911 time=12 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[16:06:41.664] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 input frame: frame: ident=75911 time=12 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[16:06:41.664] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 init nticks=10000 tbinmin=0 tbinmax=10000
[16:06:41.710] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 load plane index: 0, ntraces=1536, input bad regions: 0
[16:06:44.203] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 load plane index: 1, ntraces=1536, input bad regions: 0
[16:06:45.885] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 7.8186e-05 163.121 246.309 203.087 96.1585 4
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 7.5251e-05 0.000113954 0.000225723 7.742e-05 6.45385e-05 4
produce:tpcrawdecoder:PDVDTPCReader 81.1303 87.9226 91.3993 89.5804 4.13588 4
produce:triggerrawdecoder:PDVDTriggerReader4 0.292578 0.296927 0.307164 0.293984 0.0059779 4
produce:pdvddaphne:DAPHNEReaderPDVD 0.000407865 0.000543046 0.000797536 0.000483391 0.000158548 4
produce:ophit:OpHitFinder 0.000131307 0.000333834 0.000836709 0.00018366 0.000292941 4
produce:opflash:OpFlashFinderVerticalDrift 5.6466e-05 0.000134682 0.000364283 5.89905e-05 0.000132567 4
produce:wclsdatavd:WireCellToolkit 56.0653 61.4109 72.0319 56.1354 7.51028 3
produce:gaushit:GausHitFinder 0.974093 1.16483 1.32141 1.19897 0.143832 3
produce:nhitsfilter:NumberOfHitsFilter 0.000283211 0.000377261 0.000460804 0.000387767 7.28817e-05 3
produce:reco3d:SpacePointSolver 11.0932 13.4995 15.9832 13.4221 1.9971 3
produce:hitpdune:DisambigFromSpacePoints 0.168809 0.180411 0.200023 0.172399 0.0139457 3
produce:pandora:StandardPandora 26.9239 47.8246 59.4514 57.0986 14.8102 3
produce:pandoraTrack:LArPandoraTrackCreation 0.709746 0.897301 1.20278 0.779378 0.217868 3
produce:pandoraGnocalo:GnocchiCalorimetry 0.0265278 0.0291841 0.0320615 0.028963 0.00226453 3
[art]:TriggerResults:TriggerResultInserter 1.4006e-05 2.36977e-05 4.1678e-05 1.5409e-05 1.27269e-05 3
end_path:out1:RootOutput 3.407e-06 9.565e-06 1.6341e-05 8.947e-06 5.29834e-06 3
end_path:out1:RootOutput(write) 4.20221 4.25186 4.30076 4.25261 0.0402366 3
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.63 MB
Peak resident set size usage (VmHWM): 6679.35 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 16-Sep-2025 16:07:29 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39353 subRun: 1 event: 75911
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 