justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241295.0@dunegpschedd01.fnal.gov

Jobsub ID241295.0@dunegpschedd01.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 06:06:37
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-10-31 07:21:05
From worker nodeHostnamewn-sate-052.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 06:07:43
Input filesvd-protodune:np02vd_raw_run040266_0552_df-s03-d0_dw_0_20251025T124048.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 07:21:05
Saved logsjustin-logs:241295.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

D [  glue  ] <ChannelSelector:chsel7> input frame: ident=102024 time=92 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=102024 time=92 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[08:18:52.998] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 input frame: frame: ident=102024 time=92 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[08:18:52.998] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 init nticks=8000 tbinmin=0 tbinmax=8000 
[08:18:53.029] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 load plane index: 0, ntraces=1536, input bad regions: 0 
[08:18:54.959] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 load plane index: 1, ntraces=1536, input bad regions: 0 
[08:18:57.013] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 load plane index: 2, ntraces=1536, input bad regions: 0 
[08:19:14.220] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 0, Qtot=17757976454 Qloss=-5501176808, 5574 indices spanning [18928,24501] "wiener" 
[08:19:14.427] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 0, Qtot=16824420867 Qloss=-5465687874, 4909 indices spanning [24502,29410] "gauss" 
[08:19:15.109] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 1, Qtot=4515443842 Qloss=-5509361341, 4794 indices spanning [29411,34204] "wiener" 
[08:19:15.324] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 1, Qtot=3760751083 Qloss=-5539097959, 4071 indices spanning [34205,38275] "gauss" 
[08:19:15.737] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 2, Qtot=3164642419 Qloss=-2192498454, 13971 indices spanning [38276,52246] "wiener" 
[08:19:16.167] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 2, Qtot=2685964652 Qloss=-1714377009, 17109 indices spanning [52247,69355] "gauss" 
[08:19:16.167] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 produce 69356 traces: 24339 wiener7, 0 decon_charge7, 26089 gauss7, frame tag: sigproc 
[08:19:16.167] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 output frame: frame: ident=102024 time=92 tick=512 with 69356 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":26089 [0] "mp2_roi7":15776 [0] "mp3_roi7":3152 [0] "wiener7":24339 [24339] ] cmm:[ ] 
[08:19:18.281] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[08:19:18.282] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=102024 time=92 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=102024 time=92 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[08:19:18.282] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 input frame: frame: ident=102024 time=92 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[08:19:18.282] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 init nticks=8000 tbinmin=0 tbinmax=8000 
[08:19:18.311] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 load plane index: 0, ntraces=1536, input bad regions: 0 
[08:19:20.392] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 load plane index: 1, ntraces=1536, input bad regions: 0 
[08:19:22.467] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 load plane index: 2, ntraces=1536, input bad regions: 0 
[08:19:24.866] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 0, Qtot=37543894 Qloss=-6599197, 811 indices spanning [3649,4459] "wiener" 
[08:19:25.076] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 0, Qtot=35795156 Qloss=-5813923, 671 indices spanning [4460,5130] "gauss" 
[08:19:25.393] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 1, Qtot=43316573 Qloss=-7343582, 1701 indices spanning [5131,6831] "wiener" 
[08:19:25.600] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 1, Qtot=39792420 Qloss=-5455276, 1436 indices spanning [6832,8267] "gauss" 
[08:19:25.995] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 2, Qtot=37958067 Qloss=-2905694, 1637 indices spanning [8268,9904] "wiener" 
[08:19:26.388] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 2, Qtot=36397979 Qloss=-1306752, 1307 indices spanning [9905,11211] "gauss" 
[08:19:26.388] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 produce 11212 traces: 4149 wiener6, 0 decon_charge6, 3414 gauss6, frame tag: sigproc 
[08:19:26.389] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 output frame: frame: ident=102024 time=92 tick=512 with 11212 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":3414 [0] "mp2_roi6":2845 [0] "mp3_roi6":804 [0] "wiener6":4149 [4149] ] cmm:[ ] 
[08:19:26.398] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[08:19:26.399] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=102024 time=92 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=102024 time=92 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[08:19:26.399] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 input frame: frame: ident=102024 time=92 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[08:19:26.399] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 init nticks=8000 tbinmin=0 tbinmax=8000 
[08:19:26.430] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 load plane index: 0, ntraces=1536, input bad regions: 0 
[08:19:28.401] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 load plane index: 1, ntraces=1536, input bad regions: 0 
[08:19:30.444] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.4411e-05      200.25        1086.88       165.738       204.466        21     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.7748e-05    8.58776e-05   0.000343376   6.5403e-05    6.33729e-05      21     
produce:tpcrawdecoder:PDVDTPCReader               12.9695       21.6308       50.5605       20.0067       9.36348        21     
produce:triggerrawdecoder:PDVDTriggerReader4     0.500125      0.535376      0.589097      0.535353      0.0203583       21     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000370797   0.000401943   0.000684046   0.000384162   6.61761e-05      21     
produce:ophit:OpHitFinder                       5.6225e-05    8.40342e-05   0.000481436   6.0634e-05    8.9334e-05       21     
produce:opflash:OpFlashFinderVerticalDrift      5.0185e-05    6.57751e-05   0.000291118   5.2398e-05    5.10007e-05      21     
produce:wclsdatavd:WireCellToolkit                53.9722       85.1253       136.564       75.2692       19.5343        20     
produce:gaushit:GausHitFinder                    0.934944       1.58923       6.69566       1.29987       1.18927        20     
produce:nhitsfilter:NumberOfHitsFilter          0.000231676   0.000303122   0.00051592    0.000285181   6.23131e-05      20     
produce:reco3d:SpacePointSolver                   8.4832        14.1794       28.0142       13.4663       4.21611        20     
produce:hitpdune:DisambigFromSpacePoints         0.124912      0.212594      0.558563      0.191718      0.0912541       20     
produce:pandora:StandardPandora                   17.7366       79.9465       915.326       31.1805       192.217        20     
produce:pandoraTrack:LArPandoraTrackCreation     0.804912       1.40408       3.96451       1.16447      0.766139        20     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0278541     0.0379175     0.0710111     0.0360165     0.0095378       20     
[art]:TriggerResults:TriggerResultInserter      1.7793e-05    2.74039e-05    5.871e-05    2.5463e-05    9.95756e-06      20     
end_path:out1:RootOutput                         3.426e-06    5.4949e-06    2.0529e-05     3.917e-06    3.66899e-06      20     
end_path:out1:RootOutput(write)                   4.62861       5.11404       6.22005       5.03687       0.40407        20     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.9 MB
  Peak resident set size usage (VmHWM): 6707.36 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 08:20:35 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 102024
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:26:22 UTC       justIN version: 01.05.01