justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 47051.1@dunegpschedd01.fnal.gov

Jobsub ID47051.1@dunegpschedd01.fnal.gov
Workflow ID2329
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 16:42:07
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-09-16 18:05:28
From worker nodeHostnamewn-lot-026.farm.nikhef.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 16:43:16
Input filesvd-protodune:np02vd_raw_run039324_2818_df-s05-d3_dw_0_20250908T025006.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-16 18:05:28
Saved logsjustin-logs:47051.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t frame: ident=1478517 time=57 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=1478517 time=57 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:02:58.165] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 input frame: frame: ident=1478517 time=57 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:02:58.165] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 init nticks=6400 tbinmin=0 tbinmax=6400 
[20:02:58.191] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 load plane index: 0, ntraces=1536, input bad regions: 0 
[20:02:59.690] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 load plane index: 1, ntraces=1536, input bad regions: 0 
[20:03:01.310] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 load plane index: 2, ntraces=1536, input bad regions: 0 
[20:03:25.413] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 save plane index: 0, Qtot=3304108329 Qloss=-57542827, 10030 indices spanning [45795,55824] "wiener" 
[20:03:25.590] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 save plane index: 0, Qtot=3269008614 Qloss=-47453470, 7606 indices spanning [55825,63430] "gauss" 
[20:03:26.362] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 save plane index: 1, Qtot=575786158 Qloss=-61091433, 13114 indices spanning [63431,76544] "wiener" 
[20:03:26.551] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 save plane index: 1, Qtot=523634128 Qloss=-42988240, 9535 indices spanning [76545,86079] "gauss" 
[20:03:26.895] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 save plane index: 2, Qtot=472101461 Qloss=-36478782, 13427 indices spanning [86080,99506] "wiener" 
[20:03:27.226] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 save plane index: 2, Qtot=451022932 Qloss=-15166083, 10703 indices spanning [99507,110209] "gauss" 
[20:03:27.226] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 produce 110210 traces: 36571 wiener6, 0 decon_charge6, 27844 gauss6, frame tag: sigproc 
[20:03:27.226] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 output frame: frame: ident=1478517 time=57 tick=512 with 110210 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":27844 [0] "mp2_roi6":34196 [0] "mp3_roi6":11599 [0] "wiener6":36571 [36571] ] cmm:[ ] 
[20:03:30.349] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[20:03:30.349] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=1478517 time=57 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=1478517 time=57 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:03:30.349] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 input frame: frame: ident=1478517 time=57 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:03:30.349] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 init nticks=6400 tbinmin=0 tbinmax=6400 
[20:03:30.376] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 load plane index: 0, ntraces=1536, input bad regions: 0 
[20:03:31.988] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 load plane index: 1, ntraces=1536, input bad regions: 0 
[20:03:33.673] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 load plane index: 2, ntraces=1536, input bad regions: 0 
[20:03:54.263] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 save plane index: 0, Qtot=4330598504 Qloss=-57178108, 8834 indices spanning [43764,52597] "wiener" 
[20:03:54.445] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 save plane index: 0, Qtot=4293171135 Qloss=-49666546, 6867 indices spanning [52598,59464] "gauss" 
[20:03:55.158] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 save plane index: 1, Qtot=505521866 Qloss=-59197154, 12219 indices spanning [59465,71683] "wiener" 
[20:03:55.336] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 save plane index: 1, Qtot=460506454 Qloss=-42332806, 9003 indices spanning [71684,80686] "gauss" 
[20:03:55.678] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 save plane index: 2, Qtot=441305364 Qloss=-32969484, 12222 indices spanning [80687,92908] "wiener" 
[20:03:56.007] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 save plane index: 2, Qtot=423244824 Qloss=-14243923, 9820 indices spanning [92909,102728] "gauss" 
[20:03:56.008] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 produce 102729 traces: 33275 wiener5, 0 decon_charge5, 25690 gauss5, frame tag: sigproc 
[20:03:56.008] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 output frame: frame: ident=1478517 time=57 tick=512 with 102729 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":25690 [0] "mp2_roi5":32551 [0] "mp3_roi5":11213 [0] "wiener5":33275 [33275] ] cmm:[ ] 
[20:03:58.691] W [  glue  ] <ChannelSelector:chsel4> Untagged summary not supported, summary will be dropped. 
[20:03:58.692] D [  glue  ] <ChannelSelector:chsel4> input frame: ident=1478517 time=57 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=1478517 time=57 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:03:58.692] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=46 input frame: frame: ident=1478517 time=57 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:03:58.692] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=46 init nticks=6400 tbinmin=0 tbinmax=6400 
[20:03:58.719] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=46 load plane index: 0, ntraces=1536, input bad regions: 0 
[20:04:00.293] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=46 load plane index: 1, ntraces=1536, input bad regions: 0 
[20:04:01.955] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=46 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.6165e-05      193.183       280.129       192.056       58.9537        24     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.5283e-05    7.74564e-05   0.00016554    7.5692e-05    2.10098e-05      24     
produce:tpcrawdecoder:PDVDTPCReader               53.278        69.3222       113.566       65.2111       14.2617        24     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0346369     0.0362997     0.0497038     0.0349464     0.0038653       24     
produce:pdvddaphne:DAPHNEReaderPDVD               6.02693       8.25677       11.275        8.34678       1.35895        24     
produce:ophit:OpHitFinder                        0.0333051     0.0461243     0.0872465     0.0438315     0.0129817       24     
produce:opflash:OpFlashFinderVerticalDrift      0.00703194     0.0137281     0.0313845     0.0125199    0.00510549       24     
produce:wclsdatavd:WireCellToolkit                39.3053       52.7404       79.0167       51.4504       8.82702        23     
produce:gaushit:GausHitFinder                    0.517769       1.04537       1.61888       1.02713       0.30458        23     
produce:nhitsfilter:NumberOfHitsFilter          0.000225763   0.000540958   0.00105935    0.000508765   0.000245766      23     
produce:reco3d:SpacePointSolver                   4.49876       12.7587       24.5875       12.3974       5.13843        23     
produce:hitpdune:DisambigFromSpacePoints         0.0548995      0.19871      0.367611      0.187436      0.100345        23     
produce:pandora:StandardPandora                   11.3817       52.3918       130.93        44.2502       33.9318        23     
produce:pandoraTrack:LArPandoraTrackCreation     0.153558       0.65694       1.82696      0.568577       0.36106        23     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0128418     0.026884      0.0494102     0.027457     0.00912544       23     
[art]:TriggerResults:TriggerResultInserter       1.584e-05    2.15548e-05   4.9392e-05    2.0358e-05    6.62569e-06      23     
end_path:out1:RootOutput                         3.677e-06    5.69896e-06   1.9827e-05     5.29e-06     3.08599e-06      23     
end_path:out1:RootOutput(write)                   3.58522       4.08432       5.73192       3.98519      0.513291        23     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.84 MB
  Peak resident set size usage (VmHWM): 6708.53 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 20:05:07 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39324 subRun: 1 event: 1478517
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-19 01:44:32 UTC       justIN version: 01.05.00