justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40556.1@dunegpschedd02.fnal.gov

Jobsub ID40556.1@dunegpschedd02.fnal.gov
Workflow ID2329
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 18:20:11
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_brug
Last heartbeat2025-09-16 19:56:16
From worker nodeHostnamewn-sate-047.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 18:21:39
Input filesvd-protodune:np02vd_raw_run039324_2818_df-s05-d3_dw_0_20250908T025006.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-16 19:56:16
Saved logsjustin-logs:40556.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t frame: ident=1478517 time=57 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=1478517 time=57 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[21:53:15.249] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 input frame: frame: ident=1478517 time=57 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[21:53:15.249] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 init nticks=6400 tbinmin=0 tbinmax=6400 
[21:53:15.278] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 load plane index: 0, ntraces=1536, input bad regions: 0 
[21:53:16.856] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 load plane index: 1, ntraces=1536, input bad regions: 0 
[21:53:18.436] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 load plane index: 2, ntraces=1536, input bad regions: 0 
[21:53:49.343] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 save plane index: 0, Qtot=3304108329 Qloss=-57542827, 10030 indices spanning [45795,55824] "wiener" 
[21:53:49.535] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 save plane index: 0, Qtot=3269008614 Qloss=-47453470, 7606 indices spanning [55825,63430] "gauss" 
[21:53:50.462] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 save plane index: 1, Qtot=575786158 Qloss=-61091433, 13114 indices spanning [63431,76544] "wiener" 
[21:53:50.642] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 save plane index: 1, Qtot=523634128 Qloss=-42988240, 9535 indices spanning [76545,86079] "gauss" 
[21:53:51.002] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 save plane index: 2, Qtot=472101461 Qloss=-36478782, 13427 indices spanning [86080,99506] "wiener" 
[21:53:51.340] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 save plane index: 2, Qtot=451022932 Qloss=-15166083, 10703 indices spanning [99507,110209] "gauss" 
[21:53:51.340] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 produce 110210 traces: 36571 wiener6, 0 decon_charge6, 27844 gauss6, frame tag: sigproc 
[21:53:51.340] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=46 output frame: frame: ident=1478517 time=57 tick=512 with 110210 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":27844 [0] "mp2_roi6":34196 [0] "mp3_roi6":11599 [0] "wiener6":36571 [36571] ] cmm:[ ] 
[21:53:55.979] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[21:53:55.979] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=1478517 time=57 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=1478517 time=57 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[21:53:55.980] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 input frame: frame: ident=1478517 time=57 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[21:53:55.980] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 init nticks=6400 tbinmin=0 tbinmax=6400 
[21:53:56.008] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 load plane index: 0, ntraces=1536, input bad regions: 0 
[21:53:57.906] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 load plane index: 1, ntraces=1536, input bad regions: 0 
[21:53:59.712] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 load plane index: 2, ntraces=1536, input bad regions: 0 
[21:54:26.005] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 save plane index: 0, Qtot=4330598504 Qloss=-57178108, 8834 indices spanning [43764,52597] "wiener" 
[21:54:26.191] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 save plane index: 0, Qtot=4293171135 Qloss=-49666546, 6867 indices spanning [52598,59464] "gauss" 
[21:54:27.046] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 save plane index: 1, Qtot=505521866 Qloss=-59197154, 12219 indices spanning [59465,71683] "wiener" 
[21:54:27.236] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 save plane index: 1, Qtot=460506454 Qloss=-42332806, 9003 indices spanning [71684,80686] "gauss" 
[21:54:27.565] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 save plane index: 2, Qtot=441305364 Qloss=-32969484, 12222 indices spanning [80687,92908] "wiener" 
[21:54:27.889] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 save plane index: 2, Qtot=423244824 Qloss=-14243923, 9820 indices spanning [92909,102728] "gauss" 
[21:54:27.890] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 produce 102729 traces: 33275 wiener5, 0 decon_charge5, 25690 gauss5, frame tag: sigproc 
[21:54:27.890] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=46 output frame: frame: ident=1478517 time=57 tick=512 with 102729 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":25690 [0] "mp2_roi5":32551 [0] "mp3_roi5":11213 [0] "wiener5":33275 [33275] ] cmm:[ ] 
[21:54:31.665] W [  glue  ] <ChannelSelector:chsel4> Untagged summary not supported, summary will be dropped. 
[21:54:31.666] D [  glue  ] <ChannelSelector:chsel4> input frame: ident=1478517 time=57 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=1478517 time=57 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[21:54:31.667] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=46 input frame: frame: ident=1478517 time=57 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[21:54:31.667] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=46 init nticks=6400 tbinmin=0 tbinmax=6400 
[21:54:31.699] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=46 load plane index: 0, ntraces=1536, input bad regions: 0 
[21:54:33.312] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=46 load plane index: 1, ntraces=1536, input bad regions: 0 
[21:54:35.003] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=46 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      0.000103925     222.893       351.921       216.689       74.994         24     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.2498e-05    8.74513e-05   0.000208612   8.1373e-05    2.82763e-05      24     
produce:tpcrawdecoder:PDVDTPCReader               56.9647       75.2852       120.701       71.2974       14.6843        24     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0347675     0.0356926     0.0461621     0.0352226    0.00220282       24     
produce:pdvddaphne:DAPHNEReaderPDVD               4.16906       9.45309       11.9092       9.58231       1.73281        24     
produce:ophit:OpHitFinder                        0.0435105     0.056108      0.0646714     0.0576208     0.0053231       24     
produce:opflash:OpFlashFinderVerticalDrift      0.00727919     0.0146412     0.0204415     0.0142889    0.00295469       24     
produce:wclsdatavd:WireCellToolkit                43.1743       56.2382       84.7441       54.6199       10.2382        23     
produce:gaushit:GausHitFinder                    0.576185       1.25383       1.95736       1.26441      0.367388        23     
produce:nhitsfilter:NumberOfHitsFilter          0.000167224   0.000412733   0.000797316   0.000384221    0.0001478       23     
produce:reco3d:SpacePointSolver                   4.28911       13.434        25.1722       13.1597       5.36634        23     
produce:hitpdune:DisambigFromSpacePoints         0.0558558     0.226194       0.42984      0.207448      0.106294        23     
produce:pandora:StandardPandora                   12.1816       69.5783       171.093       59.8596       45.3943        23     
produce:pandoraTrack:LArPandoraTrackCreation     0.181874       1.39299       4.85519       1.11136      0.984949        23     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0127426     0.0316689     0.0532763     0.0320265     0.0105802       23     
[art]:TriggerResults:TriggerResultInserter      2.2021e-05    4.79931e-05   8.9388e-05    4.7749e-05    2.04623e-05      23     
end_path:out1:RootOutput                         3.867e-06    9.37587e-06   2.2432e-05     8.145e-06    5.02052e-06      23     
end_path:out1:RootOutput(write)                   3.87561       4.50259        6.403        4.36092      0.562716        23     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.84 MB
  Peak resident set size usage (VmHWM): 6710.54 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 21:55:54 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39324 subRun: 1 event: 1478517
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-04 08:12:53 UTC       justIN version: 01.05.01