justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40427.0@dunegpschedd02.fnal.gov

Jobsub ID40427.0@dunegpschedd02.fnal.gov
Workflow ID2323
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 15:33:03
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_brug
Last heartbeat2025-09-16 16:12:17
From worker nodeHostnamewn-sate-038.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 15:34:18
Input filesvd-protodune:np02vd_raw_run039324_1630_df-s05-d4_dw_0_20250907T012250.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-16 16:12:17
Saved logsjustin-logs:40427.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

input frame: ident=855158 time=29 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[18:09:45.164] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 input frame: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[18:09:45.164] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 init nticks=6400 tbinmin=0 tbinmax=6400 
[18:09:45.193] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 load plane index: 0, ntraces=1536, input bad regions: 0 
[18:09:46.476] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 load plane index: 1, ntraces=1536, input bad regions: 0 
[18:09:47.791] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 load plane index: 2, ntraces=1536, input bad regions: 0 
[18:10:16.909] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 0, Qtot=1645939690 Qloss=-67944562, 12244 indices spanning [40500,52743] "wiener" 
[18:10:17.074] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 0, Qtot=1608146478 Qloss=-53945582, 9381 indices spanning [52744,62124] "gauss" 
[18:10:17.864] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 1, Qtot=665995087 Qloss=-70309597, 14326 indices spanning [62125,76450] "wiener" 
[18:10:18.027] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 1, Qtot=595164528 Qloss=-52546399, 10661 indices spanning [76451,87111] "gauss" 
[18:10:18.346] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 2, Qtot=682854623 Qloss=-39444026, 16732 indices spanning [87112,103843] "wiener" 
[18:10:18.655] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 2, Qtot=661494271 Qloss=-16999377, 13201 indices spanning [103844,117044] "gauss" 
[18:10:18.655] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 produce 117045 traces: 43302 wiener6, 0 decon_charge6, 33243 gauss6, frame tag: sigproc 
[18:10:18.655] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 output frame: frame: ident=855158 time=29 tick=512 with 117045 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":33243 [0] "mp2_roi6":27913 [0] "mp3_roi6":12587 [0] "wiener6":43302 [43302] ] cmm:[ ] 
[18:10:22.151] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[18:10:22.152] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=855158 time=29 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[18:10:22.152] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 input frame: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[18:10:22.152] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 init nticks=6400 tbinmin=0 tbinmax=6400 
[18:10:22.176] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 load plane index: 0, ntraces=1536, input bad regions: 0 
[18:10:23.698] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 load plane index: 1, ntraces=1536, input bad regions: 0 
[18:10:25.279] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 load plane index: 2, ntraces=1536, input bad regions: 0 
[18:10:54.242] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 0, Qtot=2237240361 Qloss=-76210082, 11415 indices spanning [41557,52971] "wiener" 
[18:10:54.406] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 0, Qtot=2198384733 Qloss=-63704229, 8581 indices spanning [52972,61552] "gauss" 
[18:10:55.166] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 1, Qtot=599925440 Qloss=-62985466, 13185 indices spanning [61553,74737] "wiener" 
[18:10:55.330] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 1, Qtot=540908454 Qloss=-45413317, 9717 indices spanning [74738,84454] "gauss" 
[18:10:55.647] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 2, Qtot=606800966 Qloss=-41414355, 15841 indices spanning [84455,100295] "wiener" 
[18:10:55.936] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 2, Qtot=584427007 Qloss=-17863512, 12843 indices spanning [100296,113138] "gauss" 
[18:10:55.936] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 produce 113139 traces: 40441 wiener5, 0 decon_charge5, 31141 gauss5, frame tag: sigproc 
[18:10:55.936] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 output frame: frame: ident=855158 time=29 tick=512 with 113139 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":31141 [0] "mp2_roi5":29588 [0] "mp3_roi5":11969 [0] "wiener5":40441 [40441] ] cmm:[ ] 
[18:10:59.009] W [  glue  ] <ChannelSelector:chsel4> Untagged summary not supported, summary will be dropped. 
[18:10:59.010] D [  glue  ] <ChannelSelector:chsel4> input frame: ident=855158 time=29 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[18:10:59.010] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 input frame: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[18:10:59.010] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 init nticks=6400 tbinmin=0 tbinmax=6400 
[18:10:59.038] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 load plane index: 0, ntraces=1536, input bad regions: 0 
[18:11:00.596] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 load plane index: 1, ntraces=1536, input bad regions: 0 
[18:11:02.147] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                       7.959e-05      166.476       226.215       173.31        60.2617        12     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.7737e-05    8.13707e-05   0.000163889   7.2777e-05    2.54573e-05      12     
produce:tpcrawdecoder:PDVDTPCReader               12.5057       45.2914       72.3477       45.8933       14.2291        12     
produce:triggerrawdecoder:PDVDTriggerReader4     0.034549      0.0369525     0.0456949     0.0346168    0.00413422       12     
produce:pdvddaphne:DAPHNEReaderPDVD               3.2917        6.33765       9.00933       6.38679       1.80849        12     
produce:ophit:OpHitFinder                        0.0362541     0.0481008     0.0695378     0.0475088    0.00824501       12     
produce:opflash:OpFlashFinderVerticalDrift      0.00701275     0.0131689     0.0224636     0.012173     0.00407699       12     
produce:wclsdatavd:WireCellToolkit                44.1219       59.1213       75.5803       57.0923       8.59696        11     
produce:gaushit:GausHitFinder                    0.830659       1.16464       1.5667        1.07663      0.241259        11     
produce:nhitsfilter:NumberOfHitsFilter          0.000267624   0.000380139   0.000547922   0.000364456   9.33968e-05      11     
produce:reco3d:SpacePointSolver                   8.27751       12.8616       19.3407       11.8661       3.39618        11     
produce:hitpdune:DisambigFromSpacePoints         0.101701      0.191196      0.324438      0.174626      0.0643281       11     
produce:pandora:StandardPandora                   31.696        51.6943       91.4532       41.8518       18.6137        11     
produce:pandoraTrack:LArPandoraTrackCreation     0.415599      0.815235       1.7628       0.663769       0.39369        11     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0199785     0.0302564     0.0407669     0.0303188    0.00576701       11     
[art]:TriggerResults:TriggerResultInserter      1.9146e-05    2.88134e-05    5.855e-05    2.7462e-05    9.91876e-06      11     
end_path:out1:RootOutput                         7.064e-06    1.00994e-05   2.7362e-05     8.386e-06    5.51183e-06      11     
end_path:out1:RootOutput(write)                   4.04666       4.60184       6.18857       4.34328      0.689771        11     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.87 MB
  Peak resident set size usage (VmHWM): 6710.29 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 18:12:00 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39324 subRun: 1 event: 855158
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-19 07:39:34 UTC       justIN version: 01.05.00