justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 46999.0@dunegpschedd01.fnal.gov

Jobsub ID46999.0@dunegpschedd01.fnal.gov
Workflow ID2323
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 14:37:01
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_brug
Last heartbeat2025-09-16 15:33:18
From worker nodeHostnamewn-sate-038.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 14:58:11
Input filesvd-protodune:np02vd_raw_run039324_1630_df-s05-d4_dw_0_20250907T012250.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-16 15:33:18
Saved logsjustin-logs:46999.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

> input frame: ident=855158 time=29 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[17:30:39.239] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 input frame: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[17:30:39.239] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 init nticks=6400 tbinmin=0 tbinmax=6400 
[17:30:39.269] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 load plane index: 0, ntraces=1536, input bad regions: 0 
[17:30:40.685] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 load plane index: 1, ntraces=1536, input bad regions: 0 
[17:30:42.009] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 load plane index: 2, ntraces=1536, input bad regions: 0 
[17:31:12.914] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 0, Qtot=1645939690 Qloss=-67944562, 12244 indices spanning [40500,52743] "wiener" 
[17:31:13.084] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 0, Qtot=1608146478 Qloss=-53945582, 9381 indices spanning [52744,62124] "gauss" 
[17:31:13.898] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 1, Qtot=665995087 Qloss=-70309597, 14326 indices spanning [62125,76450] "wiener" 
[17:31:14.060] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 1, Qtot=595164528 Qloss=-52546399, 10661 indices spanning [76451,87111] "gauss" 
[17:31:14.394] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 2, Qtot=682854623 Qloss=-39444026, 16732 indices spanning [87112,103843] "wiener" 
[17:31:14.715] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 2, Qtot=661494271 Qloss=-16999377, 13201 indices spanning [103844,117044] "gauss" 
[17:31:14.715] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 produce 117045 traces: 43302 wiener6, 0 decon_charge6, 33243 gauss6, frame tag: sigproc 
[17:31:14.715] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 output frame: frame: ident=855158 time=29 tick=512 with 117045 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":33243 [0] "mp2_roi6":27913 [0] "mp3_roi6":12587 [0] "wiener6":43302 [43302] ] cmm:[ ] 
[17:31:18.567] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[17:31:18.568] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=855158 time=29 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[17:31:18.568] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 input frame: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[17:31:18.568] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 init nticks=6400 tbinmin=0 tbinmax=6400 
[17:31:18.598] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 load plane index: 0, ntraces=1536, input bad regions: 0 
[17:31:20.128] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 load plane index: 1, ntraces=1536, input bad regions: 0 
[17:31:21.690] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 load plane index: 2, ntraces=1536, input bad regions: 0 
[17:31:51.689] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 0, Qtot=2237240361 Qloss=-76210082, 11415 indices spanning [41557,52971] "wiener" 
[17:31:51.852] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 0, Qtot=2198384733 Qloss=-63704229, 8581 indices spanning [52972,61552] "gauss" 
[17:31:52.636] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 1, Qtot=599925440 Qloss=-62985466, 13185 indices spanning [61553,74737] "wiener" 
[17:31:52.795] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 1, Qtot=540908454 Qloss=-45413317, 9717 indices spanning [74738,84454] "gauss" 
[17:31:53.131] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 2, Qtot=606800966 Qloss=-41414355, 15841 indices spanning [84455,100295] "wiener" 
[17:31:53.450] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 2, Qtot=584427007 Qloss=-17863512, 12843 indices spanning [100296,113138] "gauss" 
[17:31:53.450] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 produce 113139 traces: 40441 wiener5, 0 decon_charge5, 31141 gauss5, frame tag: sigproc 
[17:31:53.450] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 output frame: frame: ident=855158 time=29 tick=512 with 113139 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":31141 [0] "mp2_roi5":29588 [0] "mp3_roi5":11969 [0] "wiener5":40441 [40441] ] cmm:[ ] 
[17:31:57.045] W [  glue  ] <ChannelSelector:chsel4> Untagged summary not supported, summary will be dropped. 
[17:31:57.046] D [  glue  ] <ChannelSelector:chsel4> input frame: ident=855158 time=29 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[17:31:57.046] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 input frame: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[17:31:57.046] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 init nticks=6400 tbinmin=0 tbinmax=6400 
[17:31:57.079] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 load plane index: 0, ntraces=1536, input bad regions: 0 
[17:31:58.443] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 load plane index: 1, ntraces=1536, input bad regions: 0 
[17:31:59.839] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.9351e-05      153.214       239.692       159.419       55.6666        12     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.9793e-05    8.59062e-05   0.000207341   7.2757e-05    3.87355e-05      12     
produce:tpcrawdecoder:PDVDTPCReader               20.2179       32.8449       63.5919       30.0927       10.898         12     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0348953     0.0386176     0.0533645     0.0351408    0.00630082       12     
produce:pdvddaphne:DAPHNEReaderPDVD               3.76086       4.6908        6.89853       4.31404       1.00953        12     
produce:ophit:OpHitFinder                        0.0395186     0.0497726     0.0724879     0.0495511    0.00812626       12     
produce:opflash:OpFlashFinderVerticalDrift      0.00725458     0.0129992     0.0188681     0.0126864    0.00349115       12     
produce:wclsdatavd:WireCellToolkit                41.2524       57.9485       74.5756       56.0832       8.77597        11     
produce:gaushit:GausHitFinder                    0.861264       1.16859       1.57374       1.08226       0.23815        11     
produce:nhitsfilter:NumberOfHitsFilter          0.000287171   0.000406158   0.000521542   0.000398932   6.22947e-05      11     
produce:reco3d:SpacePointSolver                   8.24943       12.898        19.5723       11.6874       3.39372        11     
produce:hitpdune:DisambigFromSpacePoints         0.103146      0.191448      0.322388      0.177845      0.0635719       11     
produce:pandora:StandardPandora                   31.3559       51.7297       91.9655       42.2917       18.7176        11     
produce:pandoraTrack:LArPandoraTrackCreation     0.435766      0.833696       1.93005      0.638043      0.432556        11     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0195234     0.0299747     0.0416053     0.0304419    0.00611841       11     
[art]:TriggerResults:TriggerResultInserter       2.122e-05    2.89791e-05   6.8058e-05    2.5087e-05    1.25454e-05      11     
end_path:out1:RootOutput                         7.013e-06    9.85136e-06    2.627e-05     8.456e-06    5.25018e-06      11     
end_path:out1:RootOutput(write)                   4.08977       4.65261       6.4729        4.31697      0.732448        11     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.86 MB
  Peak resident set size usage (VmHWM): 6705.5 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 17:33:01 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39324 subRun: 1 event: 855158
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-19 10:41:25 UTC       justIN version: 01.05.00