justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 235602.0@dunegpschedd02.fnal.gov

Jobsub ID235602.0@dunegpschedd02.fnal.gov
Workflow ID9373
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 00:46:20
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-10-31 01:45:33
From worker nodeHostnamewn-sate-043.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 00:47:37
Input filesvd-protodune:np02vd_raw_run040266_0017_df-s03-d2_dw_0_20251024T214052.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 01:45:33
Saved logsjustin-logs:235602.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.266] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=3282 time=74 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:43:03.267] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 input frame: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:43:03.267] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 init nticks=8064 tbinmin=0 tbinmax=8064 
[02:43:03.296] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 load plane index: 0, ntraces=1536, input bad regions: 0 
[02:43:05.343] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 load plane index: 1, ntraces=1536, input bad regions: 0 
[02:43:07.179] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 load plane index: 2, ntraces=1536, input bad regions: 0 
[02:43:22.628] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 0, Qtot=1597475914 Qloss=-44434254, 8992 indices spanning [42225,51216] "wiener" 
[02:43:22.852] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 0, Qtot=1569057724 Qloss=-33962514, 7114 indices spanning [51217,58330] "gauss" 
[02:43:23.527] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 1, Qtot=374018808 Qloss=-40901309, 8569 indices spanning [58331,66899] "wiener" 
[02:43:23.735] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 1, Qtot=337259771 Qloss=-32036510, 6411 indices spanning [66900,73310] "gauss" 
[02:43:24.178] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 2, Qtot=360472365 Qloss=-31839779, 12099 indices spanning [73311,85409] "wiener" 
[02:43:24.598] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 2, Qtot=343558979 Qloss=-14455116, 9524 indices spanning [85410,94933] "gauss" 
[02:43:24.598] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 produce 94934 traces: 29660 wiener7, 0 decon_charge7, 23049 gauss7, frame tag: sigproc 
[02:43:24.598] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 output frame: frame: ident=3282 time=74 tick=512 with 94934 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":23049 [0] "mp2_roi7":34007 [0] "mp3_roi7":8218 [0] "wiener7":29660 [29660] ] cmm:[ ] 
[02:43:26.419] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[02:43:26.420] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=3282 time=74 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:43:26.420] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 input frame: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:43:26.421] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 init nticks=8064 tbinmin=0 tbinmax=8064 
[02:43:26.452] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 load plane index: 0, ntraces=1536, input bad regions: 0 
[02:43:28.447] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 load plane index: 1, ntraces=1536, input bad regions: 0 
[02:43:30.505] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 load plane index: 2, ntraces=1536, input bad regions: 0 
[02:43:51.293] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 0, Qtot=2455727316 Qloss=-62469839, 8301 indices spanning [41425,49725] "wiener" 
[02:43:51.517] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 0, Qtot=2420552954 Qloss=-53207934, 6550 indices spanning [49726,56275] "gauss" 
[02:43:52.259] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 1, Qtot=552662726 Qloss=-45210172, 9254 indices spanning [56276,65529] "wiener" 
[02:43:52.475] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 1, Qtot=470383695 Qloss=-37420795, 7344 indices spanning [65530,72873] "gauss" 
[02:43:52.924] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 2, Qtot=543584385 Qloss=-34409715, 12987 indices spanning [72874,85860] "wiener" 
[02:43:53.356] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 2, Qtot=525424367 Qloss=-15707483, 10067 indices spanning [85861,95927] "gauss" 
[02:43:53.356] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 produce 95928 traces: 30542 wiener6, 0 decon_charge6, 23961 gauss6, frame tag: sigproc 
[02:43:53.356] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 output frame: frame: ident=3282 time=74 tick=512 with 95928 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":23961 [0] "mp2_roi6":32969 [0] "mp3_roi6":8456 [0] "wiener6":30542 [30542] ] cmm:[ ] 
[02:43:55.587] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[02:43:55.587] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=3282 time=74 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:43:55.588] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 input frame: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:43:55.588] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 init nticks=8064 tbinmin=0 tbinmax=8064 
[02:43:55.621] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 load plane index: 0, ntraces=1536, input bad regions: 0 
[02:43:57.591] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 load plane index: 1, ntraces=1536, input bad regions: 0 
[02:43:59.731] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                       9.063e-05      202.388       354.584       191.527       74.4401        16     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.2938e-05    8.59168e-05   0.000241052   7.1219e-05    4.20169e-05      16     
produce:tpcrawdecoder:PDVDTPCReader               38.7215       76.8305       190.149       70.9654       31.413         16     
produce:triggerrawdecoder:PDVDTriggerReader4     0.574536      0.722352       1.15553      0.657809      0.158592        16     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000374213   0.000437444   0.000802096   0.000407555   9.75387e-05      16     
produce:ophit:OpHitFinder                       6.4271e-05    0.000110476   0.000633881   7.29975e-05   0.000135572      16     
produce:opflash:OpFlashFinderVerticalDrift      4.8451e-05    7.99283e-05   0.000403297   5.65215e-05   8.37471e-05      16     
produce:wclsdatavd:WireCellToolkit                55.4897       75.8277       116.45        69.8199       18.1273        15     
produce:gaushit:GausHitFinder                    0.916592       1.30988       2.16393       1.19585      0.343111        15     
produce:nhitsfilter:NumberOfHitsFilter          0.000229822   0.000378595   0.000696688   0.000356289   0.000110637      15     
produce:reco3d:SpacePointSolver                   7.90278       13.6029       25.1119       11.8685       4.85573        15     
produce:hitpdune:DisambigFromSpacePoints          0.11176      0.208027      0.482385      0.167993      0.100089        15     
produce:pandora:StandardPandora                    13.45        37.7736       151.058       26.6402       33.6961        15     
produce:pandoraTrack:LArPandoraTrackCreation     0.627467       1.98195       8.16166       1.32794       1.84704        15     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0286373     0.0414294     0.0612855     0.0391549    0.00948322       15     
[art]:TriggerResults:TriggerResultInserter      3.5196e-05    5.71386e-05   0.000109175   5.8269e-05    1.6068e-05       15     
end_path:out1:RootOutput                         8.226e-06    1.38181e-05   3.5086e-05    1.2814e-05    6.26597e-06      15     
end_path:out1:RootOutput(write)                   4.58982       4.99145       6.06431       4.82999      0.412354        15     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.87 MB
  Peak resident set size usage (VmHWM): 6707.23 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 02:45:04 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 3282
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:23:24 UTC       justIN version: 01.05.01