justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241282.1@dunegpschedd01.fnal.gov

Jobsub ID241282.1@dunegpschedd01.fnal.gov
Workflow ID9373
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 02:48:27
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-10-31 03:37:20
From worker nodeHostnamewn-pep-012.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 02:50:03
Input filesvd-protodune:np02vd_raw_run040266_0017_df-s03-d2_dw_0_20251024T214052.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 03:37:20
Saved logsjustin-logs:241282.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.305] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=3282 time=74 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[04:35:00.306] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 input frame: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[04:35:00.306] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 init nticks=8064 tbinmin=0 tbinmax=8064 
[04:35:00.349] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 load plane index: 0, ntraces=1536, input bad regions: 0 
[04:35:02.470] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 load plane index: 1, ntraces=1536, input bad regions: 0 
[04:35:04.656] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 load plane index: 2, ntraces=1536, input bad regions: 0 
[04:35:19.629] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 0, Qtot=1597475914 Qloss=-44434254, 8992 indices spanning [42225,51216] "wiener" 
[04:35:19.878] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 0, Qtot=1569057724 Qloss=-33962514, 7114 indices spanning [51217,58330] "gauss" 
[04:35:20.551] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 1, Qtot=374018808 Qloss=-40901309, 8569 indices spanning [58331,66899] "wiener" 
[04:35:20.797] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 1, Qtot=337259771 Qloss=-32036510, 6411 indices spanning [66900,73310] "gauss" 
[04:35:21.243] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 2, Qtot=360472365 Qloss=-31839779, 12099 indices spanning [73311,85409] "wiener" 
[04:35:21.656] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 2, Qtot=343558979 Qloss=-14455116, 9524 indices spanning [85410,94933] "gauss" 
[04:35:21.657] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 produce 94934 traces: 29660 wiener7, 0 decon_charge7, 23049 gauss7, frame tag: sigproc 
[04:35:21.657] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 output frame: frame: ident=3282 time=74 tick=512 with 94934 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":23049 [0] "mp2_roi7":34007 [0] "mp3_roi7":8218 [0] "wiener7":29660 [29660] ] cmm:[ ] 
[04:35:23.261] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[04:35:23.262] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=3282 time=74 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[04:35:23.262] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 input frame: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[04:35:23.263] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 init nticks=8064 tbinmin=0 tbinmax=8064 
[04:35:23.303] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 load plane index: 0, ntraces=1536, input bad regions: 0 
[04:35:25.480] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 load plane index: 1, ntraces=1536, input bad regions: 0 
[04:35:27.759] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 load plane index: 2, ntraces=1536, input bad regions: 0 
[04:35:45.017] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 0, Qtot=2455727316 Qloss=-62469839, 8301 indices spanning [41425,49725] "wiener" 
[04:35:45.256] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 0, Qtot=2420552954 Qloss=-53207934, 6550 indices spanning [49726,56275] "gauss" 
[04:35:45.972] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 1, Qtot=552662726 Qloss=-45210172, 9254 indices spanning [56276,65529] "wiener" 
[04:35:46.220] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 1, Qtot=470383695 Qloss=-37420795, 7344 indices spanning [65530,72873] "gauss" 
[04:35:46.669] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 2, Qtot=543584385 Qloss=-34409715, 12987 indices spanning [72874,85860] "wiener" 
[04:35:47.103] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 2, Qtot=525424367 Qloss=-15707483, 10067 indices spanning [85861,95927] "gauss" 
[04:35:47.104] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 produce 95928 traces: 30542 wiener6, 0 decon_charge6, 23961 gauss6, frame tag: sigproc 
[04:35:47.104] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 output frame: frame: ident=3282 time=74 tick=512 with 95928 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":23961 [0] "mp2_roi6":32969 [0] "mp3_roi6":8456 [0] "wiener6":30542 [30542] ] cmm:[ ] 
[04:35:49.335] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[04:35:49.336] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=3282 time=74 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[04:35:49.336] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 input frame: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[04:35:49.336] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 init nticks=8064 tbinmin=0 tbinmax=8064 
[04:35:49.374] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 load plane index: 0, ntraces=1536, input bad regions: 0 
[04:35:51.587] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 load plane index: 1, ntraces=1536, input bad regions: 0 
[04:35:53.902] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.1842e-05      163.708       284.157       164.893       59.8562        16     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.3123e-05    0.00011223    0.00080813    6.37295e-05   0.000180031      16     
produce:tpcrawdecoder:PDVDTPCReader               15.449        38.048        85.3751       29.5981       19.4961        16     
produce:triggerrawdecoder:PDVDTriggerReader4     0.511751      0.594635      0.894754      0.546821      0.116422        16     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000376265   0.000426221   0.000766107   0.00040786    8.87953e-05      16     
produce:ophit:OpHitFinder                       5.4198e-05    8.84498e-05   0.000492951   5.9665e-05    0.000104703      16     
produce:opflash:OpFlashFinderVerticalDrift       4.234e-05    6.30091e-05   0.000311717   4.5326e-05    6.42931e-05      16     
produce:wclsdatavd:WireCellToolkit                59.544        82.1234       140.502       74.4967       23.8254        15     
produce:gaushit:GausHitFinder                    0.939845       1.34198       2.29742       1.22405      0.364858        15     
produce:nhitsfilter:NumberOfHitsFilter          0.000299262   0.000414706   0.000592631   0.000388241   8.54323e-05      15     
produce:reco3d:SpacePointSolver                   6.95047       11.9007       21.8535       10.3638       4.17661        15     
produce:hitpdune:DisambigFromSpacePoints         0.121032      0.219262      0.506117       0.16602      0.105886        15     
produce:pandora:StandardPandora                   14.4444       33.7929       128.836       24.6368       28.2608        15     
produce:pandoraTrack:LArPandoraTrackCreation     0.629874       1.9328        9.80469       1.12654       2.22109        15     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0252737     0.0353144     0.0522825     0.0317736    0.00806205       15     
[art]:TriggerResults:TriggerResultInserter       2.291e-05    3.66593e-05   8.6662e-05    3.2623e-05    1.56618e-05      15     
end_path:out1:RootOutput                         4.11e-06     9.30067e-06   2.7934e-05     8.488e-06    5.58496e-06      15     
end_path:out1:RootOutput(write)                   4.06072       4.48009       5.59889       4.34831      0.418017        15     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.89 MB
  Peak resident set size usage (VmHWM): 6709.16 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 04:36:52 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 3282
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:25:58 UTC       justIN version: 01.05.01