justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40469.1@dunegpschedd02.fnal.gov

Jobsub ID40469.1@dunegpschedd02.fnal.gov
Workflow ID2323
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 16:42:07
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-09-16 18:08:38
From worker nodeHostnamewn-lot-027.farm.nikhef.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 16:42:51
Input filesvd-protodune:np02vd_raw_run039324_1740_df-s04-d0_dw_0_20250907T045121.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-16 18:08:38
Saved logsjustin-logs:40469.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 20:06:35 CEST  run: 39324 subRun: 1 event: 912770
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 20:06:35 CEST  run: 39324 subRun: 1 event: 912770
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 20:06:35 CEST  run: 39324 subRun: 1 event: 912770
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 20:06:35 CEST  run: 39324 subRun: 1 event: 912770
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 20:06:35 CEST  run: 39324 subRun: 1 event: 912770
Error! unrecognized channel number -1. Ignoring pulse
%MSG
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6400 keeping as is
[20:06:35.885] D [  main  ] executing 1 apps, thread limit 0:
[20:06:35.885] D [  main  ] executing 1 apps, thread limit 0:
[20:06:35.885] D [  main  ] executing app: "Pgrapher"
[20:06:35.885] D [ pgraph ] <Pgrapher:> executing graph 
[20:06:35.885] D [ pgraph ] executing with 26 nodes
[20:06:35.886] D [  glue  ] <FrameFanout:nfsp> call=48: input: frame: ident=912770 time=57 tick=512 with 12288 traces.  frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=912770 time=57 tick=512 with 12288 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=912770 time=57 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=912770 time=57 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=912770 time=57 tick=512 with 12288 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=912770 time=57 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=912770 time=57 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=912770 time=57 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=912770 time=57 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]  
[20:06:35.887] W [  glue  ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped. 
[20:06:35.887] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=912770 time=57 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=912770 time=57 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:06:35.887] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 input frame: frame: ident=912770 time=57 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:06:35.887] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 init nticks=6400 tbinmin=0 tbinmax=6400 
[20:06:35.912] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 load plane index: 0, ntraces=1536, input bad regions: 0 
[20:06:37.126] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 load plane index: 1, ntraces=1536, input bad regions: 0 
[20:06:38.482] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 load plane index: 2, ntraces=1536, input bad regions: 0 
[20:07:05.028] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 save plane index: 0, Qtot=2969475778 Qloss=-58492884, 6785 indices spanning [17298,24082] "wiener" 
[20:07:05.197] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 save plane index: 0, Qtot=2929439647 Qloss=-53977161, 5324 indices spanning [24083,29406] "gauss" 
[20:07:05.765] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 save plane index: 1, Qtot=551526858 Qloss=-32753232, 8478 indices spanning [29407,37884] "wiener" 
[20:07:05.937] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 save plane index: 1, Qtot=464868618 Qloss=-25706399, 6846 indices spanning [37885,44730] "gauss" 
[20:07:06.257] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 save plane index: 2, Qtot=596680136 Qloss=-33452356, 9767 indices spanning [44731,54497] "wiener" 
[20:07:06.565] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 save plane index: 2, Qtot=579209805 Qloss=-15783261, 7376 indices spanning [54498,61873] "gauss" 
[20:07:06.565] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 produce 61874 traces: 25030 wiener7, 0 decon_charge7, 19546 gauss7, frame tag: sigproc 
[20:07:06.565] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=48 output frame: frame: ident=912770 time=57 tick=512 with 61874 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":19546 [0] "mp2_roi7":10309 [0] "mp3_roi7":6989 [0] "wiener7":25030 [25030] ] cmm:[ ] 
[20:07:10.057] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[20:07:10.057] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=912770 time=57 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=912770 time=57 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:07:10.057] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=48 input frame: frame: ident=912770 time=57 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:07:10.057] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=48 init nticks=6400 tbinmin=0 tbinmax=6400 
[20:07:10.083] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=48 load plane index: 0, ntraces=1536, input bad regions: 0 
[20:07:11.567] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=48 load plane index: 1, ntraces=1536, input bad regions: 0 
[20:07:13.129] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=48 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      8.2815e-05      196.744       378.926       181.313       71.0044        25     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.4923e-05    7.30626e-05   0.00012297     6.937e-05    1.37526e-05      25     
produce:tpcrawdecoder:PDVDTPCReader               43.8817       62.5958       95.3174       60.5909       11.4142        25     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0340902     0.0383332     0.0655383     0.0344996    0.00925706       25     
produce:pdvddaphne:DAPHNEReaderPDVD               4.39002       8.23007       11.5873       7.55704       1.8354         25     
produce:ophit:OpHitFinder                        0.0343759     0.0420595     0.0474804     0.0419222    0.00334814       25     
produce:opflash:OpFlashFinderVerticalDrift      0.00859306     0.0119391     0.0192134     0.0111122    0.00270234       25     
produce:wclsdatavd:WireCellToolkit                35.8392       46.6184       77.9711       44.7698       8.4013         24     
produce:gaushit:GausHitFinder                    0.478235       1.12341       1.73387       1.10689      0.261058        24     
produce:nhitsfilter:NumberOfHitsFilter          0.000118572   0.000385939   0.00110338    0.00033611    0.000203365      24     
produce:reco3d:SpacePointSolver                   4.10588       13.4514       23.8233       12.7122       4.13535        24     
produce:hitpdune:DisambigFromSpacePoints         0.0626367     0.197156      0.344889      0.182476      0.0701107       24     
produce:pandora:StandardPandora                   15.4143       67.6227       232.938       51.7095       48.8855        24     
produce:pandoraTrack:LArPandoraTrackCreation     0.191345      0.654393       1.75424      0.531181      0.325682        24     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0122292     0.0263646     0.0428892     0.0261034    0.00670374       24     
[art]:TriggerResults:TriggerResultInserter      1.4116e-05    1.95333e-05   5.6135e-05    1.8204e-05    7.96322e-06      24     
end_path:out1:RootOutput                         3.196e-06    4.81329e-06   1.8936e-05     4.088e-06    2.98574e-06      24     
end_path:out1:RootOutput(write)                   3.50433       4.08809       6.1264        4.04539       0.47071        24     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.87 MB
  Peak resident set size usage (VmHWM): 6709.74 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 20:08:20 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39324 subRun: 1 event: 912770
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-18 19:55:50 UTC       justIN version: 01.05.00