justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 46965.0@dunegpschedd01.fnal.gov

Jobsub ID46965.0@dunegpschedd01.fnal.gov
Workflow ID2332
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 10:58:50
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-09-16 11:23:19
From worker nodeHostnamewn-choc-034.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 11:00:43
Input filesvd-protodune:np02vd_raw_run039343_0032_df-s05-d2_dw_0_20250908T125053.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-16 11:23:19
Saved logsjustin-logs:46965.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 13:20:33 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 13:20:33 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 13:20:33 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 13:20:33 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 13:20:33 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6400 keeping as is
[13:20:33.142] D [  main  ] executing 1 apps, thread limit 0:
[13:20:33.142] D [  main  ] executing 1 apps, thread limit 0:
[13:20:33.142] D [  main  ] executing app: "Pgrapher"
[13:20:33.142] D [ pgraph ] <Pgrapher:> executing graph 
[13:20:33.142] D [ pgraph ] executing with 26 nodes
[13:20:33.144] D [  glue  ] <FrameFanout:nfsp> call=6: input: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]  
[13:20:33.144] W [  glue  ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped. 
[13:20:33.145] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=17116 time=1 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:20:33.145] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 input frame: frame: ident=17116 time=1 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:20:33.146] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 init nticks=6400 tbinmin=0 tbinmax=6400 
[13:20:33.182] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 load plane index: 0, ntraces=1536, input bad regions: 0 
[13:20:34.715] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 load plane index: 1, ntraces=1536, input bad regions: 0 
[13:20:36.317] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 load plane index: 2, ntraces=1536, input bad regions: 0 
[13:21:24.396] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 save plane index: 0, Qtot=163425163567 Qloss=-54396158975, 6685 indices spanning [16311,22995] "wiener" 
[13:21:24.599] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 save plane index: 0, Qtot=160480135473 Qloss=-54861638839, 6181 indices spanning [22996,29176] "gauss" 
[13:21:26.077] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 save plane index: 1, Qtot=52767643489 Qloss=-10944828484, 18052 indices spanning [29177,47228] "wiener" 
[13:21:26.288] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 save plane index: 1, Qtot=44660763829 Qloss=-11208793303, 17662 indices spanning [47229,64890] "gauss" 
[13:21:26.663] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 save plane index: 2, Qtot=2657290194 Qloss=-3710824228, 12002 indices spanning [64891,76892] "wiener" 
[13:21:27.030] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 save plane index: 2, Qtot=2040437807 Qloss=-3094915468, 13294 indices spanning [76893,90186] "gauss" 
[13:21:27.031] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 produce 90187 traces: 36739 wiener7, 0 decon_charge7, 37137 gauss7, frame tag: sigproc 
[13:21:27.031] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 output frame: frame: ident=17116 time=1 tick=512 with 90187 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":37137 [0] "mp2_roi7":11782 [0] "mp3_roi7":4529 [0] "wiener7":36739 [36739] ] cmm:[ ] 
[13:21:32.398] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[13:21:32.399] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=17116 time=1 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:21:32.399] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=6 input frame: frame: ident=17116 time=1 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:21:32.399] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=6 init nticks=6400 tbinmin=0 tbinmax=6400 
[13:21:32.439] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=6 load plane index: 0, ntraces=1536, input bad regions: 0 
[13:21:34.273] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=6 load plane index: 1, ntraces=1536, input bad regions: 0 
[13:21:36.141] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=6 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.1828e-05      272.54        682.283       203.94        251.784         4     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.7416e-05    8.89355e-05   0.000175925   6.12005e-05   5.02491e-05       4     
produce:tpcrawdecoder:PDVDTPCReader               38.6658       45.0078       49.0178       46.1738       3.89794         4     
produce:triggerrawdecoder:PDVDTriggerReader4     0.523899      0.569566       0.5931       0.580633      0.0270079        4     
produce:pdvddaphne:DAPHNEReaderPDVD               8.50868       9.52616       11.3805       9.1077        1.1318          4     
produce:ophit:OpHitFinder                        0.0407648     0.045994      0.0483649     0.0474232    0.00304747        4     
produce:opflash:OpFlashFinderVerticalDrift      0.00919175     0.0142882     0.0166444     0.0156583    0.00297249        4     
produce:wclsdatavd:WireCellToolkit                57.1919       76.6964       113.447       59.4509       26.0026         3     
produce:gaushit:GausHitFinder                     1.18784        3.158        6.64213       1.64402       2.47068         3     
produce:nhitsfilter:NumberOfHitsFilter          0.000299117   0.00047639    0.000572509   0.000557543    0.0001255        3     
produce:reco3d:SpacePointSolver                   9.57592       18.199        28.2176       16.8034       7.67415         3     
produce:hitpdune:DisambigFromSpacePoints         0.144818      0.307602      0.469349      0.308639      0.132491         3     
produce:pandora:StandardPandora                   39.9253       205.004       472.094       102.992       190.608         3     
produce:pandoraTrack:LArPandoraTrackCreation     0.883947       1.46563       1.82269       1.69026      0.414851         3     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0254563     0.0366386     0.0425177     0.0419419    0.00791063        3     
[art]:TriggerResults:TriggerResultInserter      2.2021e-05    3.7531e-05    6.5266e-05    2.5306e-05    1.96574e-05       3     
end_path:out1:RootOutput                         7.547e-06    1.28233e-05   2.3308e-05     7.615e-06    7.41383e-06       3     
end_path:out1:RootOutput(write)                   4.21435       4.6341        5.02187       4.66608      0.330442         3     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.81 MB
  Peak resident set size usage (VmHWM): 6713.26 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 13:23:01 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39343 subRun: 1 event: 17116
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-18 19:58:04 UTC       justIN version: 01.05.00