justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 47053.0@dunegpschedd01.fnal.gov

Jobsub ID47053.0@dunegpschedd01.fnal.gov
Workflow ID2405
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 16:42:07
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_brug
Last heartbeat2025-09-16 18:09:19
From worker nodeHostnamewn-lot-027.farm.nikhef.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 16:42:48
Input filesvd-protodune:np02vd_raw_run039255_0301_df-s05-d5_dw_0_20250830T135701.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-16 18:09:19
Saved logsjustin-logs:47053.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ophit@BeginModule 16-Sep-2025 20:06:28 CEST  run: 39255 subRun: 1 event: 108659
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 20:06:28 CEST  run: 39255 subRun: 1 event: 108659
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 20:06:28 CEST  run: 39255 subRun: 1 event: 108659
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 20:06:28 CEST  run: 39255 subRun: 1 event: 108659
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 20:06:28 CEST  run: 39255 subRun: 1 event: 108659
Error! unrecognized channel number -1. Ignoring pulse
%MSG
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=10048 keeping as is
[20:06:29.067] D [  main  ] executing 1 apps, thread limit 0:
[20:06:29.067] D [  main  ] executing 1 apps, thread limit 0:
[20:06:29.067] D [  main  ] executing app: "Pgrapher"
[20:06:29.067] D [ pgraph ] <Pgrapher:> executing graph 
[20:06:29.067] D [ pgraph ] executing with 26 nodes
[20:06:29.068] D [  glue  ] <FrameFanout:nfsp> call=28: input: frame: ident=108659 time=22 tick=512 with 12288 traces.  frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=108659 time=22 tick=512 with 12288 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=108659 time=22 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=108659 time=22 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=108659 time=22 tick=512 with 12288 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=108659 time=22 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=108659 time=22 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=108659 time=22 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=108659 time=22 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]  
[20:06:29.069] W [  glue  ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped. 
[20:06:29.069] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=108659 time=22 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=108659 time=22 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:06:29.069] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 input frame: frame: ident=108659 time=22 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:06:29.070] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 init nticks=10048 tbinmin=0 tbinmax=10048 
[20:06:29.104] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 load plane index: 0, ntraces=1536, input bad regions: 0 
[20:06:32.300] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 load plane index: 1, ntraces=1536, input bad regions: 0 
[20:06:35.617] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 load plane index: 2, ntraces=1536, input bad regions: 0 
[20:07:15.479] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 save plane index: 0, Qtot=6672723363 Qloss=-307111150, 17372 indices spanning [68130,85501] "wiener" 
[20:07:15.838] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 save plane index: 0, Qtot=6561821590 Qloss=-291491190, 13434 indices spanning [85502,98935] "gauss" 
[20:07:17.158] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 save plane index: 1, Qtot=1671921419 Qloss=-107831591, 18837 indices spanning [98936,117772] "wiener" 
[20:07:17.518] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 save plane index: 1, Qtot=1484112651 Qloss=-84998575, 14915 indices spanning [117773,132687] "gauss" 
[20:07:18.128] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 save plane index: 2, Qtot=1262446029 Qloss=-31718964, 14352 indices spanning [132688,147039] "wiener" 
[20:07:18.720] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 save plane index: 2, Qtot=1215973101 Qloss=-11823949, 11095 indices spanning [147040,158134] "gauss" 
[20:07:18.721] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 produce 158135 traces: 50561 wiener7, 0 decon_charge7, 39444 gauss7, frame tag: sigproc 
[20:07:18.721] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 output frame: frame: ident=108659 time=22 tick=512 with 158135 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":39444 [0] "mp2_roi7":44859 [0] "mp3_roi7":23271 [0] "wiener7":50561 [50561] ] cmm:[ ] 
[20:07:24.441] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[20:07:24.441] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=108659 time=22 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=108659 time=22 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:07:24.441] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=28 input frame: frame: ident=108659 time=22 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[20:07:24.441] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=28 init nticks=10048 tbinmin=0 tbinmax=10048 
[20:07:24.472] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=28 load plane index: 0, ntraces=1536, input bad regions: 0 
[20:07:27.591] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=28 load plane index: 1, ntraces=1536, input bad regions: 0 
[20:07:30.807] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=28 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      8.5871e-05      324.461       610.482       314.353       120.258        15     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.4611e-05    7.67377e-05   0.00013313    7.1484e-05    1.65174e-05      15     
produce:tpcrawdecoder:PDVDTPCReader               88.7818       100.668       117.961       96.2496       8.30552        15     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0343294     0.0464344     0.0932708     0.0418569     0.0153496       15     
produce:pdvddaphne:DAPHNEReaderPDVD               9.6906        11.565        15.0706       10.9463       1.55454        15     
produce:ophit:OpHitFinder                        0.034839      0.0513039     0.0598566     0.0543931    0.00706338       15     
produce:opflash:OpFlashFinderVerticalDrift      0.00866352     0.0144897     0.0180508     0.0150454    0.00268058       15     
produce:wclsdatavd:WireCellToolkit                66.4341       85.4466       99.8553       85.9164       8.8072         14     
produce:gaushit:GausHitFinder                     1.19325       1.60281       2.03972       1.5373       0.264901        14     
produce:nhitsfilter:NumberOfHitsFilter          0.000399908   0.00071163    0.00120872    0.000685844   0.000261812      14     
produce:reco3d:SpacePointSolver                   14.0999       24.5426       35.9307       23.8216       5.90177        14     
produce:hitpdune:DisambigFromSpacePoints         0.186637      0.389422      0.597146      0.362909      0.113992        14     
produce:pandora:StandardPandora                   39.7188       115.673       368.357       87.8359       80.5653        14     
produce:pandoraTrack:LArPandoraTrackCreation     0.785346       1.70406       2.64986       1.58684      0.529771        14     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0275365     0.0407642     0.0508698     0.0422847    0.00683268       14     
[art]:TriggerResults:TriggerResultInserter      1.6491e-05    2.1493e-05    4.9472e-05    1.9757e-05    7.94483e-06      14     
end_path:out1:RootOutput                         3.476e-06    5.51936e-06   1.9135e-05     4.553e-06    3.81883e-06      14     
end_path:out1:RootOutput(write)                   5.50058       6.02563       6.55058       6.02964      0.244464        14     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.86 MB
  Peak resident set size usage (VmHWM): 6708.66 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 20:09:01 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39255 subRun: 1 event: 108659
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-18 19:28:44 UTC       justIN version: 01.05.00