justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 235762.1@dunegpschedd02.fnal.gov

Jobsub ID235762.1@dunegpschedd02.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 12:58:59
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-10-31 13:07:24
From worker nodeHostnamewn-sate-037.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 12:59:51
Input filesvd-protodune:np02vd_raw_run040266_0351_df-s03-d1_dw_0_20251025T070137.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 13:07:24
Saved logsjustin-logs:235762.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

upported, summary will be dropped. 
[14:05:29.922] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=64833 time=2 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=64833 time=2 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[14:05:29.923] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 input frame: frame: ident=64833 time=2 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[14:05:29.923] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 init nticks=8352 tbinmin=0 tbinmax=8352 
[14:05:29.952] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 0, ntraces=1536, input bad regions: 0 
[14:05:31.784] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 1, ntraces=1536, input bad regions: 0 
[14:05:33.654] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 2, ntraces=1536, input bad regions: 0 
[14:05:46.593] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 0, Qtot=1506387823 Qloss=-34533921, 6026 indices spanning [29614,35639] "wiener" 
[14:05:46.858] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 0, Qtot=1477713649 Qloss=-28713787, 4789 indices spanning [35640,40428] "gauss" 
[14:05:47.462] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 1, Qtot=398944908 Qloss=-28885405, 6725 indices spanning [40429,47153] "wiener" 
[14:05:47.712] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 1, Qtot=356988155 Qloss=-21657004, 5254 indices spanning [47154,52407] "gauss" 
[14:05:48.136] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 2, Qtot=393862047 Qloss=-19056682, 7711 indices spanning [52408,60118] "wiener" 
[14:05:48.551] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 2, Qtot=382969197 Qloss=-7739980, 5813 indices spanning [60119,65931] "gauss" 
[14:05:48.552] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 produce 65932 traces: 20462 wiener7, 0 decon_charge7, 15856 gauss7, frame tag: sigproc 
[14:05:48.552] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 output frame: frame: ident=64833 time=2 tick=512 with 65932 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":15856 [0] "mp2_roi7":22051 [0] "mp3_roi7":7563 [0] "wiener7":20462 [20462] ] cmm:[ ] 
[14:05:49.591] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[14:05:49.591] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=64833 time=2 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=64833 time=2 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[14:05:49.591] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 input frame: frame: ident=64833 time=2 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[14:05:49.591] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 init nticks=8352 tbinmin=0 tbinmax=8352 
[14:05:49.624] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 0, ntraces=1536, input bad regions: 0 
[14:05:51.764] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 1, ntraces=1536, input bad regions: 0 
[14:05:54.000] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 2, ntraces=1536, input bad regions: 0 
[14:05:57.341] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 0, Qtot=106795177 Qloss=-13347219, 2457 indices spanning [12675,15131] "wiener" 
[14:05:57.580] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 0, Qtot=102018046 Qloss=-10701676, 1883 indices spanning [15132,17014] "gauss" 
[14:05:57.944] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 1, Qtot=108818365 Qloss=-12864195, 2461 indices spanning [17015,19475] "wiener" 
[14:05:58.156] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 1, Qtot=100527160 Qloss=-11165043, 2031 indices spanning [19476,21506] "gauss" 
[14:05:58.642] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 2, Qtot=102152852 Qloss=-7177655, 3264 indices spanning [21507,24770] "wiener" 
[14:05:59.119] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 2, Qtot=98406977 Qloss=-3330790, 2604 indices spanning [24771,27374] "gauss" 
[14:05:59.119] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 produce 27375 traces: 8182 wiener6, 0 decon_charge6, 6518 gauss6, frame tag: sigproc 
[14:05:59.119] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 output frame: frame: ident=64833 time=2 tick=512 with 27375 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":6518 [0] "mp2_roi6":10700 [0] "mp3_roi6":1975 [0] "wiener6":8182 [8182] ] cmm:[ ] 
[14:05:59.154] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[14:05:59.154] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=64833 time=2 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=64833 time=2 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[14:05:59.154] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 input frame: frame: ident=64833 time=2 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[14:05:59.154] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 init nticks=8352 tbinmin=0 tbinmax=8352 
[14:05:59.184] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 load plane index: 0, ntraces=1536, input bad regions: 0 
[14:06:01.257] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 load plane index: 1, ntraces=1536, input bad regions: 0 
[14:06:03.458] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.4371e-05      138.27        276.541       138.27        138.27          2     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.4371e-05    0.000136136   0.000207901   0.000136136   7.1765e-05        2     
produce:tpcrawdecoder:PDVDTPCReader               23.6909       105.011       186.332       105.011       81.3206         2     
produce:triggerrawdecoder:PDVDTriggerReader4     0.553274      0.720442      0.887611      0.720442      0.167168         2     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000403849   0.000577906   0.000751964   0.000577906   0.000174057       2     
produce:ophit:OpHitFinder                       6.6866e-05    0.000301852   0.000536839   0.000301852   0.000234986       2     
produce:opflash:OpFlashFinderVerticalDrift      5.1747e-05    0.00018614    0.000320533   0.00018614    0.000134393       2     
produce:wclsdatavd:WireCellToolkit                52.2873       52.2873       52.2873       52.2873          0            1     
produce:gaushit:GausHitFinder                     1.04471       1.04471       1.04471       1.04471          0            1     
produce:nhitsfilter:NumberOfHitsFilter          0.000488078   0.000488078   0.000488078   0.000488078        0            1     
produce:reco3d:SpacePointSolver                   10.4615       10.4615       10.4615       10.4615          0            1     
produce:hitpdune:DisambigFromSpacePoints         0.129183      0.129183      0.129183      0.129183          0            1     
produce:pandora:StandardPandora                   19.8292       19.8292       19.8292       19.8292          0            1     
produce:pandoraTrack:LArPandoraTrackCreation     0.770057      0.770057      0.770057      0.770057          0            1     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0338792     0.0338792     0.0338792     0.0338792         0            1     
[art]:TriggerResults:TriggerResultInserter      6.4041e-05    6.4041e-05    6.4041e-05    6.4041e-05         0            1     
end_path:out1:RootOutput                        2.2392e-05    2.2392e-05    2.2392e-05    2.2392e-05         0            1     
end_path:out1:RootOutput(write)                   4.73887       4.73887       4.73887       4.73887          0            1     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.83 MB
  Peak resident set size usage (VmHWM): 6705.5 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 14:07:01 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 64833
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:23:20 UTC       justIN version: 01.05.01