justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 47303.0@dunegpschedd01.fnal.gov

Jobsub ID47303.0@dunegpschedd01.fnal.gov
Workflow ID2644
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-17 02:58:42
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-09-17 03:04:32
From worker nodeHostnamewn-snel-010.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-17 02:59:20
Input filesvd-protodune:np02vd_raw_run039353_1005_df-s05-d3_dw_0_20250916T120858.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-17 03:04:32
Saved logsjustin-logs:47303.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

rted, summary will be dropped. 
[05:02:47.215] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=77367 time=4 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=77367 time=4 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:02:47.216] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 input frame: frame: ident=77367 time=4 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:02:47.216] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 init nticks=10000 tbinmin=0 tbinmax=10000 
[05:02:47.273] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:02:49.684] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:02:51.983] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 2, ntraces=1536, input bad regions: 0 
[05:02:55.238] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 0, Qtot=106024167 Qloss=-13861091, 3081 indices spanning [12995,16075] "wiener" 
[05:02:55.453] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 0, Qtot=100892865 Qloss=-10558875, 2544 indices spanning [16076,18619] "gauss" 
[05:02:55.829] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 1, Qtot=105257155 Qloss=-14760066, 3462 indices spanning [18620,22081] "wiener" 
[05:02:56.045] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 1, Qtot=97848788 Qloss=-11303747, 2798 indices spanning [22082,24879] "gauss" 
[05:02:56.406] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 2, Qtot=100136502 Qloss=-7988728, 3776 indices spanning [24880,28655] "wiener" 
[05:02:56.758] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 2, Qtot=96523520 Qloss=-4278502, 3164 indices spanning [28656,31819] "gauss" 
[05:02:56.760] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 produce 31820 traces: 10319 wiener7, 0 decon_charge7, 8506 gauss7, frame tag: sigproc 
[05:02:56.760] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 output frame: frame: ident=77367 time=4 tick=512 with 31820 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":8506 [0] "mp2_roi7":10595 [0] "mp3_roi7":2400 [0] "wiener7":10319 [10319] ] cmm:[ ] 
[05:02:56.825] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[05:02:56.826] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=77367 time=4 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=77367 time=4 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:02:56.826] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 input frame: frame: ident=77367 time=4 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:02:56.826] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 init nticks=10000 tbinmin=0 tbinmax=10000 
[05:02:56.866] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:02:58.862] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:03:00.923] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 2, ntraces=1536, input bad regions: 0 
[05:03:04.204] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 0, Qtot=116305841 Qloss=-14257486, 2960 indices spanning [17050,20009] "wiener" 
[05:03:04.413] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 0, Qtot=109298468 Qloss=-12008097, 2386 indices spanning [20010,22395] "gauss" 
[05:03:04.810] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 1, Qtot=114948793 Qloss=-14611353, 3579 indices spanning [22396,25974] "wiener" 
[05:03:05.021] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 1, Qtot=104821134 Qloss=-11427851, 2896 indices spanning [25975,28870] "gauss" 
[05:03:05.354] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 2, Qtot=116245827 Qloss=-9348207, 4480 indices spanning [28871,33350] "wiener" 
[05:03:05.709] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 2, Qtot=111386743 Qloss=-4332912, 3702 indices spanning [33351,37052] "gauss" 
[05:03:05.711] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 produce 37053 traces: 11019 wiener6, 0 decon_charge6, 8984 gauss6, frame tag: sigproc 
[05:03:05.711] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 output frame: frame: ident=77367 time=4 tick=512 with 37053 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":8984 [0] "mp2_roi6":14426 [0] "mp3_roi6":2624 [0] "wiener6":11019 [11019] ] cmm:[ ] 
[05:03:05.799] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[05:03:05.799] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=77367 time=4 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=77367 time=4 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:03:05.800] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 input frame: frame: ident=77367 time=4 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:03:05.800] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 init nticks=10000 tbinmin=0 tbinmax=10000 
[05:03:05.835] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:03:07.865] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:03:09.826] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      0.000103966     74.0738       148.147       74.0738       74.0737         2     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      0.000103966   0.000230919   0.000357872   0.000230919   0.000126953       2     
produce:tpcrawdecoder:PDVDTPCReader               17.9298       18.2215       18.5132       18.2215      0.291702         2     
produce:triggerrawdecoder:PDVDTriggerReader4     0.300042      0.315231      0.330419      0.315231      0.0151886        2     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000604345   0.000880248   0.00115615    0.000880248   0.000275903       2     
produce:ophit:OpHitFinder                       0.000430308   0.000856268   0.00128223    0.000856268   0.00042596        2     
produce:opflash:OpFlashFinderVerticalDrift      8.0701e-05    0.000308975   0.000537249   0.000308975   0.000228274       2     
produce:wclsdatavd:WireCellToolkit                75.6981       75.6981       75.6981       75.6981          0            1     
produce:gaushit:GausHitFinder                     1.07959       1.07959       1.07959       1.07959          0            1     
produce:nhitsfilter:NumberOfHitsFilter          0.00126453    0.00126453    0.00126453    0.00126453         0            1     
produce:reco3d:SpacePointSolver                   9.17007       9.17007       9.17007       9.17007          0            1     
produce:hitpdune:DisambigFromSpacePoints         0.145505      0.145505      0.145505      0.145505          0            1     
produce:pandora:StandardPandora                   38.5311       38.5311       38.5311       38.5311          0            1     
produce:pandoraTrack:LArPandoraTrackCreation     0.873096      0.873096      0.873096      0.873096          0            1     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0326158     0.0326158     0.0326158     0.0326158         0            1     
[art]:TriggerResults:TriggerResultInserter      9.3966e-05    9.3966e-05    9.3966e-05    9.3966e-05         0            1     
end_path:out1:RootOutput                        2.9275e-05    2.9275e-05    2.9275e-05    2.9275e-05         0            1     
end_path:out1:RootOutput(write)                   4.32975       4.32975       4.32975       4.32975          0            1     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.86 MB
  Peak resident set size usage (VmHWM): 6710.6 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 05:04:11 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39353 subRun: 1 event: 77367
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-18 20:36:08 UTC       justIN version: 01.05.00