justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 47411.24@dunegpschedd01.fnal.gov

Jobsub ID47411.24@dunegpschedd01.fnal.gov
Workflow ID2740
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-17 08:53:01
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel
Last heartbeat2025-09-17 10:02:15
From worker nodeHostnamewn-pep-010.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-17 09:26:02
Input filesvd-protodune:np02vd_raw_run039388_0571_df-s05-d2_dw_0_20250917T011350.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-17 10:02:15
Saved logsjustin-logs:47411.24-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[12:00:26.368] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=43994 time=16 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=43994 time=16 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:00:26.368] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 input frame: frame: ident=43994 time=16 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:00:26.368] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 init nticks=10000 tbinmin=0 tbinmax=10000 
[12:00:26.414] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 load plane index: 0, ntraces=1536, input bad regions: 0 
[12:00:28.522] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 load plane index: 1, ntraces=1536, input bad regions: 0 
[12:00:30.926] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 load plane index: 2, ntraces=1536, input bad regions: 0 
[12:00:42.168] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 save plane index: 0, Qtot=634281534 Qloss=-52458181, 11336 indices spanning [47246,58581] "wiener" 
[12:00:42.398] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 save plane index: 0, Qtot=595199575 Qloss=-41017103, 8928 indices spanning [58582,67509] "gauss" 
[12:00:43.053] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 save plane index: 1, Qtot=404617292 Qloss=-48314877, 10191 indices spanning [67510,77700] "wiener" 
[12:00:43.303] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 save plane index: 1, Qtot=358588592 Qloss=-36798613, 7729 indices spanning [77701,85429] "gauss" 
[12:00:43.723] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 save plane index: 2, Qtot=382262426 Qloss=-27103821, 11605 indices spanning [85430,97034] "wiener" 
[12:00:44.130] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 save plane index: 2, Qtot=367655054 Qloss=-12184346, 9614 indices spanning [97035,106648] "gauss" 
[12:00:44.131] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 produce 106649 traces: 33132 wiener7, 0 decon_charge7, 26271 gauss7, frame tag: sigproc 
[12:00:44.131] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 output frame: frame: ident=43994 time=16 tick=512 with 106649 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":26271 [0] "mp2_roi7":35907 [0] "mp3_roi7":11339 [0] "wiener7":33132 [33132] ] cmm:[ ] 
[12:00:44.918] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[12:00:44.919] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=43994 time=16 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=43994 time=16 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:00:44.919] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 input frame: frame: ident=43994 time=16 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:00:44.919] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 init nticks=10000 tbinmin=0 tbinmax=10000 
[12:00:44.962] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 load plane index: 0, ntraces=1536, input bad regions: 0 
[12:00:47.283] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 load plane index: 1, ntraces=1536, input bad regions: 0 
[12:00:49.562] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 load plane index: 2, ntraces=1536, input bad regions: 0 
[12:00:57.334] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 save plane index: 0, Qtot=324183867 Qloss=-35109061, 9433 indices spanning [49307,58739] "wiener" 
[12:00:57.576] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 save plane index: 0, Qtot=305138540 Qloss=-26433118, 7153 indices spanning [58740,65892] "gauss" 
[12:00:58.225] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 save plane index: 1, Qtot=321131173 Qloss=-41925500, 9448 indices spanning [65893,75340] "wiener" 
[12:00:58.477] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 save plane index: 1, Qtot=295725904 Qloss=-30814653, 7219 indices spanning [75341,82559] "gauss" 
[12:00:58.952] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 save plane index: 2, Qtot=299435948 Qloss=-21811148, 9852 indices spanning [82560,92411] "wiener" 
[12:00:59.406] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 save plane index: 2, Qtot=288105262 Qloss=-10130446, 8057 indices spanning [92412,100468] "gauss" 
[12:00:59.406] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 produce 100469 traces: 28733 wiener6, 0 decon_charge6, 22429 gauss6, frame tag: sigproc 
[12:00:59.406] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 output frame: frame: ident=43994 time=16 tick=512 with 100469 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":22429 [0] "mp2_roi6":40153 [0] "mp3_roi6":9154 [0] "wiener6":28733 [28733] ] cmm:[ ] 
[12:00:59.839] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[12:00:59.839] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=43994 time=16 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=43994 time=16 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:00:59.840] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=8 input frame: frame: ident=43994 time=16 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:00:59.840] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=8 init nticks=10000 tbinmin=0 tbinmax=10000 
[12:00:59.882] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=8 load plane index: 0, ntraces=1536, input bad regions: 0 
[12:01:02.155] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=8 load plane index: 1, ntraces=1536, input bad regions: 0 
[12:01:04.488] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=8 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.0057e-05      344.225       472.006       405.916       174.358         5     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.4684e-05    7.53206e-05   0.000125326   6.5264e-05    2.56004e-05       5     
produce:tpcrawdecoder:PDVDTPCReader               202.202       231.068       311.618       213.188       40.5989         5     
produce:triggerrawdecoder:PDVDTriggerReader4     0.329415      0.591622       1.53919      0.346523      0.474325         5     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000376172   0.000453865   0.000750652   0.000379125   0.000148418       5     
produce:ophit:OpHitFinder                       0.000104814   0.00025552    0.000779748   0.000108067   0.000263558       5     
produce:opflash:OpFlashFinderVerticalDrift      4.1868e-05    9.67724e-05   0.000306087   4.3093e-05    0.000104699       5     
produce:wclsdatavd:WireCellToolkit                77.7472       81.8015       85.8938       81.7826       3.57664         4     
produce:gaushit:GausHitFinder                     1.62645       1.88375       2.1132        1.89768      0.216945         4     
produce:nhitsfilter:NumberOfHitsFilter          0.000548994   0.000630392   0.000720078   0.000626247   6.48883e-05       4     
produce:reco3d:SpacePointSolver                   16.5472       20.2162       24.5731       19.8723       3.06316         4     
produce:hitpdune:DisambigFromSpacePoints         0.268837      0.316209      0.380441      0.307778      0.045834         4     
produce:pandora:StandardPandora                   75.719        107.56        142.078       106.222       23.8497         4     
produce:pandoraTrack:LArPandoraTrackCreation      1.45903       2.02131       2.82795       1.89914      0.525015         4     
produce:pandoraGnocalo:GnocchiCalorimetry        0.036886      0.0408334     0.0460191     0.0402143    0.00386658        4     
[art]:TriggerResults:TriggerResultInserter       2.511e-05    4.03852e-05   8.0119e-05    2.8156e-05    2.29946e-05       4     
end_path:out1:RootOutput                         8.042e-06    1.3075e-05     2.762e-05     8.319e-06    8.3992e-06        4     
end_path:out1:RootOutput(write)                   5.01272       5.10064       5.1654        5.11223      0.058104         4     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.92 MB
  Peak resident set size usage (VmHWM): 6706.73 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 12:01:59 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39388 subRun: 1 event: 43994
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-18 19:28:30 UTC       justIN version: 01.05.00