justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241937.2@dunegpschedd01.fnal.gov

Jobsub ID241937.2@dunegpschedd01.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-02 11:01:25
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-11-02 11:09:00
From worker nodeHostnamewn-sate-037.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-02 11:03:19
Input filesvd-protodune:np02vd_raw_run040266_0667_df-s04-d1_dw_0_20251025T155243.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-02 11:09:00
Saved logsjustin-logs:241937.2-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

65] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=122941 time=3 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=122941 time=3 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:06:12.065] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 input frame: frame: ident=122941 time=3 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:06:12.065] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 init nticks=8644 tbinmin=0 tbinmax=8644 
[12:06:12.096] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 0, ntraces=1536, input bad regions: 0 
[12:06:14.482] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 1, ntraces=1536, input bad regions: 0 
[12:06:16.952] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 2, ntraces=1536, input bad regions: 0 
[12:06:55.768] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 0, Qtot=19617688554 Qloss=-291886466, 11007 indices spanning [32065,43071] "wiener" 
[12:06:56.119] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 0, Qtot=19487414795 Qloss=-277392203, 8515 indices spanning [43072,51586] "gauss" 
[12:06:57.126] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 1, Qtot=1136122326 Qloss=-87513397, 13956 indices spanning [51587,65542] "wiener" 
[12:06:57.461] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 1, Qtot=1033977463 Qloss=-64874481, 10564 indices spanning [65543,76106] "gauss" 
[12:06:58.000] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 2, Qtot=1275497168 Qloss=-38737278, 14783 indices spanning [76107,90889] "wiener" 
[12:06:58.526] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 2, Qtot=1253625674 Qloss=-16300100, 11279 indices spanning [90890,102168] "gauss" 
[12:06:58.527] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 produce 102169 traces: 39746 wiener7, 0 decon_charge7, 30358 gauss7, frame tag: sigproc 
[12:06:58.527] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 output frame: frame: ident=122941 time=3 tick=512 with 102169 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":30358 [0] "mp2_roi7":18851 [0] "mp3_roi7":13214 [0] "wiener7":39746 [39746] ] cmm:[ ] 
[12:07:03.638] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[12:07:03.639] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=122941 time=3 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=122941 time=3 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:07:03.639] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 input frame: frame: ident=122941 time=3 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:07:03.639] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 init nticks=8644 tbinmin=0 tbinmax=8644 
[12:07:03.669] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 0, ntraces=1536, input bad regions: 0 
[12:07:06.403] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 1, ntraces=1536, input bad regions: 0 
[12:07:09.156] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 2, ntraces=1536, input bad regions: 0 
[12:07:29.593] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 0, Qtot=805031583 Qloss=-74441161, 11194 indices spanning [36786,47979] "wiener" 
[12:07:29.870] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 0, Qtot=762469705 Qloss=-62370955, 8621 indices spanning [47980,56600] "gauss" 
[12:07:30.806] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 1, Qtot=675204122 Qloss=-68353276, 12604 indices spanning [56601,69204] "wiener" 
[12:07:31.135] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 1, Qtot=597977386 Qloss=-50887793, 9576 indices spanning [69205,78780] "gauss" 
[12:07:31.705] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 2, Qtot=718629076 Qloss=-35994051, 14018 indices spanning [78781,92798] "wiener" 
[12:07:32.258] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 save plane index: 2, Qtot=699036039 Qloss=-16127294, 11010 indices spanning [92799,103808] "gauss" 
[12:07:32.259] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 produce 103809 traces: 37816 wiener6, 0 decon_charge6, 29207 gauss6, frame tag: sigproc 
[12:07:32.259] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 output frame: frame: ident=122941 time=3 tick=512 with 103809 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":29207 [0] "mp2_roi6":25095 [0] "mp3_roi6":11691 [0] "wiener6":37816 [37816] ] cmm:[ ] 
[12:07:34.621] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[12:07:34.621] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=122941 time=3 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=122941 time=3 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:07:34.621] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 input frame: frame: ident=122941 time=3 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:07:34.622] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 init nticks=8644 tbinmin=0 tbinmax=8644 
[12:07:34.652] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 load plane index: 0, ntraces=1536, input bad regions: 0 
[12:07:37.365] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 load plane index: 1, ntraces=1536, input bad regions: 0 
[12:07:40.126] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=2 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      9.1512e-05      58.1547       116.309       58.1547       58.1546         2     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      9.1512e-05    0.000151766   0.000212019   0.000151766   6.02535e-05       2     
produce:tpcrawdecoder:PDVDTPCReader               13.7543       13.8542       13.954        13.8542      0.0998175        2     
produce:triggerrawdecoder:PDVDTriggerReader4     0.529609      0.548947      0.568285      0.548947      0.0193383        2     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000392028   0.000587565   0.000783102   0.000587565   0.000195537       2     
produce:ophit:OpHitFinder                       8.0241e-05    0.000338812   0.000597383   0.000338812   0.000258571       2     
produce:opflash:OpFlashFinderVerticalDrift      4.9804e-05    0.000212991   0.000376178   0.000212991   0.000163187       2     
produce:wclsdatavd:WireCellToolkit                54.7326       54.7326       54.7326       54.7326          0            1     
produce:gaushit:GausHitFinder                     1.08949       1.08949       1.08949       1.08949          0            1     
produce:nhitsfilter:NumberOfHitsFilter          0.000560374   0.000560374   0.000560374   0.000560374        0            1     
produce:reco3d:SpacePointSolver                   11.4585       11.4585       11.4585       11.4585          0            1     
produce:hitpdune:DisambigFromSpacePoints         0.155543      0.155543      0.155543      0.155543          0            1     
produce:pandora:StandardPandora                   28.7146       28.7146       28.7146       28.7146          0            1     
produce:pandoraTrack:LArPandoraTrackCreation      1.01257       1.01257       1.01257       1.01257          0            1     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0391192     0.0391192     0.0391192     0.0391192         0            1     
[art]:TriggerResults:TriggerResultInserter      0.000104266   0.000104266   0.000104266   0.000104266        0            1     
end_path:out1:RootOutput                        3.2702e-05    3.2702e-05    3.2702e-05    3.2702e-05         0            1     
end_path:out1:RootOutput(write)                   4.78302       4.78302       4.78302       4.78302          0            1     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.91 MB
  Peak resident set size usage (VmHWM): 6712.05 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 02-Nov-2025 12:08:36 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 122941
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-04 07:30:37 UTC       justIN version: 01.05.01