justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 47119.0@dunegpschedd01.fnal.gov

Jobsub ID47119.0@dunegpschedd01.fnal.gov
Workflow ID2656
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 18:00:10
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_brug
Last heartbeat2025-09-16 19:35:39
From worker nodeHostnamewn-snel-030.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 18:01:33
Input filesvd-protodune:np02vd_raw_run039275_0471_df-s03-d0_dw_0_20250902T062014.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-16 19:35:39
Saved logsjustin-logs:47119.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tor:chsel6> input frame: ident=169906 time=143 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=169906 time=143 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[21:34:01.463] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 input frame: frame: ident=169906 time=143 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[21:34:01.463] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 init nticks=10000 tbinmin=0 tbinmax=10000 
[21:34:01.496] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[21:34:03.469] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[21:34:05.482] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 
[21:34:09.910] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 0, Qtot=158171228 Qloss=-20582702, 4367 indices spanning [17988,22354] "wiener" 
[21:34:10.112] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 0, Qtot=149391548 Qloss=-15536810, 3523 indices spanning [22355,25877] "gauss" 
[21:34:10.546] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 1, Qtot=141784869 Qloss=-21916595, 4600 indices spanning [25878,30477] "wiener" 
[21:34:10.755] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 1, Qtot=129581651 Qloss=-16535415, 3516 indices spanning [30478,33993] "gauss" 
[21:34:11.143] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 2, Qtot=131394340 Qloss=-11570365, 4805 indices spanning [33994,38798] "wiener" 
[21:34:11.544] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 2, Qtot=125951015 Qloss=-5743162, 4049 indices spanning [38799,42847] "gauss" 
[21:34:11.545] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 produce 42848 traces: 13772 wiener6, 0 decon_charge6, 11088 gauss6, frame tag: sigproc 
[21:34:11.545] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 output frame: frame: ident=169906 time=143 tick=512 with 42848 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":11088 [0] "mp2_roi6":14059 [0] "mp3_roi6":3929 [0] "wiener6":13772 [13772] ] cmm:[ ] 
[21:34:11.679] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[21:34:11.680] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=169906 time=143 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=169906 time=143 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[21:34:11.680] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 input frame: frame: ident=169906 time=143 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[21:34:11.680] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 init nticks=10000 tbinmin=0 tbinmax=10000 
[21:34:11.712] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[21:34:13.621] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[21:34:15.718] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 
[21:34:21.010] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 save plane index: 0, Qtot=267191245 Qloss=-21633874, 5645 indices spanning [26394,32038] "wiener" 
[21:34:21.223] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 save plane index: 0, Qtot=250232912 Qloss=-16050599, 4487 indices spanning [32039,36525] "gauss" 
[21:34:21.920] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 save plane index: 1, Qtot=170847581 Qloss=-22205095, 5927 indices spanning [36526,42452] "wiener" 
[21:34:22.125] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 save plane index: 1, Qtot=154369524 Qloss=-14328353, 4447 indices spanning [42453,46899] "gauss" 
[21:34:22.552] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 save plane index: 2, Qtot=142099371 Qloss=-15818897, 4933 indices spanning [46900,51832] "wiener" 
[21:34:23.017] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 save plane index: 2, Qtot=134060563 Qloss=-7102106, 4042 indices spanning [51833,55874] "gauss" 
[21:34:23.017] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 produce 55875 traces: 16505 wiener5, 0 decon_charge5, 12976 gauss5, frame tag: sigproc 
[21:34:23.018] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 output frame: frame: ident=169906 time=143 tick=512 with 55875 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":12976 [0] "mp2_roi5":21128 [0] "mp3_roi5":5266 [0] "wiener5":16505 [16505] ] cmm:[ ] 
[21:34:23.311] W [  glue  ] <ChannelSelector:chsel4> Untagged summary not supported, summary will be dropped. 
[21:34:23.312] D [  glue  ] <ChannelSelector:chsel4> input frame: ident=169906 time=143 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=169906 time=143 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[21:34:23.312] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=34 input frame: frame: ident=169906 time=143 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[21:34:23.312] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=34 init nticks=10000 tbinmin=0 tbinmax=10000 
[21:34:23.351] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[21:34:25.253] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[21:34:27.310] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      7.5131e-05      299.111       407.043       303.468       94.8795        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.8038e-05    9.15568e-05   0.000252664   7.7365e-05    4.12031e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               80.1719       105.033       159.957       97.4515       19.0474        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0349624     0.0473558     0.213885      0.0353373     0.0408361       18     
produce:pdvddaphne:DAPHNEReaderPDVD               10.8393       15.0201       21.2535       15.1119       2.29095        18     
produce:ophit:OpHitFinder                        0.0557942     0.0886429     0.105065      0.0885793     0.0126773       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00895575     0.0210259     0.0298645     0.0224808    0.00584269       18     
produce:wclsdatavd:WireCellToolkit                64.3434       78.7421       94.2428       80.5796       8.34153        17     
produce:gaushit:GausHitFinder                    0.901931       1.5004        2.21701       1.48897      0.382406        17     
produce:nhitsfilter:NumberOfHitsFilter          0.000354866   0.000767006   0.00126897    0.000723147   0.000251778      17     
produce:reco3d:SpacePointSolver                   8.69789       17.438        26.6347       17.2974       6.09407        17     
produce:hitpdune:DisambigFromSpacePoints         0.110571      0.273306      0.479721      0.244972      0.102175        17     
produce:pandora:StandardPandora                   34.6493       90.2473       162.525       76.4306       48.2792        17     
produce:pandoraTrack:LArPandoraTrackCreation      0.85174       2.6604        5.96328       2.40676       1.37511        17     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0230851     0.0400923     0.056147      0.0392867    0.00830361       17     
[art]:TriggerResults:TriggerResultInserter      1.7152e-05    4.24544e-05   6.4752e-05     4.289e-05    9.94564e-06      17     
end_path:out1:RootOutput                         3.858e-06    7.69818e-06   2.0339e-05     5.42e-06     4.70035e-06      17     
end_path:out1:RootOutput(write)                   4.67155       4.95981       5.38941       4.9441       0.226598        17     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.8 MB
  Peak resident set size usage (VmHWM): 6713.02 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 21:35:18 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39275 subRun: 1 event: 169906
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-18 19:28:33 UTC       justIN version: 01.05.00