justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 239244.146@dunegpschedd01.fnal.gov

Jobsub ID239244.146@dunegpschedd01.fnal.gov
Workflow ID9410
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-28 12:16:10
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-10-28 12:52:10
From worker nodeHostnamewn-lot-033.farm.nikhef.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-10-28 12:28:43
Input filesvd-protodune:np02vd_raw_run040267_0435_df-s04-d3_dw_0_20251026T064227.hdf5
Outputting started 
Output files
Finished2025-10-28 12:52:10
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

glue  ] <ChannelSelector:chsel7> input frame: ident=80527 time=27 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=80527 time=27 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:42:54.150] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 input frame: frame: ident=80527 time=27 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:42:54.150] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 init nticks=8576 tbinmin=0 tbinmax=8576 
[13:42:54.172] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 load plane index: 0, ntraces=1536, input bad regions: 0 
[13:42:56.597] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 load plane index: 1, ntraces=1536, input bad regions: 0 
[13:42:59.037] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 load plane index: 2, ntraces=1536, input bad regions: 0 
[13:43:12.051] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 0, Qtot=24363638110 Qloss=-37568643108, 3714 indices spanning [11591,15304] "wiener" 
[13:43:12.346] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 0, Qtot=23149790580 Qloss=-38575336571, 3209 indices spanning [15305,18513] "gauss" 
[13:43:12.839] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 1, Qtot=14483638352 Qloss=-7103744662, 3074 indices spanning [18514,21587] "wiener" 
[13:43:13.131] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 1, Qtot=13681337021 Qloss=-7676197656, 2496 indices spanning [21588,24083] "gauss" 
[13:43:13.621] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 2, Qtot=2212988294 Qloss=-1258949481, 7603 indices spanning [24084,31686] "wiener" 
[13:43:14.118] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 2, Qtot=1924065361 Qloss=-973788234, 7821 indices spanning [31687,39507] "gauss" 
[13:43:14.118] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 produce 39508 traces: 14391 wiener7, 0 decon_charge7, 13526 gauss7, frame tag: sigproc 
[13:43:14.118] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 output frame: frame: ident=80527 time=27 tick=512 with 39508 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":13526 [0] "mp2_roi7":9206 [0] "mp3_roi7":2385 [0] "wiener7":14391 [14391] ] cmm:[ ] 
[13:43:15.104] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[13:43:15.104] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=80527 time=27 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=80527 time=27 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:43:15.104] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 input frame: frame: ident=80527 time=27 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:43:15.104] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 init nticks=8576 tbinmin=0 tbinmax=8576 
[13:43:15.129] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 load plane index: 0, ntraces=1536, input bad regions: 0 
[13:43:17.672] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 load plane index: 1, ntraces=1536, input bad regions: 0 
[13:43:20.338] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 load plane index: 2, ntraces=1536, input bad regions: 0 
[13:43:23.774] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 save plane index: 0, Qtot=80090141 Qloss=-10688064, 2087 indices spanning [8616,10702] "wiener" 
[13:43:24.069] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 save plane index: 0, Qtot=76865535 Qloss=-8392034, 1680 indices spanning [10703,12382] "gauss" 
[13:43:24.500] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 save plane index: 1, Qtot=74674352 Qloss=-10008259, 2885 indices spanning [12383,15267] "wiener" 
[13:43:24.802] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 save plane index: 1, Qtot=69117253 Qloss=-6821030, 2349 indices spanning [15268,17616] "gauss" 
[13:43:25.307] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 save plane index: 2, Qtot=76034845 Qloss=-5157461, 2692 indices spanning [17617,20308] "wiener" 
[13:43:25.809] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 save plane index: 2, Qtot=73613106 Qloss=-2687604, 2236 indices spanning [20309,22544] "gauss" 
[13:43:25.809] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 produce 22545 traces: 7664 wiener6, 0 decon_charge6, 6265 gauss6, frame tag: sigproc 
[13:43:25.809] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 output frame: frame: ident=80527 time=27 tick=512 with 22545 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":6265 [0] "mp2_roi6":6667 [0] "mp3_roi6":1949 [0] "wiener6":7664 [7664] ] cmm:[ ] 
[13:43:25.821] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[13:43:25.821] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=80527 time=27 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=80527 time=27 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:43:25.821] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=12 input frame: frame: ident=80527 time=27 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:43:25.821] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=12 init nticks=8539 tbinmin=0 tbinmax=8539 
[13:43:25.846] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=12 load plane index: 0, ntraces=1536, input bad regions: 0 
[13:43:28.328] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=12 load plane index: 1, ntraces=1536, input bad regions: 0 
[13:43:30.890] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=12 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.9871e-05      109.994       158.289       118.457       48.234          7     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.355e-05    6.98063e-05   0.000106429   6.7306e-05    1.61026e-05       7     
produce:tpcrawdecoder:PDVDTPCReader               27.0516       39.485        54.2977       34.9108       9.08641         7     
produce:triggerrawdecoder:PDVDTriggerReader4      0.29939       0.38187       0.49087      0.358915      0.0659385        7     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000280556   0.000325059   0.000548368   0.000288891   9.1275e-05        7     
produce:ophit:OpHitFinder                       4.5576e-05     9.991e-05    0.00041618    4.7589e-05    0.000129121       7     
produce:opflash:OpFlashFinderVerticalDrift      3.4304e-05    6.50604e-05   0.000241232   3.5537e-05    7.1934e-05        7     
produce:wclsdatavd:WireCellToolkit                48.3425       57.8299       84.8997       54.0113       12.3023         6     
produce:gaushit:GausHitFinder                    0.624905      0.843449       1.22514       0.8113       0.208143         6     
produce:nhitsfilter:NumberOfHitsFilter          0.000138319   0.000235112   0.000377297   0.000207845   8.18076e-05       6     
produce:reco3d:SpacePointSolver                   4.94296       8.20419       13.9227        8.026        2.8083          6     
produce:hitpdune:DisambigFromSpacePoints         0.0652544     0.121491      0.193069      0.128887      0.0424905        6     
produce:pandora:StandardPandora                   10.3256       18.8475       28.5754       18.3206       6.47021         6     
produce:pandoraTrack:LArPandoraTrackCreation      0.26124      0.543316      0.813389      0.541646      0.194243         6     
produce:pandoraGnocalo:GnocchiCalorimetry         0.01451      0.023454      0.0355607     0.0230079    0.00724346        6     
[art]:TriggerResults:TriggerResultInserter      1.3825e-05    1.91958e-05   4.3902e-05    1.41565e-05   1.10576e-05       6     
end_path:out1:RootOutput                         3.206e-06    5.7125e-06    1.6892e-05     3.587e-06    5.00246e-06       6     
end_path:out1:RootOutput(write)                   3.57086       3.86201       4.08845       3.93515      0.202316         6     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.9 MB
  Peak resident set size usage (VmHWM): 6704.07 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 28-Oct-2025 13:44:25 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40267 subRun: 1 event: 80527
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:30:42 UTC       justIN version: 01.05.01