justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241271.0@dunegpschedd01.fnal.gov

Jobsub ID241271.0@dunegpschedd01.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 01:20:22
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-10-31 02:30:10
From worker nodeHostnamewn-sate-052.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 01:22:11
Input filesvd-protodune:np02vd_raw_run040266_0525_df-s04-d1_dw_0_20251025T115335.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 02:30:10
Saved logsjustin-logs:241271.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

lue  ] <ChannelSelector:chsel7> input frame: ident=96861 time=73 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=96861 time=73 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:27:25.021] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 input frame: frame: ident=96861 time=73 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:27:25.021] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 init nticks=8000 tbinmin=0 tbinmax=8000 
[03:27:25.056] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[03:27:27.147] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[03:27:29.232] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 
[03:27:50.354] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 save plane index: 0, Qtot=32647492960 Qloss=-2520632660, 3820 indices spanning [15237,19056] "wiener" 
[03:27:50.566] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 save plane index: 0, Qtot=32061389714 Qloss=-2551681212, 3335 indices spanning [19057,22391] "gauss" 
[03:27:51.181] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 save plane index: 1, Qtot=33019394778 Qloss=-248185066, 4660 indices spanning [22392,27051] "wiener" 
[03:27:51.396] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 save plane index: 1, Qtot=31783917338 Qloss=-237537149, 4052 indices spanning [27052,31103] "gauss" 
[03:27:51.757] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 save plane index: 2, Qtot=1668766019 Qloss=-2333852650, 10809 indices spanning [31104,41912] "wiener" 
[03:27:52.126] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 save plane index: 2, Qtot=1244790250 Qloss=-1921387377, 15093 indices spanning [41913,57005] "gauss" 
[03:27:52.126] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 produce 57006 traces: 19289 wiener7, 0 decon_charge7, 22480 gauss7, frame tag: sigproc 
[03:27:52.126] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 output frame: frame: ident=96861 time=73 tick=512 with 57006 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":22480 [0] "mp2_roi7":12289 [0] "mp3_roi7":2948 [0] "wiener7":19289 [19289] ] cmm:[ ] 
[03:27:54.814] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[03:27:54.814] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=96861 time=73 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=96861 time=73 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:27:54.815] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 input frame: frame: ident=96861 time=73 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:27:54.815] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 init nticks=8000 tbinmin=0 tbinmax=8000 
[03:27:54.847] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[03:27:56.910] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[03:27:58.982] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 
[03:28:02.135] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 0, Qtot=101718585 Qloss=-16031179, 2279 indices spanning [9942,12220] "wiener" 
[03:28:02.347] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 0, Qtot=98300569 Qloss=-13614511, 1872 indices spanning [12221,14092] "gauss" 
[03:28:02.740] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 1, Qtot=92917737 Qloss=-16697063, 2535 indices spanning [14093,16627] "wiener" 
[03:28:02.958] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 1, Qtot=87285710 Qloss=-14548774, 2068 indices spanning [16628,18695] "gauss" 
[03:28:03.357] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 2, Qtot=85187755 Qloss=-5929960, 3347 indices spanning [18696,22042] "wiener" 
[03:28:03.755] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 2, Qtot=82589856 Qloss=-3285529, 2809 indices spanning [22043,24851] "gauss" 
[03:28:03.755] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 produce 24852 traces: 8161 wiener6, 0 decon_charge6, 6749 gauss6, frame tag: sigproc 
[03:28:03.755] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 output frame: frame: ident=96861 time=73 tick=512 with 24852 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":6749 [0] "mp2_roi6":8101 [0] "mp3_roi6":1841 [0] "wiener6":8161 [8161] ] cmm:[ ] 
[03:28:03.779] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[03:28:03.779] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=96861 time=73 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=96861 time=73 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:28:03.780] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 input frame: frame: ident=96861 time=73 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:28:03.780] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 init nticks=8000 tbinmin=0 tbinmax=8000 
[03:28:03.811] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[03:28:05.813] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[03:28:08.078] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.7758e-05      209.672       409.804       195.82        88.089         18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.0444e-05    7.87498e-05   0.000220895   6.7788e-05    3.55349e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               34.8547       65.464        105.619       61.3624       21.9792        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.541888      0.661701      0.911293      0.608625      0.115606        18     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000378001   0.00042118    0.000811656   0.000394963   9.76837e-05      18     
produce:ophit:OpHitFinder                       6.3198e-05    0.000107107   0.000684257   6.9651e-05    0.000140302      18     
produce:opflash:OpFlashFinderVerticalDrift       4.785e-05    7.54741e-05   0.000427353   5.49485e-05   8.53852e-05      18     
produce:wclsdatavd:WireCellToolkit                57.2057       91.2845       180.131       84.8981       30.1194        17     
produce:gaushit:GausHitFinder                     0.74668       1.69556       7.11946       1.41861       1.38998        17     
produce:nhitsfilter:NumberOfHitsFilter          0.000258736   0.000357149   0.000541388   0.000344177   7.10235e-05      17     
produce:reco3d:SpacePointSolver                   6.70608       13.5082       31.5321       11.4695       5.63686        17     
produce:hitpdune:DisambigFromSpacePoints         0.0886448     0.212221      0.442646       0.19656      0.0980352       17     
produce:pandora:StandardPandora                   10.5282       44.1137       146.953       27.5903       36.6228        17     
produce:pandoraTrack:LArPandoraTrackCreation     0.413539       1.38863       3.58822       1.27496      0.721183        17     
produce:pandoraGnocalo:GnocchiCalorimetry        0.024083      0.0365942     0.0505267     0.0356453    0.00779237       17     
[art]:TriggerResults:TriggerResultInserter      1.8825e-05    3.95904e-05   0.000113082   3.0408e-05    2.1985e-05       17     
end_path:out1:RootOutput                         6.562e-06    9.77712e-06   3.4224e-05     8.346e-06    6.20968e-06      17     
end_path:out1:RootOutput(write)                    4.477        5.15268       6.04375       5.05158       0.44663        17     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.88 MB
  Peak resident set size usage (VmHWM): 6708.4 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 03:29:41 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 96861
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:26:22 UTC       justIN version: 01.05.01