justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 235696.2@dunegpschedd02.fnal.gov

Jobsub ID235696.2@dunegpschedd02.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 08:50:46
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-10-31 09:38:18
From worker nodeHostnamewn-sate-038.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 08:56:02
Input filesvd-protodune:np02vd_raw_run040266_0350_df-s03-d0_dw_0_20251025T065918.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 09:38:18
Saved logsjustin-logs:235696.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ctor:chsel7> input frame: ident=64664 time=58 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=64664 time=58 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[10:34:55.799] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 input frame: frame: ident=64664 time=58 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[10:34:55.799] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 init nticks=9432 tbinmin=0 tbinmax=9432 
[10:34:55.833] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 0, ntraces=1536, input bad regions: 0 
[10:34:59.087] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 1, ntraces=1536, input bad regions: 0 
[10:35:02.626] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 2, ntraces=1536, input bad regions: 0 
[10:35:47.079] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 0, Qtot=7687049354 Qloss=-95095517, 10623 indices spanning [35974,46596] "wiener" 
[10:35:47.454] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 0, Qtot=7538426578 Qloss=-85662006, 8485 indices spanning [46597,55081] "gauss" 
[10:35:48.526] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 1, Qtot=1720006802 Qloss=-106271951, 13851 indices spanning [55082,68932] "wiener" 
[10:35:48.915] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 1, Qtot=1476236206 Qloss=-100066431, 11307 indices spanning [68933,80239] "gauss" 
[10:35:49.531] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 2, Qtot=1826823410 Qloss=-42180004, 12362 indices spanning [80240,92601] "wiener" 
[10:35:50.137] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 2, Qtot=1804806797 Qloss=-19219733, 9337 indices spanning [92602,101938] "gauss" 
[10:35:50.138] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 produce 101939 traces: 36836 wiener7, 0 decon_charge7, 29129 gauss7, frame tag: sigproc 
[10:35:50.138] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 output frame: frame: ident=64664 time=58 tick=512 with 101939 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":29129 [0] "mp2_roi7":24697 [0] "mp3_roi7":11277 [0] "wiener7":36836 [36836] ] cmm:[ ] 
[10:35:55.938] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[10:35:55.938] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=64664 time=58 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=64664 time=58 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[10:35:55.939] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 input frame: frame: ident=64664 time=58 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[10:35:55.939] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 init nticks=9432 tbinmin=0 tbinmax=9432 
[10:35:55.983] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 0, ntraces=1536, input bad regions: 0 
[10:35:59.477] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 1, ntraces=1536, input bad regions: 0 
[10:36:02.814] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 2, ntraces=1536, input bad regions: 0 
[10:36:30.953] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 0, Qtot=5600869486 Qloss=-81446037, 9421 indices spanning [38626,48046] "wiener" 
[10:36:31.347] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 0, Qtot=5514248651 Qloss=-71069363, 7573 indices spanning [48047,55619] "gauss" 
[10:36:32.345] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 1, Qtot=1352825880 Qloss=-119592638, 12177 indices spanning [55620,67796] "wiener" 
[10:36:32.691] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 1, Qtot=1144774170 Qloss=-117681709, 9988 indices spanning [67797,77784] "gauss" 
[10:36:33.329] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 2, Qtot=1520183861 Qloss=-26304807, 9685 indices spanning [77785,87469] "wiener" 
[10:36:34.011] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 2, Qtot=1504535773 Qloss=-10598539, 7517 indices spanning [87470,94986] "gauss" 
[10:36:34.011] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 produce 94987 traces: 31283 wiener6, 0 decon_charge6, 25078 gauss6, frame tag: sigproc 
[10:36:34.011] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 output frame: frame: ident=64664 time=58 tick=512 with 94987 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":25078 [0] "mp2_roi6":28053 [0] "mp3_roi6":10573 [0] "wiener6":31283 [31283] ] cmm:[ ] 
[10:36:37.415] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[10:36:37.416] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=64664 time=58 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=64664 time=58 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[10:36:37.416] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 input frame: frame: ident=64664 time=58 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[10:36:37.416] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 init nticks=9432 tbinmin=0 tbinmax=9432 
[10:36:37.453] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 load plane index: 0, ntraces=1536, input bad regions: 0 
[10:36:40.566] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 load plane index: 1, ntraces=1536, input bad regions: 0 
[10:36:43.826] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.8289e-05      174.559       527.951       149.169       115.063        13     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.0133e-05    7.79391e-05   0.000193374    6.937e-05    3.37145e-05      13     
produce:tpcrawdecoder:PDVDTPCReader               12.8305       19.3631       25.4117       19.1445       3.79893        13     
produce:triggerrawdecoder:PDVDTriggerReader4     0.515067      0.535859      0.583482      0.523359      0.0220588       13     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000356721   0.000409316   0.000684419   0.000383302   8.15746e-05      13     
produce:ophit:OpHitFinder                       6.0904e-05    0.000100232   0.000471588   6.8468e-05    0.000107574      13     
produce:opflash:OpFlashFinderVerticalDrift      5.1757e-05    7.77634e-05   0.000294645   5.7238e-05    6.3196e-05       13     
produce:wclsdatavd:WireCellToolkit                58.5038       88.5951       153.973       69.7865       29.0359        12     
produce:gaushit:GausHitFinder                    0.934584       1.58816       3.30119       1.38888      0.730814        12     
produce:nhitsfilter:NumberOfHitsFilter          0.000262253   0.000370905   0.000595682   0.000339951   9.77153e-05      12     
produce:reco3d:SpacePointSolver                   8.43957       14.713        34.6852       13.5728       6.65252        12     
produce:hitpdune:DisambigFromSpacePoints         0.118095      0.238688      0.669525      0.216161      0.136002        12     
produce:pandora:StandardPandora                   13.8639       57.0773       307.028       31.694        76.7833        12     
produce:pandoraTrack:LArPandoraTrackCreation     0.720231       1.71302       4.88652       1.32584       1.08304        12     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0272512     0.0397551     0.0538503     0.0373959    0.00814781       12     
[art]:TriggerResults:TriggerResultInserter       2.102e-05    3.98218e-05   9.3607e-05    3.25715e-05   2.05617e-05      12     
end_path:out1:RootOutput                         6.242e-06    9.68733e-06   2.5347e-05     7.714e-06    4.98754e-06      12     
end_path:out1:RootOutput(write)                   4.67316       5.16321       6.22747       4.98116      0.516472        12     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.84 MB
  Peak resident set size usage (VmHWM): 6701.39 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 10:37:44 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 64664
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 08:42:17 UTC       justIN version: 01.05.01