justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242090.0@dunegpschedd01.fnal.gov

Jobsub ID242090.0@dunegpschedd01.fnal.gov
Workflow ID9373
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-02 20:49:59
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-11-02 21:40:43
From worker nodeHostnamewn-sate-043.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-02 20:51:00
Input filesvd-protodune:np02vd_raw_run040266_0214_df-s04-d3_dw_0_20251025T030803.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-02 21:40:43
Saved logsjustin-logs:242090.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

el6> input frame: ident=39551 time=71 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=39551 time=71 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:37:55.049] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 input frame: frame: ident=39551 time=71 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:37:55.049] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 init nticks=8640 tbinmin=0 tbinmax=8640 
[22:37:55.079] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:37:56.956] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:37:58.863] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 load plane index: 2, ntraces=1536, input bad regions: 0 
[22:38:17.227] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 0, Qtot=1073946749 Qloss=-198106991, 13227 indices spanning [33836,47062] "wiener" 
[22:38:17.443] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 0, Qtot=909394392 Qloss=-185681435, 10927 indices spanning [47063,57989] "gauss" 
[22:38:18.347] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 1, Qtot=1388025033 Qloss=-90851214, 13599 indices spanning [57990,71588] "wiener" 
[22:38:18.557] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 1, Qtot=1168620212 Qloss=-80093625, 11057 indices spanning [71589,82645] "gauss" 
[22:38:18.925] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 2, Qtot=1381105114 Qloss=-39877680, 8622 indices spanning [82646,91267] "wiener" 
[22:38:19.282] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 2, Qtot=1356329130 Qloss=-19785513, 6644 indices spanning [91268,97911] "gauss" 
[22:38:19.282] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 produce 97912 traces: 35448 wiener6, 0 decon_charge6, 28628 gauss6, frame tag: sigproc 
[22:38:19.282] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 output frame: frame: ident=39551 time=71 tick=512 with 97912 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":28628 [0] "mp2_roi6":22977 [0] "mp3_roi6":10859 [0] "wiener6":35448 [35448] ] cmm:[ ] 
[22:38:21.159] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[22:38:21.160] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=39551 time=71 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=39551 time=71 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:38:21.160] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 input frame: frame: ident=39551 time=71 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:38:21.160] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 init nticks=8640 tbinmin=0 tbinmax=8640 
[22:38:21.191] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:38:23.086] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:38:25.049] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 load plane index: 2, ntraces=1536, input bad regions: 0 
[22:39:06.371] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 save plane index: 0, Qtot=2928205854 Qloss=-149030585, 14989 indices spanning [40938,55926] "wiener" 
[22:39:06.598] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 save plane index: 0, Qtot=2812131274 Qloss=-133440132, 11777 indices spanning [55927,67703] "gauss" 
[22:39:07.784] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 save plane index: 1, Qtot=1649159810 Qloss=-95931145, 18198 indices spanning [67704,85901] "wiener" 
[22:39:08.025] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 save plane index: 1, Qtot=1431101394 Qloss=-72534951, 14475 indices spanning [85902,100376] "gauss" 
[22:39:08.440] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 save plane index: 2, Qtot=1209198340 Qloss=-47358506, 13831 indices spanning [100377,114207] "wiener" 
[22:39:08.829] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 save plane index: 2, Qtot=1179463182 Qloss=-20464678, 10476 indices spanning [114208,124683] "gauss" 
[22:39:08.829] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 produce 124684 traces: 47018 wiener5, 0 decon_charge5, 36728 gauss5, frame tag: sigproc 
[22:39:08.829] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 output frame: frame: ident=39551 time=71 tick=512 with 124684 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":36728 [0] "mp2_roi5":21290 [0] "mp3_roi5":19648 [0] "wiener5":47018 [47018] ] cmm:[ ] 
[22:39:14.009] W [  glue  ] <ChannelSelector:chsel4> Untagged summary not supported, summary will be dropped. 
[22:39:14.010] D [  glue  ] <ChannelSelector:chsel4> input frame: ident=39551 time=71 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=39551 time=71 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:39:14.010] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=30 input frame: frame: ident=39551 time=71 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:39:14.010] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=30 init nticks=8640 tbinmin=0 tbinmax=8640 
[22:39:14.043] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=30 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:39:16.025] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=30 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:39:18.120] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=30 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.6625e-05      169.119       220.066       171.294       49.4018        16     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.1395e-05    7.8034e-05    0.00021293    6.77125e-05   3.58179e-05      16     
produce:tpcrawdecoder:PDVDTPCReader               33.9046       59.378        73.3535       64.697        11.6292        16     
produce:triggerrawdecoder:PDVDTriggerReader4     0.534558      0.709646       1.24582      0.684899      0.175809        16     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000368242   0.000410315   0.000787679   0.000383946   9.87057e-05      16     
produce:ophit:OpHitFinder                       6.4811e-05    0.000101235   0.000557327   6.7873e-05    0.000117912      16     
produce:opflash:OpFlashFinderVerticalDrift      5.1046e-05    7.15106e-05   0.000324179   5.4673e-05    6.52829e-05      16     
produce:wclsdatavd:WireCellToolkit                54.6592       72.2419       91.7892       64.1597       13.3408        15     
produce:gaushit:GausHitFinder                    0.807708       1.22983       1.53342        1.271       0.195686        15     
produce:nhitsfilter:NumberOfHitsFilter          0.000222808   0.000306468   0.000567455   0.000281258   7.8398e-05       15     
produce:reco3d:SpacePointSolver                   8.07491       12.4971       17.4133       12.9303       2.43028        15     
produce:hitpdune:DisambigFromSpacePoints         0.107536      0.183418      0.270671      0.182582      0.0475226       15     
produce:pandora:StandardPandora                   15.7041       27.5325       40.2982       27.7203       6.76001        15     
produce:pandoraTrack:LArPandoraTrackCreation      0.57551       1.2483        1.9269        1.19842      0.386231        15     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0246817     0.0361141     0.0545906     0.0335004    0.00713038       15     
[art]:TriggerResults:TriggerResultInserter      2.2142e-05    3.61345e-05   8.9388e-05    2.9896e-05    1.66429e-05      15     
end_path:out1:RootOutput                         6.372e-06    9.10373e-06   2.9505e-05     7.073e-06    5.61124e-06      15     
end_path:out1:RootOutput(write)                   4.51816       4.96234       6.18947       4.91376      0.394553        15     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.89 MB
  Peak resident set size usage (VmHWM): 6704.32 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 02-Nov-2025 22:40:21 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 39551
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-04 01:29:22 UTC       justIN version: 01.05.01