justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40441.1@dunegpschedd02.fnal.gov

Jobsub ID40441.1@dunegpschedd02.fnal.gov
Workflow ID2323
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 16:13:05
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-09-16 16:39:30
From worker nodeHostnamewn-snel-025.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 16:14:17
Input filesvd-protodune:np02vd_raw_run039324_1630_df-s05-d4_dw_0_20250907T012250.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-16 16:39:30
Saved logsjustin-logs:40441.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

input frame: ident=855158 time=29 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[18:37:27.475] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 input frame: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[18:37:27.475] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 init nticks=6400 tbinmin=0 tbinmax=6400 
[18:37:27.496] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 load plane index: 0, ntraces=1536, input bad regions: 0 
[18:37:28.480] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 load plane index: 1, ntraces=1536, input bad regions: 0 
[18:37:29.593] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 load plane index: 2, ntraces=1536, input bad regions: 0 
[18:37:51.848] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 0, Qtot=1645939690 Qloss=-67944562, 12244 indices spanning [40500,52743] "wiener" 
[18:37:51.974] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 0, Qtot=1608146478 Qloss=-53945582, 9381 indices spanning [52744,62124] "gauss" 
[18:37:52.602] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 1, Qtot=665995087 Qloss=-70309597, 14326 indices spanning [62125,76450] "wiener" 
[18:37:52.728] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 1, Qtot=595164528 Qloss=-52546399, 10661 indices spanning [76451,87111] "gauss" 
[18:37:52.979] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 2, Qtot=682854623 Qloss=-39444026, 16732 indices spanning [87112,103843] "wiener" 
[18:37:53.220] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 save plane index: 2, Qtot=661494271 Qloss=-16999377, 13201 indices spanning [103844,117044] "gauss" 
[18:37:53.220] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 produce 117045 traces: 43302 wiener6, 0 decon_charge6, 33243 gauss6, frame tag: sigproc 
[18:37:53.220] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=22 output frame: frame: ident=855158 time=29 tick=512 with 117045 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":33243 [0] "mp2_roi6":27913 [0] "mp3_roi6":12587 [0] "wiener6":43302 [43302] ] cmm:[ ] 
[18:37:56.160] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[18:37:56.160] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=855158 time=29 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[18:37:56.160] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 input frame: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[18:37:56.160] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 init nticks=6400 tbinmin=0 tbinmax=6400 
[18:37:56.182] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 load plane index: 0, ntraces=1536, input bad regions: 0 
[18:37:57.324] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 load plane index: 1, ntraces=1536, input bad regions: 0 
[18:37:58.545] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 load plane index: 2, ntraces=1536, input bad regions: 0 
[18:38:20.527] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 0, Qtot=2237240361 Qloss=-76210082, 11415 indices spanning [41557,52971] "wiener" 
[18:38:20.634] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 0, Qtot=2198384733 Qloss=-63704229, 8581 indices spanning [52972,61552] "gauss" 
[18:38:21.259] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 1, Qtot=599925440 Qloss=-62985466, 13185 indices spanning [61553,74737] "wiener" 
[18:38:21.389] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 1, Qtot=540908454 Qloss=-45413317, 9717 indices spanning [74738,84454] "gauss" 
[18:38:21.632] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 2, Qtot=606800966 Qloss=-41414355, 15841 indices spanning [84455,100295] "wiener" 
[18:38:21.867] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 save plane index: 2, Qtot=584427007 Qloss=-17863512, 12843 indices spanning [100296,113138] "gauss" 
[18:38:21.867] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 produce 113139 traces: 40441 wiener5, 0 decon_charge5, 31141 gauss5, frame tag: sigproc 
[18:38:21.867] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=22 output frame: frame: ident=855158 time=29 tick=512 with 113139 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":31141 [0] "mp2_roi5":29588 [0] "mp3_roi5":11969 [0] "wiener5":40441 [40441] ] cmm:[ ] 
[18:38:24.631] W [  glue  ] <ChannelSelector:chsel4> Untagged summary not supported, summary will be dropped. 
[18:38:24.632] D [  glue  ] <ChannelSelector:chsel4> input frame: ident=855158 time=29 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[18:38:24.632] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 input frame: frame: ident=855158 time=29 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[18:38:24.632] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 init nticks=6400 tbinmin=0 tbinmax=6400 
[18:38:24.653] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 load plane index: 0, ntraces=1536, input bad regions: 0 
[18:38:25.773] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 load plane index: 1, ntraces=1536, input bad regions: 0 
[18:38:27.048] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=22 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.8358e-05      109.07        152.964       112.427       37.7582        12     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.5835e-05    7.87804e-05   0.000180889   7.1519e-05    3.1263e-05       12     
produce:tpcrawdecoder:PDVDTPCReader               12.6721       15.5316       33.9028       13.1686       5.74529        12     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0346181     0.034921      0.0359494     0.0348212    0.000346216      12     
produce:pdvddaphne:DAPHNEReaderPDVD               3.22078       3.57145       4.1338        3.55194      0.237822        12     
produce:ophit:OpHitFinder                        0.0277259     0.0378275     0.0543647     0.0378245    0.00667912       12     
produce:opflash:OpFlashFinderVerticalDrift       0.0073835     0.0107062     0.0198114    0.00905843     0.0037213       12     
produce:wclsdatavd:WireCellToolkit                33.3672       45.8109       59.2301       44.4732       7.10404        11     
produce:gaushit:GausHitFinder                    0.673603      0.954122       1.27578      0.871534      0.202597        11     
produce:nhitsfilter:NumberOfHitsFilter          0.000193604   0.000307641   0.000544783   0.000281528   9.91649e-05      11     
produce:reco3d:SpacePointSolver                   6.45616       10.1938       15.2286       9.39156       2.75542        11     
produce:hitpdune:DisambigFromSpacePoints         0.0771289     0.148916      0.259973       0.13624      0.0521829       11     
produce:pandora:StandardPandora                   23.0177       38.3649       68.1345       32.3259       13.849         11     
produce:pandoraTrack:LArPandoraTrackCreation     0.288582      0.553896       1.19125       0.44809      0.263844        11     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0153498     0.0234797     0.0320222     0.0239453    0.00488385       11     
[art]:TriggerResults:TriggerResultInserter      1.3345e-05    1.82559e-05   4.1237e-05     1.621e-05    7.3751e-06       11     
end_path:out1:RootOutput                         3.346e-06    4.84182e-06   1.5749e-05     3.918e-06    3.46381e-06      11     
end_path:out1:RootOutput(write)                   3.01864       3.46299       4.66941       3.23934      0.538693        11     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.81 MB
  Peak resident set size usage (VmHWM): 6711.81 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 18:39:12 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39324 subRun: 1 event: 855158
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-19 01:41:24 UTC       justIN version: 01.05.00