justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 235684.2@dunegpschedd02.fnal.gov

Jobsub ID235684.2@dunegpschedd02.fnal.gov
Workflow ID9373
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 07:40:42
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-10-31 08:33:26
From worker nodeHostnamewn-pep-012.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 07:41:30
Input filesvd-protodune:np02vd_raw_run040266_0017_df-s04-d0_dw_0_20251024T214043.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 08:33:26
Saved logsjustin-logs:235684.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.142] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=3284 time=81 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3284 time=81 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:31:08.142] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 input frame: frame: ident=3284 time=81 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:31:08.143] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 init nticks=8512 tbinmin=0 tbinmax=8512 
[09:31:08.180] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[09:31:10.611] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[09:31:13.118] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 
[09:31:27.750] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 save plane index: 0, Qtot=1593863872 Qloss=-42814221, 9117 indices spanning [42739,51855] "wiener" 
[09:31:28.029] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 save plane index: 0, Qtot=1563772874 Qloss=-32148732, 7217 indices spanning [51856,59072] "gauss" 
[09:31:28.697] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 save plane index: 1, Qtot=374937224 Qloss=-41743621, 8740 indices spanning [59073,67812] "wiener" 
[09:31:28.954] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 save plane index: 1, Qtot=337736601 Qloss=-32472703, 6552 indices spanning [67813,74364] "gauss" 
[09:31:29.446] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 save plane index: 2, Qtot=363181914 Qloss=-31719851, 12294 indices spanning [74365,86658] "wiener" 
[09:31:29.893] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 save plane index: 2, Qtot=346251419 Qloss=-14295614, 9647 indices spanning [86659,96305] "gauss" 
[09:31:29.893] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 produce 96306 traces: 30151 wiener7, 0 decon_charge7, 23416 gauss7, frame tag: sigproc 
[09:31:29.894] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=34 output frame: frame: ident=3284 time=81 tick=512 with 96306 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":23416 [0] "mp2_roi7":34403 [0] "mp3_roi7":8336 [0] "wiener7":30151 [30151] ] cmm:[ ] 
[09:31:31.500] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[09:31:31.501] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=3284 time=81 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3284 time=81 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:31:31.501] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 input frame: frame: ident=3284 time=81 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:31:31.501] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 init nticks=8512 tbinmin=0 tbinmax=8512 
[09:31:31.537] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[09:31:34.005] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[09:31:36.402] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 
[09:31:54.200] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 0, Qtot=2438661084 Qloss=-69266950, 8464 indices spanning [42283,50746] "wiener" 
[09:31:54.462] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 0, Qtot=2405301381 Qloss=-59057776, 6696 indices spanning [50747,57442] "gauss" 
[09:31:55.207] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 1, Qtot=564289224 Qloss=-46564564, 9650 indices spanning [57443,67092] "wiener" 
[09:31:55.470] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 1, Qtot=480481055 Qloss=-38188126, 7648 indices spanning [67093,74740] "gauss" 
[09:31:55.974] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 2, Qtot=557266984 Qloss=-34787034, 13533 indices spanning [74741,88273] "wiener" 
[09:31:56.423] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 2, Qtot=538788019 Qloss=-15764383, 10504 indices spanning [88274,98777] "gauss" 
[09:31:56.424] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 produce 98778 traces: 31647 wiener6, 0 decon_charge6, 24848 gauss6, frame tag: sigproc 
[09:31:56.424] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 output frame: frame: ident=3284 time=81 tick=512 with 98778 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":24848 [0] "mp2_roi6":33616 [0] "mp3_roi6":8667 [0] "wiener6":31647 [31647] ] cmm:[ ] 
[09:31:58.816] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[09:31:58.816] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=3284 time=81 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3284 time=81 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:31:58.817] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 input frame: frame: ident=3284 time=81 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:31:58.817] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 init nticks=8512 tbinmin=0 tbinmax=8512 
[09:31:58.857] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[09:32:01.323] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[09:32:03.774] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.1383e-05      162.042       348.713       162.386       75.5986        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.2336e-05    7.76313e-05   0.00017942    6.86575e-05   2.84902e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               11.7997       17.9619       22.8245       17.4196       3.33569        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.526066      0.564543      0.597189       0.56238      0.0232094       18     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000368065   0.000410023   0.000702802   0.000392473   7.22809e-05      18     
produce:ophit:OpHitFinder                       5.7874e-05    9.34585e-05   0.000503231   6.2723e-05    0.000100417      18     
produce:opflash:OpFlashFinderVerticalDrift      4.4237e-05    6.37034e-05   0.000293611   4.94875e-05   5.5954e-05       18     
produce:wclsdatavd:WireCellToolkit                58.2432       88.7445       128.665       86.4582       23.7718        17     
produce:gaushit:GausHitFinder                    0.658791       1.30618       1.73879       1.38004      0.301298        17     
produce:nhitsfilter:NumberOfHitsFilter          0.000255654   0.000412882   0.000653331   0.000384813   9.03231e-05      17     
produce:reco3d:SpacePointSolver                   5.87647       11.4086       17.6938       11.2856       3.21386        17     
produce:hitpdune:DisambigFromSpacePoints         0.0958995     0.232193      0.462308      0.215738      0.099512        17     
produce:pandora:StandardPandora                   11.4645       45.1323       174.741       29.0674       43.9843        17     
produce:pandoraTrack:LArPandoraTrackCreation     0.467409       2.19601       5.29081       1.75594       1.3594         17     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0178544     0.0350181     0.0500249     0.0354241    0.00814544       17     
[art]:TriggerResults:TriggerResultInserter      2.7117e-05    3.8349e-05    8.9999e-05    3.4215e-05    1.3873e-05       17     
end_path:out1:RootOutput                         4.425e-06    9.20935e-06   2.8607e-05     8.342e-06    5.14044e-06      17     
end_path:out1:RootOutput(write)                   3.68346       4.21651       4.64086       4.27439      0.283268        17     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.89 MB
  Peak resident set size usage (VmHWM): 6707.58 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 09:33:02 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 3284
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:25:57 UTC       justIN version: 01.05.01