justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241287.0@dunegpschedd01.fnal.gov

Jobsub ID241287.0@dunegpschedd01.fnal.gov
Workflow ID9373
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 04:22:31
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-10-31 04:54:11
From worker nodeHostnamewn-sate-037.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 04:23:25
Input filesvd-protodune:np02vd_raw_run040266_0035_df-s04-d2_dw_0_20251024T221117.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 04:54:11
Saved logsjustin-logs:241287.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

=35 tick=500 with 1536 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:52:10.857] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 init nticks=8306 tbinmin=0 tbinmax=8306 
[05:52:10.888] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:52:14.037] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:52:17.087] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 load plane index: 2, ntraces=1536, input bad regions: 0 
[05:52:20.317] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 0, Qtot=37853797 Qloss=-5081593, 1559 indices spanning [7008,8566] "wiener" 
[05:52:20.635] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 0, Qtot=35431589 Qloss=-3241994, 1235 indices spanning [8567,9801] "gauss" 
[05:52:21.098] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 1, Qtot=44560004 Qloss=-6043089, 1736 indices spanning [9802,11537] "wiener" 
[05:52:21.438] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 1, Qtot=40852274 Qloss=-4042327, 1352 indices spanning [11538,12889] "gauss" 
[05:52:22.013] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 2, Qtot=33806285 Qloss=-1233981, 1108 indices spanning [12890,13997] "wiener" 
[05:52:22.581] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 2, Qtot=33079789 Qloss=-473960, 917 indices spanning [13998,14914] "gauss" 
[05:52:22.581] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 produce 14915 traces: 4403 wiener3, 0 decon_charge3, 3504 gauss3, frame tag: sigproc 
[05:52:22.582] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 output frame: frame: ident=6518 time=35 tick=500 with 14915 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":3504 [0] "mp2_roi3":5772 [0] "mp3_roi3":1236 [0] "wiener3":4403 [4403] ] cmm:[ ] 
[05:52:22.599] W [  glue  ] <ChannelSelector:chsel2> Untagged summary not supported, summary will be dropped. 
[05:52:22.599] D [  glue  ] <ChannelSelector:chsel2> input frame: ident=6518 time=35 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=6518 time=35 tick=512 with 1536 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:52:22.600] D [  aux   ] <Resampler:resmp2> first ch=3406 Ts=512 Ns=8112 Ns_pad=8125 Nrat=125 Tr=500 Nr=8320 Nout=8306 padding:linear 
[05:52:22.885] D [  aux   ] <Resampler:resmp2> resample 1536 traces at call=16 
[05:52:22.885] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 input frame: frame: ident=6518 time=35 tick=500 with 1536 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:52:22.885] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 init nticks=8306 tbinmin=0 tbinmax=8306 
[05:52:22.917] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:52:25.825] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:52:28.727] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 load plane index: 2, ntraces=1536, input bad regions: 0 
[05:52:32.374] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 0, Qtot=50823215 Qloss=-7170427, 2035 indices spanning [6649,8683] "wiener" 
[05:52:32.695] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 0, Qtot=47131367 Qloss=-4834119, 1389 indices spanning [8684,10072] "gauss" 
[05:52:33.192] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 1, Qtot=51603358 Qloss=-5932364, 1913 indices spanning [10073,11985] "wiener" 
[05:52:33.507] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 1, Qtot=45971072 Qloss=-3644810, 1282 indices spanning [11986,13267] "gauss" 
[05:52:34.079] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 2, Qtot=43124555 Qloss=-2573818, 1451 indices spanning [13268,14718] "wiener" 
[05:52:34.654] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 2, Qtot=41472278 Qloss=-876064, 1103 indices spanning [14719,15821] "gauss" 
[05:52:34.654] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 produce 15822 traces: 5399 wiener2, 0 decon_charge2, 3774 gauss2, frame tag: sigproc 
[05:52:34.654] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 output frame: frame: ident=6518 time=35 tick=500 with 15822 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":3774 [0] "mp2_roi2":5357 [0] "mp3_roi2":1292 [0] "wiener2":5399 [5399] ] cmm:[ ] 
[05:52:34.671] W [  glue  ] <ChannelSelector:chsel1> Untagged summary not supported, summary will be dropped. 
[05:52:34.672] D [  glue  ] <ChannelSelector:chsel1> input frame: ident=6518 time=35 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=6518 time=35 tick=512 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:52:34.672] D [  aux   ] <Resampler:resmp1> first ch=617 Ts=512 Ns=8112 Ns_pad=8125 Nrat=125 Tr=500 Nr=8320 Nout=8306 padding:linear 
[05:52:34.954] D [  aux   ] <Resampler:resmp1> resample 1536 traces at call=16 
[05:52:34.954] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 input frame: frame: ident=6518 time=35 tick=500 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:52:34.954] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 init nticks=8306 tbinmin=0 tbinmax=8306 
[05:52:34.984] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:52:37.883] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:52:40.723] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.2758e-05      172.038       365.839       155.513       95.7537         9     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.9983e-05    7.54364e-05   0.000172535    6.364e-05    3.43602e-05       9     
produce:tpcrawdecoder:PDVDTPCReader               13.6289       23.5159       64.7258       17.5462       14.9199         9     
produce:triggerrawdecoder:PDVDTriggerReader4     0.519806      0.541387       0.57836      0.540556      0.0171821        9     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000379924   0.000425538   0.000678566   0.000397016   9.01699e-05       9     
produce:ophit:OpHitFinder                        5.85e-05     0.00011052    0.000480624   6.0925e-05    0.000131046       9     
produce:opflash:OpFlashFinderVerticalDrift      5.0325e-05    8.00027e-05   0.000290296   5.1818e-05    7.44283e-05       9     
produce:wclsdatavd:WireCellToolkit                63.6854       87.2772       135.13        76.3016       25.2035         8     
produce:gaushit:GausHitFinder                    0.840748       1.41885       1.86981       1.42389      0.306744         8     
produce:nhitsfilter:NumberOfHitsFilter          0.000227088   0.000333129   0.000587044   0.000311882   0.000102413       8     
produce:reco3d:SpacePointSolver                   8.95567       15.1127       22.7061       14.5487       3.82342         8     
produce:hitpdune:DisambigFromSpacePoints         0.113861      0.248268      0.515097      0.224119      0.116487         8     
produce:pandora:StandardPandora                   14.2086       57.7742       215.136       34.7099       61.0724         8     
produce:pandoraTrack:LArPandoraTrackCreation     0.514613       1.55204       2.56424       1.41152      0.622078         8     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0292239     0.0434647     0.0814884     0.040717      0.0150899        8     
[art]:TriggerResults:TriggerResultInserter      2.7572e-05    3.69069e-05   6.5443e-05    2.90145e-05   1.27083e-05       8     
end_path:out1:RootOutput                         7.183e-06    1.05649e-05   2.2543e-05     7.86e-06     5.26396e-06       8     
end_path:out1:RootOutput(write)                   4.63218       5.36657       6.20573       5.25284      0.573868         8     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.86 MB
  Peak resident set size usage (VmHWM): 6706.06 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 05:53:42 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 6518
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:26:20 UTC       justIN version: 01.05.01