justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 41115.38@dunegpschedd02.fnal.gov

Jobsub ID41115.38@dunegpschedd02.fnal.gov
Workflow ID2742
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-17 17:17:36
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_brug
Last heartbeat2025-09-17 17:34:40
From worker nodeHostnamewn-snel-002.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-17 17:20:56
Input filesvd-protodune:np02vd_raw_run039433_0006_df-s05-d5_dw_0_20250917T155325.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-17 17:34:40
Saved logsjustin-logs:41115.38-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e: frame: ident=1757 time=11 tick=500 with 1536 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] 
[19:32:50.670] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=8 init nticks=10776 tbinmin=0 tbinmax=10776 
[19:32:50.705] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=8 load plane index: 0, ntraces=1536, input bad regions: 0 
[19:32:53.608] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=8 load plane index: 1, ntraces=1536, input bad regions: 0 
[19:32:56.440] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=8 load plane index: 2, ntraces=1536, input bad regions: 0 
[19:33:00.137] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=8 save plane index: 0, Qtot=46860739 Qloss=-8010153, 1291 indices spanning [4865,6155] "wiener" 
[19:33:00.440] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=8 save plane index: 0, Qtot=45964611 Qloss=-7568345, 1057 indices spanning [6156,7212] "gauss" 
[19:33:00.885] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=8 save plane index: 1, Qtot=51387989 Qloss=-7763446, 2347 indices spanning [7213,9559] "wiener" 
[19:33:01.193] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=8 save plane index: 1, Qtot=48778784 Qloss=-6198169, 1924 indices spanning [9560,11483] "gauss" 
[19:33:01.690] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=8 save plane index: 2, Qtot=43125380 Qloss=-1948482, 3348 indices spanning [11484,14831] "wiener" 
[19:33:02.187] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=8 save plane index: 2, Qtot=42131201 Qloss=-951435, 2695 indices spanning [14832,17526] "gauss" 
[19:33:02.188] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=8 produce 17527 traces: 6986 wiener3, 0 decon_charge3, 5676 gauss3, frame tag: sigproc 
[19:33:02.188] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=8 output frame: frame: ident=1757 time=11 tick=500 with 17527 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":5676 [0] "mp2_roi3":3342 [0] "mp3_roi3":1523 [0] "wiener3":6986 [6986] ] cmm:[ ] 
[19:33:02.198] W [  glue  ] <ChannelSelector:chsel2> Untagged summary not supported, summary will be dropped. 
[19:33:02.198] D [  glue  ] <ChannelSelector:chsel2> input frame: ident=1757 time=11 tick=512 with 6144 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=1757 time=11 tick=512 with 1536 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[19:33:02.199] D [  aux   ] <Resampler:resmp2> first ch=3406 Ts=512 Ns=10524 Ns_pad=10625 Nrat=125 Tr=500 Nr=10880 Nout=10776 padding:linear 
[19:33:02.642] D [  aux   ] <Resampler:resmp2> resample 1536 traces at call=8 
[19:33:02.642] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 input frame: frame: ident=1757 time=11 tick=500 with 1536 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[19:33:02.642] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 init nticks=10776 tbinmin=0 tbinmax=10776 
[19:33:02.668] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 load plane index: 0, ntraces=1536, input bad regions: 0 
[19:33:05.387] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 load plane index: 1, ntraces=1536, input bad regions: 0 
[19:33:08.217] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 load plane index: 2, ntraces=1536, input bad regions: 0 
[19:33:11.628] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 save plane index: 0, Qtot=41391986 Qloss=-6996879, 1208 indices spanning [5003,6210] "wiener" 
[19:33:11.925] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 save plane index: 0, Qtot=40086555 Qloss=-6590495, 1006 indices spanning [6211,7216] "gauss" 
[19:33:12.382] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 save plane index: 1, Qtot=42254781 Qloss=-6542473, 1749 indices spanning [7217,8965] "wiener" 
[19:33:12.689] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 save plane index: 1, Qtot=39101772 Qloss=-5407132, 1410 indices spanning [8966,10375] "gauss" 
[19:33:13.185] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 save plane index: 2, Qtot=34565079 Qloss=-2094633, 2635 indices spanning [10376,13010] "wiener" 
[19:33:13.685] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 save plane index: 2, Qtot=33394871 Qloss=-916377, 2164 indices spanning [13011,15174] "gauss" 
[19:33:13.685] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 produce 15175 traces: 5592 wiener2, 0 decon_charge2, 4580 gauss2, frame tag: sigproc 
[19:33:13.685] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=8 output frame: frame: ident=1757 time=11 tick=500 with 15175 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":4580 [0] "mp2_roi2":3898 [0] "mp3_roi2":1105 [0] "wiener2":5592 [5592] ] cmm:[ ] 
[19:33:13.697] W [  glue  ] <ChannelSelector:chsel1> Untagged summary not supported, summary will be dropped. 
[19:33:13.697] D [  glue  ] <ChannelSelector:chsel1> input frame: ident=1757 time=11 tick=512 with 6144 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=1757 time=11 tick=512 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[19:33:13.698] D [  aux   ] <Resampler:resmp1> first ch=617 Ts=512 Ns=10524 Ns_pad=10625 Nrat=125 Tr=500 Nr=10880 Nout=10776 padding:linear 
[19:33:14.140] D [  aux   ] <Resampler:resmp1> resample 1536 traces at call=8 
[19:33:14.141] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=8 input frame: frame: ident=1757 time=11 tick=500 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[19:33:14.141] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=8 init nticks=10776 tbinmin=0 tbinmax=10776 
[19:33:14.167] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=8 load plane index: 0, ntraces=1536, input bad regions: 0 
[19:33:16.831] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=8 load plane index: 1, ntraces=1536, input bad regions: 0 
[19:33:19.607] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=8 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.2307e-05      69.0006       116.54        72.6916       38.3636         5     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.2307e-05     0.0001053    0.000247505   6.7416e-05    7.15056e-05       5     
produce:tpcrawdecoder:PDVDTPCReader               31.3092       41.6683       51.666        40.6435       8.24436         5     
produce:triggerrawdecoder:PDVDTriggerReader4     0.289691      0.393031      0.563873      0.322948      0.113652         5     
produce:pdvddaphne:DAPHNEReaderPDVD               7.56949       9.40205       13.561        8.32682       2.20141         5     
produce:ophit:OpHitFinder                        0.0462245     0.0600812     0.0804838     0.0559985     0.0116397        5     
produce:opflash:OpFlashFinderVerticalDrift      0.00700525     0.0145721     0.0226006     0.0150435    0.00504343        5     
produce:wclsdatavd:WireCellToolkit                16.9341       24.1781       31.6702       24.054        6.64152         4     
produce:gaushit:GausHitFinder                    0.263996      0.370619      0.494536      0.361973      0.0845012        4     
produce:nhitsfilter:NumberOfHitsFilter          5.4332e-05    0.000123763   0.000272954   8.3883e-05    8.7828e-05        4     
produce:reco3d:SpacePointSolver                   1.79799       2.55716       3.33614       2.54726      0.699178         4     
produce:hitpdune:DisambigFromSpacePoints         0.015132      0.0464864     0.093449      0.0386822     0.0303703        4     
produce:pandora:StandardPandora                   3.43447       7.51055       13.1234       6.74218       4.09315         4     
produce:pandoraTrack:LArPandoraTrackCreation     0.0750159     0.176273      0.272707      0.178685      0.0841764        4     
produce:pandoraGnocalo:GnocchiCalorimetry       0.00762178    0.00983039     0.0118257    0.00993705     0.0017939        4     
[art]:TriggerResults:TriggerResultInserter      1.1722e-05    2.05182e-05   4.5054e-05    1.26485e-05   1.4173e-05        4     
end_path:out1:RootOutput                         3.466e-06    6.75525e-06    1.573e-05    3.9125e-06    5.18513e-06       4     
end_path:out1:RootOutput(write)                   1.50034       1.71024       2.01718       1.66172       0.19527         4     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.84 MB
  Peak resident set size usage (VmHWM): 6708.7 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 19:34:22 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39433 subRun: 1 event: 1757
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-18 23:58:56 UTC       justIN version: 01.05.00