justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241288.5@dunegpschedd01.fnal.gov

Jobsub ID241288.5@dunegpschedd01.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 04:50:33
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-10-31 05:00:51
From worker nodeHostnamewn-sate-039.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 04:51:01
Input filesvd-protodune:np02vd_raw_run040266_0279_df-s03-d1_dw_0_20251025T050008.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 05:00:51
Saved logsjustin-logs:241288.5-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

500 with 1536 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:58:39.798] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=4 init nticks=8441 tbinmin=0 tbinmax=8441 
[05:58:39.831] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=4 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:58:43.168] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=4 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:58:46.489] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=4 load plane index: 2, ntraces=1536, input bad regions: 0 
[05:58:53.523] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=4 save plane index: 0, Qtot=337089811 Qloss=-39132277, 9257 indices spanning [41325,50581] "wiener" 
[05:58:53.884] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=4 save plane index: 0, Qtot=301580661 Qloss=-28137200, 7029 indices spanning [50582,57610] "gauss" 
[05:58:54.725] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=4 save plane index: 1, Qtot=381629900 Qloss=-30291123, 8678 indices spanning [57611,66288] "wiener" 
[05:58:55.087] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=4 save plane index: 1, Qtot=319361144 Qloss=-22019715, 6659 indices spanning [66289,72947] "gauss" 
[05:58:55.697] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=4 save plane index: 2, Qtot=262183739 Qloss=-15664433, 9700 indices spanning [72948,82647] "wiener" 
[05:58:56.298] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=4 save plane index: 2, Qtot=251525011 Qloss=-4765331, 7140 indices spanning [82648,89787] "gauss" 
[05:58:56.298] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=4 produce 89788 traces: 27635 wiener3, 0 decon_charge3, 20828 gauss3, frame tag: sigproc 
[05:58:56.298] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=4 output frame: frame: ident=51537 time=6 tick=500 with 89788 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":20828 [0] "mp2_roi3":33577 [0] "mp3_roi3":7748 [0] "wiener3":27635 [27635] ] cmm:[ ] 
[05:58:56.560] W [  glue  ] <ChannelSelector:chsel2> Untagged summary not supported, summary will be dropped. 
[05:58:56.561] D [  glue  ] <ChannelSelector:chsel2> input frame: ident=51537 time=6 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=51537 time=6 tick=512 with 1536 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:58:56.561] D [  aux   ] <Resampler:resmp2> first ch=3406 Ts=512 Ns=8244 Ns_pad=8250 Nrat=125 Tr=500 Nr=8448 Nout=8441 padding:linear 
[05:58:56.811] D [  aux   ] <Resampler:resmp2> resample 1536 traces at call=4 
[05:58:56.811] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 input frame: frame: ident=51537 time=6 tick=500 with 1536 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:58:56.811] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 init nticks=8441 tbinmin=0 tbinmax=8441 
[05:58:56.844] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:59:00.179] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:59:03.542] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 load plane index: 2, ntraces=1536, input bad regions: 0 
[05:59:08.443] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 save plane index: 0, Qtot=126492531 Qloss=-16716299, 3517 indices spanning [24125,27641] "wiener" 
[05:59:08.808] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 save plane index: 0, Qtot=119163125 Qloss=-13494962, 2633 indices spanning [27642,30274] "gauss" 
[05:59:09.460] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 save plane index: 1, Qtot=132781087 Qloss=-14412492, 3676 indices spanning [30275,33950] "wiener" 
[05:59:09.821] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 save plane index: 1, Qtot=117403698 Qloss=-10908135, 2852 indices spanning [33951,36802] "gauss" 
[05:59:10.438] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 save plane index: 2, Qtot=99946860 Qloss=-5973449, 4422 indices spanning [36803,41224] "wiener" 
[05:59:11.044] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 save plane index: 2, Qtot=96103425 Qloss=-2015217, 3335 indices spanning [41225,44559] "gauss" 
[05:59:11.044] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 produce 44560 traces: 11615 wiener2, 0 decon_charge2, 8820 gauss2, frame tag: sigproc 
[05:59:11.044] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=4 output frame: frame: ident=51537 time=6 tick=500 with 44560 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":8820 [0] "mp2_roi2":21491 [0] "mp3_roi2":2634 [0] "wiener2":11615 [11615] ] cmm:[ ] 
[05:59:11.093] W [  glue  ] <ChannelSelector:chsel1> Untagged summary not supported, summary will be dropped. 
[05:59:11.094] D [  glue  ] <ChannelSelector:chsel1> input frame: ident=51537 time=6 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=51537 time=6 tick=512 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:59:11.094] D [  aux   ] <Resampler:resmp1> first ch=617 Ts=512 Ns=8244 Ns_pad=8250 Nrat=125 Tr=500 Nr=8448 Nout=8441 padding:linear 
[05:59:11.339] D [  aux   ] <Resampler:resmp1> resample 1536 traces at call=4 
[05:59:11.339] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=4 input frame: frame: ident=51537 time=6 tick=500 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:59:11.340] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=4 init nticks=8441 tbinmin=0 tbinmax=8441 
[05:59:11.374] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=4 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:59:14.648] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=4 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:59:17.923] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=4 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.8328e-05      95.9923       152.422       135.555       68.2252         3     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.6506e-05    0.00011506    0.000210346   6.8328e-05    6.73815e-05       3     
produce:tpcrawdecoder:PDVDTPCReader               12.7318        13.47        14.4647       13.2134      0.730343         3     
produce:triggerrawdecoder:PDVDTriggerReader4      0.51644      0.525587      0.533582       0.52674     0.00704558        3     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000376419   0.000528472   0.00078633    0.000422666   0.000183308       3     
produce:ophit:OpHitFinder                       5.7708e-05    0.000218798   0.000530629   6.8058e-05    0.000220538       3     
produce:opflash:OpFlashFinderVerticalDrift      5.0295e-05    0.000137833   0.000309693   5.3511e-05    0.00012153        3     
produce:wclsdatavd:WireCellToolkit                77.9728       81.8824       85.792        81.8824       3.90965         2     
produce:gaushit:GausHitFinder                     1.16416       1.29274       1.42133       1.29274      0.128586         2     
produce:nhitsfilter:NumberOfHitsFilter          0.000336954   0.000461603   0.000586253   0.000461603   0.000124649       2     
produce:reco3d:SpacePointSolver                   11.9356       13.5487       15.1618       13.5487       1.61309         2     
produce:hitpdune:DisambigFromSpacePoints          0.15799      0.181497      0.205003      0.181497      0.0235067        2     
produce:pandora:StandardPandora                   24.6539       25.8569       27.0599       25.8569        1.203          2     
produce:pandoraTrack:LArPandoraTrackCreation      1.1629        1.44005       1.7172        1.44005      0.277149         2     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0370795     0.043304      0.0495285     0.043304     0.00622453        2     
[art]:TriggerResults:TriggerResultInserter      5.4573e-05    8.9804e-05    0.000125035   8.9804e-05    3.5231e-05        2     
end_path:out1:RootOutput                         9.168e-06    2.62295e-05   4.3291e-05    2.62295e-05   1.70615e-05       2     
end_path:out1:RootOutput(write)                   5.15378       5.56362       5.97346       5.56362      0.409839         2     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.71 MB
  Peak resident set size usage (VmHWM): 6681.92 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 06:00:19 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 51537
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:26:20 UTC       justIN version: 01.05.01