justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241285.4@dunegpschedd01.fnal.gov

Jobsub ID241285.4@dunegpschedd01.fnal.gov
Workflow ID9373
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 03:50:30
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-10-31 04:22:42
From worker nodeHostnamewn-sate-037.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 03:51:33
Input filesvd-protodune:np02vd_raw_run040266_0035_df-s04-d2_dw_0_20251024T221117.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 04:22:42
Saved logsjustin-logs:241285.4-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

=35 tick=500 with 1536 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:20:42.725] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 init nticks=8306 tbinmin=0 tbinmax=8306 
[05:20:42.752] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:20:45.605] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:20:48.485] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 load plane index: 2, ntraces=1536, input bad regions: 0 
[05:20:51.703] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 0, Qtot=37853797 Qloss=-5081593, 1559 indices spanning [7008,8566] "wiener" 
[05:20:52.009] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 0, Qtot=35431589 Qloss=-3241994, 1235 indices spanning [8567,9801] "gauss" 
[05:20:52.462] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 1, Qtot=44560004 Qloss=-6043089, 1736 indices spanning [9802,11537] "wiener" 
[05:20:52.768] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 1, Qtot=40852274 Qloss=-4042327, 1352 indices spanning [11538,12889] "gauss" 
[05:20:53.343] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 2, Qtot=33806285 Qloss=-1233981, 1108 indices spanning [12890,13997] "wiener" 
[05:20:53.913] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 2, Qtot=33079789 Qloss=-473960, 917 indices spanning [13998,14914] "gauss" 
[05:20:53.913] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 produce 14915 traces: 4403 wiener3, 0 decon_charge3, 3504 gauss3, frame tag: sigproc 
[05:20:53.913] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 output frame: frame: ident=6518 time=35 tick=500 with 14915 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":3504 [0] "mp2_roi3":5772 [0] "mp3_roi3":1236 [0] "wiener3":4403 [4403] ] cmm:[ ] 
[05:20:53.929] W [  glue  ] <ChannelSelector:chsel2> Untagged summary not supported, summary will be dropped. 
[05:20:53.930] D [  glue  ] <ChannelSelector:chsel2> input frame: ident=6518 time=35 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=6518 time=35 tick=512 with 1536 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:20:53.931] D [  aux   ] <Resampler:resmp2> first ch=3406 Ts=512 Ns=8112 Ns_pad=8125 Nrat=125 Tr=500 Nr=8320 Nout=8306 padding:linear 
[05:20:54.215] D [  aux   ] <Resampler:resmp2> resample 1536 traces at call=16 
[05:20:54.216] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 input frame: frame: ident=6518 time=35 tick=500 with 1536 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:20:54.216] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 init nticks=8306 tbinmin=0 tbinmax=8306 
[05:20:54.244] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:20:57.081] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:20:59.984] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 load plane index: 2, ntraces=1536, input bad regions: 0 
[05:21:03.619] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 0, Qtot=50823215 Qloss=-7170427, 2035 indices spanning [6649,8683] "wiener" 
[05:21:03.925] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 0, Qtot=47131367 Qloss=-4834119, 1389 indices spanning [8684,10072] "gauss" 
[05:21:04.410] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 1, Qtot=51603358 Qloss=-5932364, 1913 indices spanning [10073,11985] "wiener" 
[05:21:04.715] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 1, Qtot=45971072 Qloss=-3644810, 1282 indices spanning [11986,13267] "gauss" 
[05:21:05.288] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 2, Qtot=43124555 Qloss=-2573818, 1451 indices spanning [13268,14718] "wiener" 
[05:21:05.863] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 2, Qtot=41472278 Qloss=-876064, 1103 indices spanning [14719,15821] "gauss" 
[05:21:05.863] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 produce 15822 traces: 5399 wiener2, 0 decon_charge2, 3774 gauss2, frame tag: sigproc 
[05:21:05.863] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 output frame: frame: ident=6518 time=35 tick=500 with 15822 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":3774 [0] "mp2_roi2":5357 [0] "mp3_roi2":1292 [0] "wiener2":5399 [5399] ] cmm:[ ] 
[05:21:05.880] W [  glue  ] <ChannelSelector:chsel1> Untagged summary not supported, summary will be dropped. 
[05:21:05.880] D [  glue  ] <ChannelSelector:chsel1> input frame: ident=6518 time=35 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=6518 time=35 tick=512 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:21:05.881] D [  aux   ] <Resampler:resmp1> first ch=617 Ts=512 Ns=8112 Ns_pad=8125 Nrat=125 Tr=500 Nr=8320 Nout=8306 padding:linear 
[05:21:06.162] D [  aux   ] <Resampler:resmp1> resample 1536 traces at call=16 
[05:21:06.162] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 input frame: frame: ident=6518 time=35 tick=500 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:21:06.162] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 init nticks=8306 tbinmin=0 tbinmax=8306 
[05:21:06.188] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:21:09.009] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:21:11.845] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.4681e-05      173.072       369.819       160.186       96.9252         9     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.7358e-05    7.55022e-05   0.000171914    6.36e-05     3.42029e-05       9     
produce:tpcrawdecoder:PDVDTPCReader               13.2716       23.7055       54.1428       15.9819       12.7079         9     
produce:triggerrawdecoder:PDVDTriggerReader4     0.520486      0.553989       0.65043       0.54141      0.0379556        9     
produce:pdvddaphne:DAPHNEReaderPDVD             0.00037258    0.000444114   0.000726035   0.000412265   0.000103589       9     
produce:ophit:OpHitFinder                       6.0564e-05    0.000129981   0.000529817   8.4419e-05    0.00014211        9     
produce:opflash:OpFlashFinderVerticalDrift       5.317e-05    0.000103884   0.00037216    7.3648e-05    9.59732e-05       9     
produce:wclsdatavd:WireCellToolkit                62.9398       88.3769       139.465       76.8557       26.1024         8     
produce:gaushit:GausHitFinder                    0.851453       1.44631       2.08146       1.40441      0.350339         8     
produce:nhitsfilter:NumberOfHitsFilter          0.000230374   0.000335175   0.000570923   0.000313359   9.53964e-05       8     
produce:reco3d:SpacePointSolver                   9.20621       15.2651       22.5948       14.6971       3.75068         8     
produce:hitpdune:DisambigFromSpacePoints         0.115312      0.250747      0.525545      0.224266      0.118867         8     
produce:pandora:StandardPandora                   14.2483       58.4625       215.84        35.0723       61.1558         8     
produce:pandoraTrack:LArPandoraTrackCreation      0.62515       1.7483        3.23289       1.45593       0.75854         8     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0313457     0.0414889     0.0617429     0.0409977    0.00868725        8     
[art]:TriggerResults:TriggerResultInserter      3.0197e-05    4.98086e-05   0.000103234   4.5365e-05    2.11129e-05       8     
end_path:out1:RootOutput                         7.294e-06    1.23809e-05   3.3032e-05    1.0209e-05    7.90499e-06       8     
end_path:out1:RootOutput(write)                    4.63         5.32035       6.12572       5.22023      0.526744         8     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.81 MB
  Peak resident set size usage (VmHWM): 6700.02 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 05:22:13 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 6518
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:26:24 UTC       justIN version: 01.05.01