justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241288.4@dunegpschedd01.fnal.gov

Jobsub ID241288.4@dunegpschedd01.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 04:50:33
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-10-31 06:07:04
From worker nodeHostnamewn-sate-052.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 04:51:28
Input filesvd-protodune:np02vd_raw_run040266_0552_df-s03-d0_dw_0_20251025T124048.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 06:07:04
Saved logsjustin-logs:241288.4-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[  glue  ] <ChannelSelector:chsel7> input frame: ident=102024 time=92 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=102024 time=92 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[07:04:52.072] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 input frame: frame: ident=102024 time=92 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[07:04:52.073] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 init nticks=8000 tbinmin=0 tbinmax=8000 
[07:04:52.105] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 load plane index: 0, ntraces=1536, input bad regions: 0 
[07:04:54.176] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 load plane index: 1, ntraces=1536, input bad regions: 0 
[07:04:56.203] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 load plane index: 2, ntraces=1536, input bad regions: 0 
[07:05:19.173] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 0, Qtot=32098482643 Qloss=-37682949836, 4814 indices spanning [18946,23759] "wiener" 
[07:05:19.390] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 0, Qtot=30772466716 Qloss=-38195901725, 4129 indices spanning [23760,27888] "gauss" 
[07:05:19.958] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 1, Qtot=8595479267 Qloss=-1229034248, 4690 indices spanning [27889,32578] "wiener" 
[07:05:20.168] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 1, Qtot=7588475324 Qloss=-1378720034, 3955 indices spanning [32579,36533] "gauss" 
[07:05:20.597] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 2, Qtot=2962029711 Qloss=-1933466877, 11531 indices spanning [36534,48064] "wiener" 
[07:05:21.029] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 2, Qtot=2569237704 Qloss=-1544483011, 14173 indices spanning [48065,62237] "gauss" 
[07:05:21.029] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 produce 62238 traces: 21035 wiener7, 0 decon_charge7, 22257 gauss7, frame tag: sigproc 
[07:05:21.029] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 output frame: frame: ident=102024 time=92 tick=512 with 62238 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":22257 [0] "mp2_roi7":15887 [0] "mp3_roi7":3059 [0] "wiener7":21035 [21035] ] cmm:[ ] 
[07:05:23.464] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[07:05:23.465] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=102024 time=92 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=102024 time=92 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[07:05:23.465] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 input frame: frame: ident=102024 time=92 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[07:05:23.465] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 init nticks=8000 tbinmin=0 tbinmax=8000 
[07:05:23.496] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 load plane index: 0, ntraces=1536, input bad regions: 0 
[07:05:25.547] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 load plane index: 1, ntraces=1536, input bad regions: 0 
[07:05:27.644] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 load plane index: 2, ntraces=1536, input bad regions: 0 
[07:05:30.154] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 0, Qtot=37543894 Qloss=-6599197, 811 indices spanning [3649,4459] "wiener" 
[07:05:30.388] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 0, Qtot=35795156 Qloss=-5813923, 671 indices spanning [4460,5130] "gauss" 
[07:05:30.738] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 1, Qtot=43316573 Qloss=-7343582, 1701 indices spanning [5131,6831] "wiener" 
[07:05:30.957] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 1, Qtot=39792420 Qloss=-5455276, 1436 indices spanning [6832,8267] "gauss" 
[07:05:31.376] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 2, Qtot=37958067 Qloss=-2905694, 1637 indices spanning [8268,9904] "wiener" 
[07:05:31.789] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 2, Qtot=36397979 Qloss=-1306752, 1307 indices spanning [9905,11211] "gauss" 
[07:05:31.789] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 produce 11212 traces: 4149 wiener6, 0 decon_charge6, 3414 gauss6, frame tag: sigproc 
[07:05:31.789] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 output frame: frame: ident=102024 time=92 tick=512 with 11212 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":3414 [0] "mp2_roi6":2845 [0] "mp3_roi6":804 [0] "wiener6":4149 [4149] ] cmm:[ ] 
[07:05:31.801] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[07:05:31.801] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=102024 time=92 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=102024 time=92 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[07:05:31.802] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 input frame: frame: ident=102024 time=92 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[07:05:31.802] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 init nticks=8000 tbinmin=0 tbinmax=8000 
[07:05:31.834] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 load plane index: 0, ntraces=1536, input bad regions: 0 
[07:05:33.935] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 load plane index: 1, ntraces=1536, input bad regions: 0 
[07:05:35.978] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.7106e-05      206.014       1108.55       161.359       208.72         21     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.5785e-05    7.22329e-05   0.000167304   6.6485e-05    2.32698e-05      21     
produce:tpcrawdecoder:PDVDTPCReader               12.7386       28.8964       61.513        24.5847       14.1027        21     
produce:triggerrawdecoder:PDVDTriggerReader4     0.505991      0.643078       2.65121       0.53203      0.451059        21     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000367831   0.000404364   0.000719002   0.000384302   7.33052e-05      21     
produce:ophit:OpHitFinder                       5.7669e-05    8.39746e-05   0.000484952   6.1165e-05    8.99278e-05      21     
produce:opflash:OpFlashFinderVerticalDrift      5.1527e-05    6.54978e-05   0.000295786   5.3541e-05    5.15291e-05      21     
produce:wclsdatavd:WireCellToolkit                54.0682       83.2686       132.918       74.1803       19.1609        20     
produce:gaushit:GausHitFinder                    0.940891       1.62732       7.40207       1.30487       1.33992        20     
produce:nhitsfilter:NumberOfHitsFilter          0.000234521   0.000307053   0.000520459   0.000280607   6.3921e-05       20     
produce:reco3d:SpacePointSolver                   8.56943       14.2805       28.088        13.633        4.27903        20     
produce:hitpdune:DisambigFromSpacePoints         0.124339      0.209268      0.559682      0.187371      0.0929874       20     
produce:pandora:StandardPandora                   18.0577       80.8434       927.083       31.505        194.741        20     
produce:pandoraTrack:LArPandoraTrackCreation     0.798009       1.37886       3.9583        1.13104      0.719483        20     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0281051     0.0379781     0.0706178     0.0364909    0.00913968       20     
[art]:TriggerResults:TriggerResultInserter      1.8405e-05    2.76836e-05    6.392e-05    2.6284e-05    9.43983e-06      20     
end_path:out1:RootOutput                         6.973e-06    8.59815e-06   2.1721e-05    7.7245e-06    3.07136e-06      20     
end_path:out1:RootOutput(write)                   4.61793       5.11878       6.17669       5.01555      0.407697        20     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.7 MB
  Peak resident set size usage (VmHWM): 6654.43 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 07:06:34 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 102024
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:26:21 UTC       justIN version: 01.05.01