justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236277.0@dunegpschedd02.fnal.gov

Jobsub ID236277.0@dunegpschedd02.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-02 10:45:24
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-11-02 11:40:22
From worker nodeHostnamewn-sate-051.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-02 10:45:49
Input filesvd-protodune:np02vd_raw_run040266_0363_df-s04-d1_dw_0_20251025T071956.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-02 11:40:22
Saved logsjustin-logs:236277.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

orig3" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:37:55.253] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 init nticks=9911 tbinmin=0 tbinmax=9911 
[12:37:55.286] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 load plane index: 0, ntraces=1536, input bad regions: 0 
[12:38:00.169] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 load plane index: 1, ntraces=1536, input bad regions: 0 
[12:38:04.996] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 load plane index: 2, ntraces=1536, input bad regions: 0 
[12:38:14.024] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 save plane index: 0, Qtot=422520443 Qloss=-55237517, 6419 indices spanning [28621,35039] "wiener" 
[12:38:14.569] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 save plane index: 0, Qtot=366951554 Qloss=-50060155, 5035 indices spanning [35040,40074] "gauss" 
[12:38:15.528] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 save plane index: 1, Qtot=344323530 Qloss=-24553996, 7445 indices spanning [40075,47519] "wiener" 
[12:38:16.069] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 save plane index: 1, Qtot=263944263 Qloss=-18435384, 5639 indices spanning [47520,53158] "gauss" 
[12:38:16.921] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 save plane index: 2, Qtot=231611773 Qloss=-18713676, 8090 indices spanning [53159,61248] "wiener" 
[12:38:17.763] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 save plane index: 2, Qtot=218603469 Qloss=-5464523, 5973 indices spanning [61249,67221] "gauss" 
[12:38:17.763] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 produce 67222 traces: 21954 wiener3, 0 decon_charge3, 16647 gauss3, frame tag: sigproc 
[12:38:17.763] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 output frame: frame: ident=66973 time=64 tick=500 with 67222 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":16647 [0] "mp2_roi3":23065 [0] "mp3_roi3":5556 [0] "wiener3":21954 [21954] ] cmm:[ ] 
[12:38:17.955] W [  glue  ] <ChannelSelector:chsel2> Untagged summary not supported, summary will be dropped. 
[12:38:17.955] D [  glue  ] <ChannelSelector:chsel2> input frame: ident=66973 time=64 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=66973 time=64 tick=512 with 1536 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:38:17.956] D [  aux   ] <Resampler:resmp2> first ch=3406 Ts=512 Ns=9679 Ns_pad=9750 Nrat=125 Tr=500 Nr=9984 Nout=9911 padding:linear 
[12:38:18.244] D [  aux   ] <Resampler:resmp2> resample 1536 traces at call=28 
[12:38:18.244] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 input frame: frame: ident=66973 time=64 tick=500 with 1536 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:38:18.244] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 init nticks=9911 tbinmin=0 tbinmax=9911 
[12:38:18.278] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 load plane index: 0, ntraces=1536, input bad regions: 0 
[12:38:23.100] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 load plane index: 1, ntraces=1536, input bad regions: 0 
[12:38:28.212] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 load plane index: 2, ntraces=1536, input bad regions: 0 
[12:38:35.013] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 save plane index: 0, Qtot=119281491 Qloss=-17388054, 2634 indices spanning [12887,15520] "wiener" 
[12:38:35.562] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 save plane index: 0, Qtot=113357839 Qloss=-14800482, 2172 indices spanning [15521,17692] "gauss" 
[12:38:36.408] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 save plane index: 1, Qtot=112248720 Qloss=-17840914, 2904 indices spanning [17693,20596] "wiener" 
[12:38:36.963] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 save plane index: 1, Qtot=102036603 Qloss=-14837819, 2151 indices spanning [20597,22747] "gauss" 
[12:38:37.832] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 save plane index: 2, Qtot=95809783 Qloss=-6082536, 3941 indices spanning [22748,26688] "wiener" 
[12:38:38.691] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 save plane index: 2, Qtot=92315519 Qloss=-2538706, 3107 indices spanning [26689,29795] "gauss" 
[12:38:38.691] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 produce 29796 traces: 9479 wiener2, 0 decon_charge2, 7430 gauss2, frame tag: sigproc 
[12:38:38.691] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 output frame: frame: ident=66973 time=64 tick=500 with 29796 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":7430 [0] "mp2_roi2":10770 [0] "mp3_roi2":2117 [0] "wiener2":9479 [9479] ] cmm:[ ] 
[12:38:38.723] W [  glue  ] <ChannelSelector:chsel1> Untagged summary not supported, summary will be dropped. 
[12:38:38.724] D [  glue  ] <ChannelSelector:chsel1> input frame: ident=66973 time=64 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=66973 time=64 tick=512 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:38:38.724] D [  aux   ] <Resampler:resmp1> first ch=617 Ts=512 Ns=9679 Ns_pad=9750 Nrat=125 Tr=500 Nr=9984 Nout=9911 padding:linear 
[12:38:39.014] D [  aux   ] <Resampler:resmp1> resample 1536 traces at call=28 
[12:38:39.014] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=28 input frame: frame: ident=66973 time=64 tick=500 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:38:39.014] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=28 init nticks=9911 tbinmin=0 tbinmax=9911 
[12:38:39.047] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=28 load plane index: 0, ntraces=1536, input bad regions: 0 
[12:38:43.861] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=28 load plane index: 1, ntraces=1536, input bad regions: 0 
[12:38:48.650] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=28 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.6445e-05      197.023       328.291       187.538       69.8293        15     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.6727e-05    7.32126e-05   0.000178785   6.6445e-05    2.90549e-05      15     
produce:tpcrawdecoder:PDVDTPCReader               46.6349       64.3558       117.15        61.2856       16.4363        15     
produce:triggerrawdecoder:PDVDTriggerReader4     0.530903      0.637177       1.12394      0.561484      0.155931        15     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000367439   0.000403314   0.000748964   0.000377427   9.27591e-05      15     
produce:ophit:OpHitFinder                       5.9552e-05    0.000103099   0.000605655   6.2998e-05    0.000134518      15     
produce:opflash:OpFlashFinderVerticalDrift      5.0865e-05    8.42755e-05   0.000489398   5.3871e-05    0.00010833       15     
produce:wclsdatavd:WireCellToolkit                68.5222       84.8651       115.512       75.5495       17.3887        14     
produce:gaushit:GausHitFinder                    0.961432       1.24545       1.77009       1.21213      0.229432        14     
produce:nhitsfilter:NumberOfHitsFilter          0.000222467   0.000287632   0.000512791   0.000273012   6.94813e-05      14     
produce:reco3d:SpacePointSolver                   7.17124       12.3418       17.9993       11.7923       3.12269        14     
produce:hitpdune:DisambigFromSpacePoints         0.104887      0.189705      0.342952      0.165031      0.0695022       14     
produce:pandora:StandardPandora                   16.4491       41.5874       158.632       28.6445       35.6487        14     
produce:pandoraTrack:LArPandoraTrackCreation     0.621678       1.39547       2.50203       1.25283      0.622434        14     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0249725     0.0347865     0.0476326     0.0359032    0.00687796       14     
[art]:TriggerResults:TriggerResultInserter      2.3825e-05    3.50735e-05   7.1645e-05    3.14535e-05   1.18075e-05      14     
end_path:out1:RootOutput                         3.927e-06    7.36757e-06   2.1781e-05     6.578e-06    4.37066e-06      14     
end_path:out1:RootOutput(write)                   4.60126       5.01273       5.73242       4.95909      0.302661        14     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.89 MB
  Peak resident set size usage (VmHWM): 6699.53 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 02-Nov-2025 12:40:01 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 66973
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-04 01:28:18 UTC       justIN version: 01.05.01