justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236254.4@dunegpschedd02.fnal.gov

Jobsub ID236254.4@dunegpschedd02.fnal.gov
Workflow ID9405
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-02 08:01:16
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-11-02 08:12:44
From worker nodeHostnamewn-sate-046.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-02 08:01:30
Input filesvd-protodune:np02vd_raw_run040270_0656_df-s03-d1_dw_0_20251028T070349.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-02 08:12:44
Saved logsjustin-logs:236254.4-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

10:18.947] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=121321 time=11 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=121321 time=11 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:10:18.947] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 input frame: frame: ident=121321 time=11 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:10:18.947] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 init nticks=8640 tbinmin=0 tbinmax=8640 
[09:10:18.976] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 load plane index: 0, ntraces=1536, input bad regions: 0 
[09:10:20.896] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 load plane index: 1, ntraces=1536, input bad regions: 0 
[09:10:22.930] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 load plane index: 2, ntraces=1536, input bad regions: 0 
[09:10:50.718] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 save plane index: 0, Qtot=40863744435 Qloss=-171505042674, 4550 indices spanning [12222,16771] "wiener" 
[09:10:50.949] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 save plane index: 0, Qtot=39620172215 Qloss=-170510179744, 4090 indices spanning [16772,20861] "gauss" 
[09:10:51.539] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 save plane index: 1, Qtot=14183496525 Qloss=-2437800821, 4098 indices spanning [20862,24959] "wiener" 
[09:10:51.754] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 save plane index: 1, Qtot=13373431152 Qloss=-2554421779, 3557 indices spanning [24960,28516] "gauss" 
[09:10:52.207] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 save plane index: 2, Qtot=5277807627 Qloss=-2697933352, 12765 indices spanning [28517,41281] "wiener" 
[09:10:52.679] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 save plane index: 2, Qtot=4655059678 Qloss=-2082485689, 14509 indices spanning [41282,55790] "gauss" 
[09:10:52.679] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 produce 55791 traces: 21413 wiener7, 0 decon_charge7, 22156 gauss7, frame tag: sigproc 
[09:10:52.679] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=4 output frame: frame: ident=121321 time=11 tick=512 with 55791 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":22156 [0] "mp2_roi7":9743 [0] "mp3_roi7":2479 [0] "wiener7":21413 [21413] ] cmm:[ ] 
[09:10:55.957] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[09:10:55.958] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=121321 time=11 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=121321 time=11 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:10:55.958] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 input frame: frame: ident=121321 time=11 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:10:55.958] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 init nticks=8640 tbinmin=0 tbinmax=8640 
[09:10:55.990] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 load plane index: 0, ntraces=1536, input bad regions: 0 
[09:10:58.154] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 load plane index: 1, ntraces=1536, input bad regions: 0 
[09:11:00.373] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 load plane index: 2, ntraces=1536, input bad regions: 0 
[09:11:03.989] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 save plane index: 0, Qtot=97922682 Qloss=-14965758, 2009 indices spanning [7146,9154] "wiener" 
[09:11:04.210] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 save plane index: 0, Qtot=94368045 Qloss=-13010453, 1722 indices spanning [9155,10876] "gauss" 
[09:11:04.585] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 save plane index: 1, Qtot=76533040 Qloss=-10740618, 2463 indices spanning [10877,13339] "wiener" 
[09:11:04.806] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 save plane index: 1, Qtot=70739338 Qloss=-8340272, 2059 indices spanning [13340,15398] "gauss" 
[09:11:05.251] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 save plane index: 2, Qtot=76988674 Qloss=-4432568, 2485 indices spanning [15399,17883] "wiener" 
[09:11:05.695] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 save plane index: 2, Qtot=75123502 Qloss=-2534341, 2142 indices spanning [17884,20025] "gauss" 
[09:11:05.695] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 produce 20026 traces: 6957 wiener6, 0 decon_charge6, 5923 gauss6, frame tag: sigproc 
[09:11:05.696] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=4 output frame: frame: ident=121321 time=11 tick=512 with 20026 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":5923 [0] "mp2_roi6":5436 [0] "mp3_roi6":1710 [0] "wiener6":6957 [6957] ] cmm:[ ] 
[09:11:05.712] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[09:11:05.712] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=121321 time=11 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=121321 time=11 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:11:05.713] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=4 input frame: frame: ident=121321 time=11 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:11:05.713] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=4 init nticks=8626 tbinmin=0 tbinmax=8626 
[09:11:05.747] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=4 load plane index: 0, ntraces=1536, input bad regions: 0 
[09:11:09.185] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=4 load plane index: 1, ntraces=1536, input bad regions: 0 
[09:11:12.620] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=4 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.6265e-05      139.062       226.909       190.275       99.4622         3     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.9912e-05    0.000104359   0.000186901   6.6265e-05    5.84234e-05       3     
produce:tpcrawdecoder:PDVDTPCReader               38.7332       58.9982       69.5897       68.6715       14.3344         3     
produce:triggerrawdecoder:PDVDTriggerReader4      0.32552       0.3734       0.429351       0.36533      0.0427715        3     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000386756   0.000512913   0.000762933   0.000389051   0.000176793       3     
produce:ophit:OpHitFinder                       6.9972e-05    0.000204097   0.000471486   7.0833e-05    0.000189073       3     
produce:opflash:OpFlashFinderVerticalDrift      5.7598e-05    0.000142725   0.00029789    7.2686e-05    0.000109891       3     
produce:wclsdatavd:WireCellToolkit                58.8442       87.8307       116.817       87.8307       28.9865         2     
produce:gaushit:GausHitFinder                     1.36025       1.43666       1.51306       1.43666      0.0764022        2     
produce:nhitsfilter:NumberOfHitsFilter          0.000333106   0.000413321   0.000493537   0.000413321   8.02155e-05       2     
produce:reco3d:SpacePointSolver                   13.1073       14.1612       15.215        14.1612       1.05385         2     
produce:hitpdune:DisambigFromSpacePoints         0.220184      0.242076      0.263969      0.242076      0.0218925        2     
produce:pandora:StandardPandora                   40.0102       43.3967       46.7832       43.3967       3.38651         2     
produce:pandoraTrack:LArPandoraTrackCreation      1.68327       1.72764       1.77202       1.72764      0.0443746        2     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0420918     0.0442072     0.0463226     0.0442072    0.00211539        2     
[art]:TriggerResults:TriggerResultInserter       4.246e-05    6.77175e-05   9.2975e-05    6.77175e-05   2.52575e-05       2     
end_path:out1:RootOutput                         5.36e-06     1.47325e-05   2.4105e-05    1.47325e-05   9.3725e-06        2     
end_path:out1:RootOutput(write)                   4.94749       5.1939        5.44031       5.1939       0.246413         2     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.52 MB
  Peak resident set size usage (VmHWM): 6655.25 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 02-Nov-2025 09:12:21 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40270 subRun: 1 event: 121321
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-04 01:28:20 UTC       justIN version: 01.05.01