justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 235927.0@dunegpschedd02.fnal.gov

Jobsub ID235927.0@dunegpschedd02.fnal.gov
Workflow ID9373
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-01 05:53:53
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-11-01 06:40:47
From worker nodeHostnamewn-sate-053.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-01 05:54:29
Input filesvd-protodune:np02vd_raw_run040266_0225_df-s04-d2_dw_0_20251025T032811.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-01 06:40:47
Saved logsjustin-logs:235927.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

6] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=41630 time=43 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=41630 time=43 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[07:38:42.767] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 input frame: frame: ident=41630 time=43 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[07:38:42.767] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 init nticks=8634 tbinmin=0 tbinmax=8634 
[07:38:42.799] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 load plane index: 0, ntraces=1536, input bad regions: 0 
[07:38:45.746] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 load plane index: 1, ntraces=1536, input bad regions: 0 
[07:38:48.730] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 load plane index: 2, ntraces=1536, input bad regions: 0 
[07:38:55.732] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 save plane index: 0, Qtot=313922323 Qloss=-36000495, 8398 indices spanning [51597,59994] "wiener" 
[07:38:56.051] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 save plane index: 0, Qtot=296355064 Qloss=-24312922, 6158 indices spanning [59995,66152] "gauss" 
[07:38:56.767] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 save plane index: 1, Qtot=351045954 Qloss=-43986102, 8389 indices spanning [66153,74541] "wiener" 
[07:38:57.089] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 save plane index: 1, Qtot=316435302 Qloss=-30399552, 6363 indices spanning [74542,80904] "gauss" 
[07:38:57.671] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 save plane index: 2, Qtot=287504791 Qloss=-23114123, 9145 indices spanning [80905,90049] "wiener" 
[07:38:58.244] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 save plane index: 2, Qtot=274432690 Qloss=-9795925, 7251 indices spanning [90050,97300] "gauss" 
[07:38:58.245] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 produce 97301 traces: 25932 wiener6, 0 decon_charge6, 19772 gauss6, frame tag: sigproc 
[07:38:58.245] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=18 output frame: frame: ident=41630 time=43 tick=512 with 97301 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":19772 [0] "mp2_roi6":43544 [0] "mp3_roi6":8053 [0] "wiener6":25932 [25932] ] cmm:[ ] 
[07:38:58.474] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[07:38:58.474] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=41630 time=43 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=41630 time=43 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[07:38:58.474] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 input frame: frame: ident=41630 time=43 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[07:38:58.474] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 init nticks=8634 tbinmin=0 tbinmax=8634 
[07:38:58.508] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 load plane index: 0, ntraces=1536, input bad regions: 0 
[07:39:01.515] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 load plane index: 1, ntraces=1536, input bad regions: 0 
[07:39:04.530] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 load plane index: 2, ntraces=1536, input bad regions: 0 
[07:39:13.477] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 save plane index: 0, Qtot=613887400 Qloss=-56375063, 7496 indices spanning [41208,48703] "wiener" 
[07:39:13.794] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 save plane index: 0, Qtot=584365767 Qloss=-48355241, 5651 indices spanning [48704,54354] "gauss" 
[07:39:14.519] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 save plane index: 1, Qtot=395206695 Qloss=-36470864, 8266 indices spanning [54355,62620] "wiener" 
[07:39:14.861] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 save plane index: 1, Qtot=343449446 Qloss=-26597196, 6306 indices spanning [62621,68926] "gauss" 
[07:39:15.457] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 save plane index: 2, Qtot=317954367 Qloss=-24233223, 8263 indices spanning [68927,77189] "wiener" 
[07:39:16.030] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 save plane index: 2, Qtot=303975531 Qloss=-9776294, 6546 indices spanning [77190,83735] "gauss" 
[07:39:16.030] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 produce 83736 traces: 24025 wiener5, 0 decon_charge5, 18503 gauss5, frame tag: sigproc 
[07:39:16.030] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=18 output frame: frame: ident=41630 time=43 tick=512 with 83736 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":18503 [0] "mp2_roi5":32601 [0] "mp3_roi5":8607 [0] "wiener5":24025 [24025] ] cmm:[ ] 
[07:39:16.646] W [  glue  ] <ChannelSelector:chsel4> Untagged summary not supported, summary will be dropped. 
[07:39:16.647] D [  glue  ] <ChannelSelector:chsel4> input frame: ident=41630 time=43 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=41630 time=43 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[07:39:16.647] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=18 input frame: frame: ident=41630 time=43 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[07:39:16.647] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=18 init nticks=8634 tbinmin=0 tbinmax=8634 
[07:39:16.682] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=18 load plane index: 0, ntraces=1536, input bad regions: 0 
[07:39:19.627] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=18 load plane index: 1, ntraces=1536, input bad regions: 0 
[07:39:22.685] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=18 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.5794e-05      251.925       947.094       194.581        239.3         10     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.5014e-05    7.87143e-05   0.000172224   6.5458e-05    3.29696e-05      10     
produce:tpcrawdecoder:PDVDTPCReader               38.7666       61.7139       79.4864       62.4819       13.7358        10     
produce:triggerrawdecoder:PDVDTriggerReader4     0.520408      0.656828       1.27978      0.572593      0.217048        10     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000364797   0.000419594   0.000676113   0.000386463   8.78592e-05      10     
produce:ophit:OpHitFinder                       5.5074e-05    0.000111863   0.000492046   5.9312e-05    0.000127989      10     
produce:opflash:OpFlashFinderVerticalDrift      5.1827e-05    8.38226e-05   0.000282071   5.7268e-05    6.78926e-05      10     
produce:wclsdatavd:WireCellToolkit                68.5222       82.7999       132.617       74.0624       19.359          9     
produce:gaushit:GausHitFinder                      1.01         1.56689       3.17729       1.30898       0.62606         9     
produce:nhitsfilter:NumberOfHitsFilter          0.000221407   0.000344217   0.000566998   0.000302069   0.000106715       9     
produce:reco3d:SpacePointSolver                   9.39078       17.1963       49.3262       12.2327       11.7963         9     
produce:hitpdune:DisambigFromSpacePoints         0.114486      0.323958       1.0994       0.248117      0.282348         9     
produce:pandora:StandardPandora                   17.236        109.008       660.615       46.4918       195.556         9     
produce:pandoraTrack:LArPandoraTrackCreation     0.646181       2.93787       14.9234       1.40903       4.26978         9     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0311744     0.0429082     0.0915258     0.0338666     0.0183368        9     
[art]:TriggerResults:TriggerResultInserter      2.4175e-05    3.33102e-05   6.5433e-05    2.8203e-05    1.17076e-05       9     
end_path:out1:RootOutput                         6.523e-06    8.90778e-06   2.1079e-05     7.073e-06    4.37596e-06       9     
end_path:out1:RootOutput(write)                   4.66547       5.24382       6.42486       5.15483      0.488619         9     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.62 MB
  Peak resident set size usage (VmHWM): 6685.05 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 01-Nov-2025 07:40:18 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 41630
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-04 01:28:04 UTC       justIN version: 01.05.01