justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241275.1@dunegpschedd01.fnal.gov

Jobsub ID241275.1@dunegpschedd01.fnal.gov
Workflow ID9373
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 01:46:23
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-10-31 02:47:15
From worker nodeHostnamewn-pep-012.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 01:47:54
Input filesvd-protodune:np02vd_raw_run040266_0017_df-s03-d2_dw_0_20251024T214052.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 02:47:15
Saved logsjustin-logs:241275.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

5.628] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=3282 time=74 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:44:55.628] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 input frame: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:44:55.628] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 init nticks=8064 tbinmin=0 tbinmax=8064 
[03:44:55.665] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 load plane index: 0, ntraces=1536, input bad regions: 0 
[03:44:57.893] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 load plane index: 1, ntraces=1536, input bad regions: 0 
[03:45:00.119] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 load plane index: 2, ntraces=1536, input bad regions: 0 
[03:45:15.438] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 0, Qtot=1597475914 Qloss=-44434254, 8992 indices spanning [42225,51216] "wiener" 
[03:45:15.690] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 0, Qtot=1569057724 Qloss=-33962514, 7114 indices spanning [51217,58330] "gauss" 
[03:45:16.344] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 1, Qtot=374018808 Qloss=-40901309, 8569 indices spanning [58331,66899] "wiener" 
[03:45:16.580] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 1, Qtot=337259771 Qloss=-32036510, 6411 indices spanning [66900,73310] "gauss" 
[03:45:17.046] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 2, Qtot=360472365 Qloss=-31839779, 12099 indices spanning [73311,85409] "wiener" 
[03:45:17.485] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 save plane index: 2, Qtot=343558979 Qloss=-14455116, 9524 indices spanning [85410,94933] "gauss" 
[03:45:17.485] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 produce 94934 traces: 29660 wiener7, 0 decon_charge7, 23049 gauss7, frame tag: sigproc 
[03:45:17.485] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=30 output frame: frame: ident=3282 time=74 tick=512 with 94934 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":23049 [0] "mp2_roi7":34007 [0] "mp3_roi7":8218 [0] "wiener7":29660 [29660] ] cmm:[ ] 
[03:45:19.163] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[03:45:19.164] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=3282 time=74 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:45:19.164] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 input frame: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:45:19.164] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 init nticks=8064 tbinmin=0 tbinmax=8064 
[03:45:19.202] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 load plane index: 0, ntraces=1536, input bad regions: 0 
[03:45:21.336] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 load plane index: 1, ntraces=1536, input bad regions: 0 
[03:45:23.479] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 load plane index: 2, ntraces=1536, input bad regions: 0 
[03:45:41.710] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 0, Qtot=2455727316 Qloss=-62469839, 8301 indices spanning [41425,49725] "wiener" 
[03:45:41.960] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 0, Qtot=2420552954 Qloss=-53207934, 6550 indices spanning [49726,56275] "gauss" 
[03:45:42.677] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 1, Qtot=552662726 Qloss=-45210172, 9254 indices spanning [56276,65529] "wiener" 
[03:45:42.915] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 1, Qtot=470383695 Qloss=-37420795, 7344 indices spanning [65530,72873] "gauss" 
[03:45:43.375] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 2, Qtot=543584385 Qloss=-34409715, 12987 indices spanning [72874,85860] "wiener" 
[03:45:43.811] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 save plane index: 2, Qtot=525424367 Qloss=-15707483, 10067 indices spanning [85861,95927] "gauss" 
[03:45:43.811] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 produce 95928 traces: 30542 wiener6, 0 decon_charge6, 23961 gauss6, frame tag: sigproc 
[03:45:43.811] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=30 output frame: frame: ident=3282 time=74 tick=512 with 95928 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":23961 [0] "mp2_roi6":32969 [0] "mp3_roi6":8456 [0] "wiener6":30542 [30542] ] cmm:[ ] 
[03:45:46.202] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[03:45:46.203] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=3282 time=74 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:45:46.203] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 input frame: frame: ident=3282 time=74 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:45:46.203] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 init nticks=8064 tbinmin=0 tbinmax=8064 
[03:45:46.242] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 load plane index: 0, ntraces=1536, input bad regions: 0 
[03:45:48.345] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 load plane index: 1, ntraces=1536, input bad regions: 0 
[03:45:50.551] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=30 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                       7.098e-05      206.445       325.841       213.385       75.8803        16     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.1991e-05    8.02181e-05   0.000162417   7.1515e-05    2.78708e-05      16     
produce:tpcrawdecoder:PDVDTPCReader               45.1232       81.475        144.473       80.449        25.4238        16     
produce:triggerrawdecoder:PDVDTriggerReader4     0.534674      0.754254       1.44254       0.69868      0.219206        16     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000389216   0.00042462     0.0007132    0.000404025   7.53936e-05      16     
produce:ophit:OpHitFinder                       5.2374e-05    9.2968e-05    0.000510675   5.99635e-05   0.000108809      16     
produce:opflash:OpFlashFinderVerticalDrift      4.3302e-05    6.95733e-05   0.00030284    5.3803e-05    6.07226e-05      16     
produce:wclsdatavd:WireCellToolkit                60.7943       83.5993       139.889       76.2566       22.6772        15     
produce:gaushit:GausHitFinder                    0.970276       1.37541       2.23549       1.23608      0.349346        15     
produce:nhitsfilter:NumberOfHitsFilter          0.000293057   0.000421722   0.000566581   0.000402968   8.2692e-05       15     
produce:reco3d:SpacePointSolver                   7.03433       12.0187       21.5462       10.3566       4.14515        15     
produce:hitpdune:DisambigFromSpacePoints         0.117933      0.221524      0.494498      0.172956      0.103732        15     
produce:pandora:StandardPandora                   13.1245       34.187        131.337       23.9414        28.96         15     
produce:pandoraTrack:LArPandoraTrackCreation     0.553439       1.85613       7.1699        1.07712       1.63802        15     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0232108     0.0349919     0.052077      0.0345587    0.00809727       15     
[art]:TriggerResults:TriggerResultInserter      2.4475e-05    3.77243e-05   8.1185e-05    3.6031e-05    1.31539e-05      15     
end_path:out1:RootOutput                         3.467e-06    9.25527e-06   2.4763e-05     7.937e-06    4.85641e-06      15     
end_path:out1:RootOutput(write)                   3.94133       4.46409       5.61429       4.32065      0.416263        15     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.82 MB
  Peak resident set size usage (VmHWM): 6706.2 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 03:46:51 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 3282
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:26:24 UTC       justIN version: 01.05.01