justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241335.15@dunegpschedd01.fnal.gov

Jobsub ID241335.15@dunegpschedd01.fnal.gov
Workflow ID9405
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 07:50:43
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-10-31 09:17:11
From worker nodeHostnamewn-pep-012.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 07:54:41
Input filesvd-protodune:np02vd_raw_run040270_0780_df-s03-d1_dw_0_20251028T103337.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 09:17:11
Saved logsjustin-logs:241335.15-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

<ChannelSelector:chsel7> input frame: ident=144345 time=49 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=144345 time=49 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[10:14:53.287] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 input frame: frame: ident=144345 time=49 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[10:14:53.287] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 init nticks=8640 tbinmin=0 tbinmax=8640 
[10:14:53.330] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 0, ntraces=1536, input bad regions: 0 
[10:14:55.456] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 1, ntraces=1536, input bad regions: 0 
[10:14:57.608] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 2, ntraces=1536, input bad regions: 0 
[10:15:06.551] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 0, Qtot=703573601 Qloss=-73696857, 10578 indices spanning [47903,58480] "wiener" 
[10:15:06.760] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 0, Qtot=669030269 Qloss=-59440788, 7905 indices spanning [58481,66385] "gauss" 
[10:15:07.408] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 1, Qtot=399106212 Qloss=-41243287, 8732 indices spanning [66386,75117] "wiener" 
[10:15:07.660] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 1, Qtot=342636233 Qloss=-30325620, 6563 indices spanning [75118,81680] "gauss" 
[10:15:08.149] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 2, Qtot=318342752 Qloss=-27709193, 9391 indices spanning [81681,91071] "wiener" 
[10:15:08.592] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 2, Qtot=302975224 Qloss=-12155790, 7515 indices spanning [91072,98586] "gauss" 
[10:15:08.592] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 produce 98587 traces: 28701 wiener7, 0 decon_charge7, 21983 gauss7, frame tag: sigproc 
[10:15:08.592] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 output frame: frame: ident=144345 time=49 tick=512 with 98587 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":21983 [0] "mp2_roi7":36849 [0] "mp3_roi7":11054 [0] "wiener7":28701 [28701] ] cmm:[ ] 
[10:15:09.317] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[10:15:09.318] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=144345 time=49 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=144345 time=49 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[10:15:09.318] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 input frame: frame: ident=144345 time=49 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[10:15:09.318] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 init nticks=8640 tbinmin=0 tbinmax=8640 
[10:15:09.353] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 0, ntraces=1536, input bad regions: 0 
[10:15:11.526] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 1, ntraces=1536, input bad regions: 0 
[10:15:13.699] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 2, ntraces=1536, input bad regions: 0 
[10:15:30.406] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 0, Qtot=1035549782 Qloss=-40529162, 7754 indices spanning [38269,46022] "wiener" 
[10:15:30.645] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 0, Qtot=997918956 Qloss=-30587836, 5870 indices spanning [46023,51892] "gauss" 
[10:15:31.297] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 1, Qtot=472243204 Qloss=-38954648, 9138 indices spanning [51893,61030] "wiener" 
[10:15:31.536] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 1, Qtot=411698248 Qloss=-28055499, 6921 indices spanning [61031,67951] "gauss" 
[10:15:31.997] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 2, Qtot=365755891 Qloss=-31178349, 9812 indices spanning [67952,77763] "wiener" 
[10:15:32.428] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 2, Qtot=348884702 Qloss=-14198211, 7732 indices spanning [77764,85495] "gauss" 
[10:15:32.428] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 produce 85496 traces: 26704 wiener6, 0 decon_charge6, 20523 gauss6, frame tag: sigproc 
[10:15:32.429] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 output frame: frame: ident=144345 time=49 tick=512 with 85496 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":20523 [0] "mp2_roi6":29673 [0] "mp3_roi6":8596 [0] "wiener6":26704 [26704] ] cmm:[ ] 
[10:15:34.175] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[10:15:34.176] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=144345 time=49 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=144345 time=49 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[10:15:34.176] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 input frame: frame: ident=144345 time=49 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[10:15:34.176] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 init nticks=8640 tbinmin=0 tbinmax=8640 
[10:15:34.215] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 load plane index: 0, ntraces=1536, input bad regions: 0 
[10:15:36.310] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 load plane index: 1, ntraces=1536, input bad regions: 0 
[10:15:38.488] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      8.5112e-05      365.38        2658.58       133.726       683.346        13     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.231e-05    8.25315e-05   0.000160501   7.3912e-05    2.59605e-05      13     
produce:tpcrawdecoder:PDVDTPCReader               12.405        21.1206       33.1927       21.1322       5.59846        13     
produce:triggerrawdecoder:PDVDTriggerReader4     0.288954      0.312989      0.482483       0.29755      0.0496945       13     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000372962   0.000410574   0.00069092    0.000383736   8.18039e-05      13     
produce:ophit:OpHitFinder                       5.6441e-05    9.61312e-05   0.000447445   6.3571e-05    0.000101887      13     
produce:opflash:OpFlashFinderVerticalDrift      4.3564e-05    6.72351e-05   0.000268731   4.8413e-05    5.85284e-05      13     
produce:wclsdatavd:WireCellToolkit                57.7238       84.0309       143.986       67.9794        28.98         12     
produce:gaushit:GausHitFinder                    0.862004       1.36748       2.19507       1.33826       0.38637        12     
produce:nhitsfilter:NumberOfHitsFilter          0.000298569   0.000408472   0.000547527   0.00040144    7.33543e-05      12     
produce:reco3d:SpacePointSolver                   6.68485        11.65        22.5851       10.8775       4.65863        12     
produce:hitpdune:DisambigFromSpacePoints         0.114241      0.224259      0.571807      0.176296       0.1339         12     
produce:pandora:StandardPandora                   16.0155       271.494       2457.45       26.3053       675.396        12     
produce:pandoraTrack:LArPandoraTrackCreation      0.54272       1.56923       5.27329       1.08026       1.26626        12     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0208709     0.0324947     0.0589104     0.0330496    0.00955027       12     
[art]:TriggerResults:TriggerResultInserter      2.8631e-05    3.90051e-05   7.7741e-05    3.5346e-05    1.26705e-05      12     
end_path:out1:RootOutput                         7.042e-06    9.25733e-06   2.5497e-05    7.7665e-06    4.92667e-06      12     
end_path:out1:RootOutput(write)                   3.90722       4.35841       5.50403       4.1884       0.464379        12     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.8 MB
  Peak resident set size usage (VmHWM): 6707.85 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 10:16:48 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40270 subRun: 1 event: 144345
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:24:45 UTC       justIN version: 01.05.01