justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 235645.0@dunegpschedd02.fnal.gov

Jobsub ID235645.0@dunegpschedd02.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 03:06:27
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-10-31 04:09:54
From worker nodeHostnamewn-sate-053.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 03:07:42
Input filesvd-protodune:np02vd_raw_run040266_0287_df-s03-d3_dw_0_20251025T051201.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 04:09:54
Saved logsjustin-logs:235645.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[  glue  ] <ChannelSelector:chsel7> input frame: ident=53027 time=91 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=53027 time=91 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:07:08.976] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 input frame: frame: ident=53027 time=91 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:07:08.976] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 init nticks=8000 tbinmin=0 tbinmax=8000 
[05:07:09.004] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:07:10.659] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:07:12.390] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 load plane index: 2, ntraces=1536, input bad regions: 0 
[05:07:30.320] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 0, Qtot=17047663910 Qloss=-1036484376, 4216 indices spanning [11554,15769] "wiener" 
[05:07:30.523] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 0, Qtot=16496109352 Qloss=-1040404693, 3613 indices spanning [15770,19382] "gauss" 
[05:07:31.047] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 1, Qtot=16943985357 Qloss=-1497500364, 3672 indices spanning [19383,23054] "wiener" 
[05:07:31.255] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 1, Qtot=16364625640 Qloss=-1566989653, 3166 indices spanning [23055,26220] "gauss" 
[05:07:31.639] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 2, Qtot=1668246236 Qloss=-1972817067, 9794 indices spanning [26221,36014] "wiener" 
[05:07:32.035] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 2, Qtot=1297518723 Qloss=-1602718913, 11121 indices spanning [36015,47135] "gauss" 
[05:07:32.035] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 produce 47136 traces: 17682 wiener7, 0 decon_charge7, 17900 gauss7, frame tag: sigproc 
[05:07:32.035] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 output frame: frame: ident=53027 time=91 tick=512 with 47136 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":17900 [0] "mp2_roi7":8978 [0] "mp3_roi7":2576 [0] "wiener7":17682 [17682] ] cmm:[ ] 
[05:07:34.122] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[05:07:34.123] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=53027 time=91 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=53027 time=91 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:07:34.123] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 input frame: frame: ident=53027 time=91 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:07:34.123] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 init nticks=8000 tbinmin=0 tbinmax=8000 
[05:07:34.151] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:07:35.881] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:07:37.693] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 load plane index: 2, ntraces=1536, input bad regions: 0 
[05:07:40.605] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 0, Qtot=73427209 Qloss=-14135621, 1423 indices spanning [7475,8897] "wiener" 
[05:07:40.812] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 0, Qtot=71315959 Qloss=-12468977, 1151 indices spanning [8898,10048] "gauss" 
[05:07:41.168] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 1, Qtot=75357021 Qloss=-11969639, 2376 indices spanning [10049,12424] "wiener" 
[05:07:41.375] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 1, Qtot=69928268 Qloss=-9900914, 2017 indices spanning [12425,14441] "gauss" 
[05:07:41.746] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 2, Qtot=66179436 Qloss=-4546122, 2675 indices spanning [14442,17116] "wiener" 
[05:07:42.114] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 2, Qtot=63847560 Qloss=-2181090, 2222 indices spanning [17117,19338] "gauss" 
[05:07:42.114] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 produce 19339 traces: 6474 wiener6, 0 decon_charge6, 5390 gauss6, frame tag: sigproc 
[05:07:42.114] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 output frame: frame: ident=53027 time=91 tick=512 with 19339 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":5390 [0] "mp2_roi6":6177 [0] "mp3_roi6":1298 [0] "wiener6":6474 [6474] ] cmm:[ ] 
[05:07:42.130] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[05:07:42.131] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=53027 time=91 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=53027 time=91 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:07:42.131] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 input frame: frame: ident=53027 time=91 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[05:07:42.131] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 init nticks=8000 tbinmin=0 tbinmax=8000 
[05:07:42.158] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 load plane index: 0, ntraces=1536, input bad regions: 0 
[05:07:44.020] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 load plane index: 1, ntraces=1536, input bad regions: 0 
[05:07:45.849] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.8349e-05      165.696       263.609       159.542       53.3591        21     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.85e-05     8.30183e-05   0.000193274   7.3027e-05    2.87426e-05      21     
produce:tpcrawdecoder:PDVDTPCReader               13.0588       27.6988       83.9727       24.0495       14.4416        21     
produce:triggerrawdecoder:PDVDTriggerReader4     0.520946      0.548519      0.579826       0.54948      0.0165616       21     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000359507   0.000418472   0.000748139   0.000389643   8.20192e-05      21     
produce:ophit:OpHitFinder                       5.7578e-05    0.000104879   0.000558071   6.7147e-05    0.000110624      21     
produce:opflash:OpFlashFinderVerticalDrift      5.1827e-05    7.38552e-05   0.000374775   5.6356e-05    6.78974e-05      21     
produce:wclsdatavd:WireCellToolkit                52.3821       85.9674       135.376       85.9851       21.2425        20     
produce:gaushit:GausHitFinder                    0.730214       1.41842       2.06814       1.43887      0.309976        20     
produce:nhitsfilter:NumberOfHitsFilter          0.000255852   0.000364704   0.000595091    0.0003346    9.21554e-05      20     
produce:reco3d:SpacePointSolver                   5.54215       14.4985       21.1457       13.7434       4.16721        20     
produce:hitpdune:DisambigFromSpacePoints         0.0875565      0.22088      0.391271      0.225271      0.0796338       20     
produce:pandora:StandardPandora                   13.9359       37.3913       73.4846       35.1703       14.7525        20     
produce:pandoraTrack:LArPandoraTrackCreation     0.519098       1.60929       3.5579        1.54517      0.677174        20     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0236806     0.0424195     0.0627559     0.0392889     0.0111218       20     
[art]:TriggerResults:TriggerResultInserter      2.1942e-05    4.31388e-05   9.3336e-05    3.69095e-05   1.88995e-05      20     
end_path:out1:RootOutput                         4.087e-06    1.88043e-05   0.000178035    8.736e-06    3.68688e-05      20     
end_path:out1:RootOutput(write)                   4.65357       5.20693       6.25364       5.10742      0.411387        20     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.81 MB
  Peak resident set size usage (VmHWM): 6291.7 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 05:09:23 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 53027
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:23:09 UTC       justIN version: 01.05.01