justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241771.4@dunegpschedd01.fnal.gov

Jobsub ID241771.4@dunegpschedd01.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-01 20:58:41
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-11-01 21:53:42
From worker nodeHostnamewn-sate-040.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-01 20:59:24
Input filesvd-protodune:np02vd_raw_run040266_0350_df-s03-d0_dw_0_20251025T065918.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-01 21:53:42
Saved logsjustin-logs:241771.4-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ctor:chsel7> input frame: ident=64664 time=58 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=64664 time=58 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:50:36.542] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 input frame: frame: ident=64664 time=58 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:50:36.542] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 init nticks=9432 tbinmin=0 tbinmax=9432 
[22:50:36.577] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:50:39.737] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:50:42.945] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 2, ntraces=1536, input bad regions: 0 
[22:51:27.130] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 0, Qtot=7687049354 Qloss=-95095517, 10623 indices spanning [35974,46596] "wiener" 
[22:51:27.536] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 0, Qtot=7538426578 Qloss=-85662006, 8485 indices spanning [46597,55081] "gauss" 
[22:51:28.621] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 1, Qtot=1720006802 Qloss=-106271951, 13851 indices spanning [55082,68932] "wiener" 
[22:51:28.999] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 1, Qtot=1476236206 Qloss=-100066431, 11307 indices spanning [68933,80239] "gauss" 
[22:51:29.641] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 2, Qtot=1826823410 Qloss=-42180004, 12362 indices spanning [80240,92601] "wiener" 
[22:51:30.264] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 2, Qtot=1804806797 Qloss=-19219733, 9337 indices spanning [92602,101938] "gauss" 
[22:51:30.264] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 produce 101939 traces: 36836 wiener7, 0 decon_charge7, 29129 gauss7, frame tag: sigproc 
[22:51:30.264] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 output frame: frame: ident=64664 time=58 tick=512 with 101939 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":29129 [0] "mp2_roi7":24697 [0] "mp3_roi7":11277 [0] "wiener7":36836 [36836] ] cmm:[ ] 
[22:51:35.100] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[22:51:35.100] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=64664 time=58 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=64664 time=58 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:51:35.101] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 input frame: frame: ident=64664 time=58 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:51:35.101] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 init nticks=9432 tbinmin=0 tbinmax=9432 
[22:51:35.137] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:51:38.448] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:51:41.869] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 2, ntraces=1536, input bad regions: 0 
[22:52:08.937] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 0, Qtot=5600869486 Qloss=-81446037, 9421 indices spanning [38626,48046] "wiener" 
[22:52:09.336] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 0, Qtot=5514248651 Qloss=-71069363, 7573 indices spanning [48047,55619] "gauss" 
[22:52:10.355] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 1, Qtot=1352825880 Qloss=-119592638, 12177 indices spanning [55620,67796] "wiener" 
[22:52:10.734] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 1, Qtot=1144774170 Qloss=-117681709, 9988 indices spanning [67797,77784] "gauss" 
[22:52:11.366] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 2, Qtot=1520183861 Qloss=-26304807, 9685 indices spanning [77785,87469] "wiener" 
[22:52:11.982] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 save plane index: 2, Qtot=1504535773 Qloss=-10598539, 7517 indices spanning [87470,94986] "gauss" 
[22:52:11.982] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 produce 94987 traces: 31283 wiener6, 0 decon_charge6, 25078 gauss6, frame tag: sigproc 
[22:52:11.982] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 output frame: frame: ident=64664 time=58 tick=512 with 94987 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":25078 [0] "mp2_roi6":28053 [0] "mp3_roi6":10573 [0] "wiener6":31283 [31283] ] cmm:[ ] 
[22:52:14.912] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[22:52:14.913] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=64664 time=58 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=64664 time=58 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:52:14.913] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 input frame: frame: ident=64664 time=58 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:52:14.913] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 init nticks=9432 tbinmin=0 tbinmax=9432 
[22:52:14.948] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:52:18.247] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:52:21.867] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=24 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.7186e-05      227.597       628.755        187.4        135.472        13     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.0403e-05    8.03599e-05   0.000209744   6.7476e-05    3.78747e-05      13     
produce:tpcrawdecoder:PDVDTPCReader               52.0331       77.1891       103.148       75.1262       16.3725        13     
produce:triggerrawdecoder:PDVDTriggerReader4     0.541403      0.775944       1.20412      0.660685      0.218484        13     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000366457   0.00040773    0.000752942   0.000378821   0.00010008       13     
produce:ophit:OpHitFinder                       6.2949e-05    0.000110106   0.000553649   7.0863e-05    0.000128301      13     
produce:opflash:OpFlashFinderVerticalDrift      4.9673e-05    7.61683e-05   0.000333485   5.4061e-05    7.43603e-05      13     
produce:wclsdatavd:WireCellToolkit                56.4975       86.3342       145.93        67.7876       28.1739        12     
produce:gaushit:GausHitFinder                    0.947155       1.62645       3.44678       1.3971        0.77425        12     
produce:nhitsfilter:NumberOfHitsFilter          0.000242876   0.000333949   0.000566513   0.000304987   9.05479e-05      12     
produce:reco3d:SpacePointSolver                   8.45302       14.4584       34.0649       13.2153       6.53174        12     
produce:hitpdune:DisambigFromSpacePoints          0.11404      0.228014      0.665349      0.207865      0.138672        12     
produce:pandora:StandardPandora                   13.8769       58.4076       329.335       31.7631       82.7619        12     
produce:pandoraTrack:LArPandoraTrackCreation      0.66879       1.62062       4.73024       1.30608       1.00345        12     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0292142     0.0400528     0.0646504     0.0360858     0.0095155       12     
[art]:TriggerResults:TriggerResultInserter      2.6129e-05    4.03532e-05   0.000103774   3.3443e-05    2.0413e-05       12     
end_path:out1:RootOutput                         6.772e-06    9.56467e-06   2.7491e-05    7.6395e-06    5.55739e-06      12     
end_path:out1:RootOutput(write)                   4.60943       5.16476       6.28051       4.99812      0.525428        12     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.91 MB
  Peak resident set size usage (VmHWM): 6702.43 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 01-Nov-2025 22:53:20 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 64664
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-04 01:29:57 UTC       justIN version: 01.05.01