justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40866.1@dunegpschedd02.fnal.gov

Jobsub ID40866.1@dunegpschedd02.fnal.gov
Workflow ID2576
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-17 07:42:55
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-09-17 08:00:49
From worker nodeHostnamewn-lot-032.farm.nikhef.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-17 07:43:46
Input filesvd-protodune:np02vd_raw_run039275_0094_df-s03-d3_dw_0_20250901T181110.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-17 08:00:49
Saved logsjustin-logs:40866.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 09:58:41 CEST  run: 39275 subRun: 1 event: 33929
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 09:58:41 CEST  run: 39275 subRun: 1 event: 33929
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 09:58:41 CEST  run: 39275 subRun: 1 event: 33929
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 09:58:41 CEST  run: 39275 subRun: 1 event: 33929
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 09:58:41 CEST  run: 39275 subRun: 1 event: 33929
Error! unrecognized channel number -1. Ignoring pulse
%MSG
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=10000 keeping as is
[09:58:42.084] D [  main  ] executing 1 apps, thread limit 0:
[09:58:42.085] D [  main  ] executing 1 apps, thread limit 0:
[09:58:42.085] D [  main  ] executing app: "Pgrapher"
[09:58:42.085] D [ pgraph ] <Pgrapher:> executing graph 
[09:58:42.085] D [ pgraph ] executing with 26 nodes
[09:58:42.085] D [  glue  ] <FrameFanout:nfsp> call=8: input: frame: ident=33929 time=23 tick=512 with 12288 traces.  frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=33929 time=23 tick=512 with 12288 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=33929 time=23 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=33929 time=23 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=33929 time=23 tick=512 with 12288 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=33929 time=23 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=33929 time=23 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=33929 time=23 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=33929 time=23 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]  
[09:58:42.086] W [  glue  ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped. 
[09:58:42.086] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=33929 time=23 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=33929 time=23 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:58:42.086] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 input frame: frame: ident=33929 time=23 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:58:42.086] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 init nticks=10000 tbinmin=0 tbinmax=10000 
[09:58:42.117] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 load plane index: 0, ntraces=1536, input bad regions: 0 
[09:58:43.759] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 load plane index: 1, ntraces=1536, input bad regions: 0 
[09:58:45.644] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 load plane index: 2, ntraces=1536, input bad regions: 0 
[09:59:26.705] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 save plane index: 0, Qtot=99367238328 Qloss=-1913033995, 7431 indices spanning [50007,57437] "wiener" 
[09:59:26.909] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 save plane index: 0, Qtot=99266646181 Qloss=-1924521084, 5980 indices spanning [57438,63417] "gauss" 
[09:59:27.927] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 save plane index: 1, Qtot=2181966003 Qloss=-328915940, 17624 indices spanning [63418,81041] "wiener" 
[09:59:28.141] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 save plane index: 1, Qtot=1985145432 Qloss=-320987379, 14143 indices spanning [81042,95184] "gauss" 
[09:59:28.586] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 save plane index: 2, Qtot=1422127204 Qloss=-37860298, 13697 indices spanning [95185,108881] "wiener" 
[09:59:29.019] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 save plane index: 2, Qtot=1393566003 Qloss=-17321315, 10762 indices spanning [108882,119643] "gauss" 
[09:59:29.020] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 produce 119644 traces: 38752 wiener7, 0 decon_charge7, 30885 gauss7, frame tag: sigproc 
[09:59:29.020] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=8 output frame: frame: ident=33929 time=23 tick=512 with 119644 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":30885 [0] "mp2_roi7":34688 [0] "mp3_roi7":15319 [0] "wiener7":38752 [38752] ] cmm:[ ] 
[09:59:35.114] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[09:59:35.114] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=33929 time=23 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=33929 time=23 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:59:35.115] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 input frame: frame: ident=33929 time=23 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:59:35.115] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 init nticks=10000 tbinmin=0 tbinmax=10000 
[09:59:35.146] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 load plane index: 0, ntraces=1536, input bad regions: 0 
[09:59:37.279] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 load plane index: 1, ntraces=1536, input bad regions: 0 
[09:59:39.360] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=8 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      7.3318e-05      167.925       254.911       194.543       87.5412         5     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.0773e-05    0.000100444   0.000208711    7.461e-05    5.41524e-05       5     
produce:tpcrawdecoder:PDVDTPCReader               17.1417       17.4405       17.9369       17.2808      0.298703         5     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0347269     0.0349927     0.0358709     0.0347675    0.000440869       5     
produce:pdvddaphne:DAPHNEReaderPDVD               4.58654       4.65068       4.75178       4.62296      0.0575449        5     
produce:ophit:OpHitFinder                        0.0432885     0.0506111     0.0571876     0.0511879    0.00585016        5     
produce:opflash:OpFlashFinderVerticalDrift      0.00979884     0.0140776     0.0244727     0.0110245    0.00547199        5     
produce:wclsdatavd:WireCellToolkit                71.0175       81.5479       89.7446       82.7147       7.02014         4     
produce:gaushit:GausHitFinder                     1.46377       1.62229       1.96635       1.52952      0.201297         4     
produce:nhitsfilter:NumberOfHitsFilter          0.000439453   0.000552775   0.000684993   0.000543328   8.74292e-05       4     
produce:reco3d:SpacePointSolver                   16.593        20.2603       26.2699       19.0892       3.77858         4     
produce:hitpdune:DisambigFromSpacePoints         0.244766      0.317035      0.486876      0.268248      0.0986603        4     
produce:pandora:StandardPandora                   62.6483       76.6432       106.541       68.6916       17.5454         4     
produce:pandoraTrack:LArPandoraTrackCreation      1.02656       1.33508       1.7227        1.29553      0.265349         4     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0335679     0.0399509     0.0448179     0.0407089    0.00467577        4     
[art]:TriggerResults:TriggerResultInserter      1.8594e-05    2.80673e-05   5.4712e-05    1.94815e-05   1.5389e-05        4     
end_path:out1:RootOutput                         4.438e-06    8.46075e-06   1.9957e-05     4.724e-06    6.63841e-06       4     
end_path:out1:RootOutput(write)                   5.70541       5.81764       5.96342       5.80086      0.0927674        4     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.84 MB
  Peak resident set size usage (VmHWM): 6708.5 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 10:00:30 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39275 subRun: 1 event: 33929
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-19 03:35:55 UTC       justIN version: 01.05.00