justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 41012.0@dunegpschedd02.fnal.gov

Jobsub ID41012.0@dunegpschedd02.fnal.gov
Workflow ID2650
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-17 13:55:16
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_brug
Last heartbeat2025-09-17 14:48:00
From worker nodeHostnamewn-sate-044.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-17 13:56:13
Input filesvd-protodune:np02vd_raw_run039275_0395_df-s05-d3_dw_0_20250902T032900.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-17 14:48:00
Saved logsjustin-logs:41012.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

pHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 16:45:23 CEST  run: 39275 subRun: 1 event: 142377
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 16:45:23 CEST  run: 39275 subRun: 1 event: 142377
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 16:45:23 CEST  run: 39275 subRun: 1 event: 142377
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 16:45:23 CEST  run: 39275 subRun: 1 event: 142377
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 16:45:23 CEST  run: 39275 subRun: 1 event: 142377
Error! unrecognized channel number -1. Ignoring pulse
%MSG
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=10048 keeping as is
[16:45:23.342] D [  main  ] executing 1 apps, thread limit 0:
[16:45:23.342] D [  main  ] executing 1 apps, thread limit 0:
[16:45:23.342] D [  main  ] executing app: "Pgrapher"
[16:45:23.342] D [ pgraph ] <Pgrapher:> executing graph 
[16:45:23.342] D [ pgraph ] executing with 26 nodes
[16:45:23.344] D [  glue  ] <FrameFanout:nfsp> call=16: input: frame: ident=142377 time=39 tick=512 with 12288 traces.  frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=142377 time=39 tick=512 with 12288 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=142377 time=39 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=142377 time=39 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=142377 time=39 tick=512 with 12288 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=142377 time=39 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=142377 time=39 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=142377 time=39 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=142377 time=39 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]  
[16:45:23.345] W [  glue  ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped. 
[16:45:23.345] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=142377 time=39 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=142377 time=39 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[16:45:23.345] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 input frame: frame: ident=142377 time=39 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[16:45:23.345] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 init nticks=10048 tbinmin=0 tbinmax=10048 
[16:45:23.389] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 load plane index: 0, ntraces=1536, input bad regions: 0 
[16:45:26.637] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 load plane index: 1, ntraces=1536, input bad regions: 0 
[16:45:30.006] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 load plane index: 2, ntraces=1536, input bad regions: 0 
[16:45:40.427] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 save plane index: 0, Qtot=404106718 Qloss=-34829613, 8674 indices spanning [36004,44677] "wiener" 
[16:45:40.806] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 save plane index: 0, Qtot=375579260 Qloss=-29456806, 6757 indices spanning [44678,51434] "gauss" 
[16:45:41.670] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 save plane index: 1, Qtot=374661725 Qloss=-38527633, 8370 indices spanning [51435,59804] "wiener" 
[16:45:42.048] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 save plane index: 1, Qtot=335797217 Qloss=-30632128, 6614 indices spanning [59805,66418] "gauss" 
[16:45:42.678] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 save plane index: 2, Qtot=349432231 Qloss=-21308883, 8331 indices spanning [66419,74749] "wiener" 
[16:45:43.300] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 save plane index: 2, Qtot=338901815 Qloss=-10792262, 6808 indices spanning [74750,81557] "gauss" 
[16:45:43.300] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 produce 81558 traces: 25375 wiener7, 0 decon_charge7, 20179 gauss7, frame tag: sigproc 
[16:45:43.301] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 output frame: frame: ident=142377 time=39 tick=512 with 81558 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":20179 [0] "mp2_roi7":26970 [0] "mp3_roi7":9034 [0] "wiener7":25375 [25375] ] cmm:[ ] 
[16:45:43.966] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[16:45:43.967] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=142377 time=39 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=142377 time=39 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[16:45:43.967] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=16 input frame: frame: ident=142377 time=39 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[16:45:43.967] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=16 init nticks=10048 tbinmin=0 tbinmax=10048 
[16:45:44.004] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=16 load plane index: 0, ntraces=1536, input bad regions: 0 
[16:45:47.567] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=16 load plane index: 1, ntraces=1536, input bad regions: 0 
[16:45:51.381] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=16 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.8709e-05      311.551       391.121       345.648       113.494         9     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.2809e-05    7.9959e-05    0.000243858   5.9682e-05    5.82138e-05       9     
produce:tpcrawdecoder:PDVDTPCReader               88.1413       127.016       142.066       133.47        16.6569         9     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0344817     0.0375688     0.0485939     0.0345857    0.00531303        9     
produce:pdvddaphne:DAPHNEReaderPDVD               10.6745       16.464        21.6136       16.1135       3.12249         9     
produce:ophit:OpHitFinder                        0.0595295     0.0719752     0.0781761     0.0735524    0.00558428        9     
produce:opflash:OpFlashFinderVerticalDrift       0.0111242     0.0192541     0.0272182     0.0193803    0.00430445        9     
produce:wclsdatavd:WireCellToolkit                83.7411       91.8224       99.9552       92.4932       6.8087          8     
produce:gaushit:GausHitFinder                     1.69176       1.83694       2.04591       1.84253      0.116366         8     
produce:nhitsfilter:NumberOfHitsFilter          0.000375635   0.00044414    0.000713761   0.000409168   0.000103536       8     
produce:reco3d:SpacePointSolver                   15.2127       18.3732       21.7104       18.3188       2.53839         8     
produce:hitpdune:DisambigFromSpacePoints         0.216279      0.276336       0.39066      0.273405      0.0532215        8     
produce:pandora:StandardPandora                   49.0912       80.3994       110.083       76.6139       23.7605         8     
produce:pandoraTrack:LArPandoraTrackCreation     0.995991        1.411        1.84032       1.43591      0.314222         8     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0362728     0.0436036     0.0563686     0.0413918    0.00642839        8     
[art]:TriggerResults:TriggerResultInserter      1.8164e-05    2.53926e-05   6.6435e-05    1.88755e-05   1.56515e-05       8     
end_path:out1:RootOutput                         4.558e-06    1.1197e-05     3.204e-05    8.8465e-06    8.00856e-06       8     
end_path:out1:RootOutput(write)                   6.90179       7.10375       7.45042       7.04243      0.172409         8     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.86 MB
  Peak resident set size usage (VmHWM): 6715.28 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 16:47:38 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39275 subRun: 1 event: 142377
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-18 20:59:36 UTC       justIN version: 01.05.00