justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40696.1@dunegpschedd02.fnal.gov

Jobsub ID40696.1@dunegpschedd02.fnal.gov
Workflow ID2332
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 23:28:27
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_brug
Last heartbeat2025-09-17 00:01:35
From worker nodeHostnamewn-lot-023.farm.nikhef.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 23:29:21
Input filesvd-protodune:np02vd_raw_run039343_0032_df-s05-d0_dw_0_20250908T125034.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-17 00:01:35
Saved logsjustin-logs:40696.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Finder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 01:59:17 CEST  run: 39343 subRun: 1 event: 17114
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 01:59:17 CEST  run: 39343 subRun: 1 event: 17114
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 01:59:17 CEST  run: 39343 subRun: 1 event: 17114
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 01:59:17 CEST  run: 39343 subRun: 1 event: 17114
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 01:59:17 CEST  run: 39343 subRun: 1 event: 17114
Error! unrecognized channel number -1. Ignoring pulse
%MSG
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6400 keeping as is
[01:59:17.108] D [  main  ] executing 1 apps, thread limit 0:
[01:59:17.108] D [  main  ] executing 1 apps, thread limit 0:
[01:59:17.108] D [  main  ] executing app: "Pgrapher"
[01:59:17.108] D [ pgraph ] <Pgrapher:> executing graph 
[01:59:17.108] D [ pgraph ] executing with 26 nodes
[01:59:17.109] D [  glue  ] <FrameFanout:nfsp> call=12: input: frame: ident=17114 time=23 tick=512 with 12288 traces.  frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=17114 time=23 tick=512 with 12288 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=17114 time=23 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=17114 time=23 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=17114 time=23 tick=512 with 12288 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=17114 time=23 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=17114 time=23 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=17114 time=23 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=17114 time=23 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]  
[01:59:17.109] W [  glue  ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped. 
[01:59:17.110] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=17114 time=23 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=17114 time=23 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[01:59:17.110] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 input frame: frame: ident=17114 time=23 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[01:59:17.110] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 init nticks=6400 tbinmin=0 tbinmax=6400 
[01:59:17.136] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 load plane index: 0, ntraces=1536, input bad regions: 0 
[01:59:18.602] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 load plane index: 1, ntraces=1536, input bad regions: 0 
[01:59:20.275] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 load plane index: 2, ntraces=1536, input bad regions: 0 
[01:59:54.155] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 0, Qtot=92955401681 Qloss=-161459415045, 6471 indices spanning [17956,24426] "wiener" 
[01:59:54.328] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 0, Qtot=90291647114 Qloss=-162312642006, 5870 indices spanning [24427,30296] "gauss" 
[01:59:55.667] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 1, Qtot=46748559276 Qloss=-10642113547, 16879 indices spanning [30297,47175] "wiener" 
[01:59:55.843] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 1, Qtot=40172939869 Qloss=-10783702115, 16749 indices spanning [47176,63924] "gauss" 
[01:59:56.177] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 2, Qtot=2226220771 Qloss=-3563504624, 10672 indices spanning [63925,74596] "wiener" 
[01:59:56.525] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 2, Qtot=1762106568 Qloss=-3107261053, 12213 indices spanning [74597,86809] "gauss" 
[01:59:56.525] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 produce 86810 traces: 34022 wiener7, 0 decon_charge7, 34832 gauss7, frame tag: sigproc 
[01:59:56.525] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 output frame: frame: ident=17114 time=23 tick=512 with 86810 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":34832 [0] "mp2_roi7":12787 [0] "mp3_roi7":5169 [0] "wiener7":34022 [34022] ] cmm:[ ] 
[02:00:00.770] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[02:00:00.771] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=17114 time=23 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=17114 time=23 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:00:00.771] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 input frame: frame: ident=17114 time=23 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:00:00.771] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 init nticks=6400 tbinmin=0 tbinmax=6400 
[02:00:00.798] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 load plane index: 0, ntraces=1536, input bad regions: 0 
[02:00:02.200] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 load plane index: 1, ntraces=1536, input bad regions: 0 
[02:00:03.616] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      8.5551e-05      240.742       488.408       229.19        137.44          7     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.6114e-05    8.73997e-05   0.000160631   7.2115e-05    3.10002e-05       7     
produce:tpcrawdecoder:PDVDTPCReader               50.1632       72.9421       114.538       69.8867       18.4875         7     
produce:triggerrawdecoder:PDVDTriggerReader4      0.5189       0.542659      0.594475      0.539573      0.0246749        7     
produce:pdvddaphne:DAPHNEReaderPDVD               7.03184       9.34853       11.2405       9.46244       1.43149         7     
produce:ophit:OpHitFinder                        0.0414161     0.0445751     0.0485048     0.0451814     0.002349         7     
produce:opflash:OpFlashFinderVerticalDrift      0.00841419     0.0129075     0.017461      0.0139325    0.00300267        7     
produce:wclsdatavd:WireCellToolkit                47.5456       67.0146       100.033       52.7766       23.0117         6     
produce:gaushit:GausHitFinder                    0.948492       1.87642       5.16404       1.30133       1.47959         6     
produce:nhitsfilter:NumberOfHitsFilter          0.000269335   0.000702103   0.00135463    0.000656516   0.000345642       6     
produce:reco3d:SpacePointSolver                   10.489        15.8358       22.1883       15.6999       4.11224         6     
produce:hitpdune:DisambigFromSpacePoints         0.141094      0.224901      0.361239      0.222645      0.0715674        6     
produce:pandora:StandardPandora                   35.7041       106.01        278.941       82.2125       82.5055         6     
produce:pandoraTrack:LArPandoraTrackCreation     0.410536      0.775572       1.4386       0.737045       0.33059         6     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0244618     0.0322937     0.0428567     0.0302972    0.00594921        6     
[art]:TriggerResults:TriggerResultInserter      2.0388e-05    2.8011e-05    5.9422e-05    2.21165e-05   1.40805e-05       6     
end_path:out1:RootOutput                         4.518e-06    8.65933e-06    2.673e-05    5.1345e-06    8.09287e-06       6     
end_path:out1:RootOutput(write)                   4.00619       4.47682       6.03879       4.17016      0.716853         6     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.8 MB
  Peak resident set size usage (VmHWM): 6706.48 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 02:01:17 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39343 subRun: 1 event: 17114
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-18 19:26:30 UTC       justIN version: 01.05.00