justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242100.0@dunegpschedd01.fnal.gov

Jobsub ID242100.0@dunegpschedd01.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-02 22:02:03
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-11-02 22:24:40
From worker nodeHostnamewn-sate-043.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-02 22:02:16
Input filesvd-protodune:np02vd_raw_run040266_0548_df-s04-d2_dw_0_20251025T123423.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-02 22:24:40
Saved logsjustin-logs:242100.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

XVs -> 4 XUVs
C:0 T:3 843 XUs and 1157 XVs -> 44 XUVs
C:0 T:4 2749 XUs and 2257 XVs -> 119 XUVs
C:0 T:5 1864 XUs and 1020 XVs -> 130 XUVs
C:0 T:6 20157 XUs and 14973 XVs -> 1567 XUVs
C:0 T:7 1162 XUs and 1366 XVs -> 68 XUVs
C:0 T:8 243 XUs and 169 XVs -> 10 XUVs
C:0 T:9 207 XUs and 236 XVs -> 11 XUVs
C:0 T:10 519 XUs and 381 XVs -> 29 XUVs
C:0 T:11 321 XUs and 261 XVs -> 21 XUVs
C:0 T:12 766 XUs and 1267 XVs -> 55 XUVs
C:0 T:13 1106 XUs and 1117 XVs -> 54 XUVs
C:0 T:14 72 XUs and 108 XVs -> 7 XUVs
C:0 T:15 414 XUs and 906 XVs -> 42 XUVs
2196 XUVs total
1117 collection wire objects
2196 potential space points
Neighbour search...
110508 tests to find 28906 neighbours
Iterating with no regularization...
Begin: 3.59194e+09
0 3.00182e+09
1 2.92815e+09
2 2.92477e+09
3 2.92417e+09
Now with regularization...
Begin: 2.88394e+09
0 2.88358e+09
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=8704 keeping as is
[23:22:16.167] D [  main  ] executing 1 apps, thread limit 0:
[23:22:16.167] D [  main  ] executing 1 apps, thread limit 0:
[23:22:16.167] D [  main  ] executing app: "Pgrapher"
[23:22:16.167] D [ pgraph ] <Pgrapher:> executing graph 
[23:22:16.167] D [ pgraph ] executing with 26 nodes
[23:22:16.168] D [  glue  ] <FrameFanout:nfsp> call=12: input: frame: ident=101222 time=27 tick=512 with 12288 traces.  frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=101222 time=27 tick=512 with 12288 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=101222 time=27 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=101222 time=27 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=101222 time=27 tick=512 with 12288 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=101222 time=27 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=101222 time=27 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=101222 time=27 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=101222 time=27 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]  
[23:22:16.168] W [  glue  ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped. 
[23:22:16.169] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=101222 time=27 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=101222 time=27 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[23:22:16.169] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 input frame: frame: ident=101222 time=27 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[23:22:16.169] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 init nticks=8704 tbinmin=0 tbinmax=8704 
[23:22:16.198] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 load plane index: 0, ntraces=1536, input bad regions: 0 
[23:22:18.601] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 load plane index: 1, ntraces=1536, input bad regions: 0 
[23:22:21.018] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 load plane index: 2, ntraces=1536, input bad regions: 0 
[23:22:53.936] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 0, Qtot=967078869 Qloss=-78414303, 15608 indices spanning [40479,56086] "wiener" 
[23:22:54.188] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 0, Qtot=860202632 Qloss=-62473433, 12497 indices spanning [56087,68583] "gauss" 
[23:22:55.193] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 1, Qtot=880101365 Qloss=-69660823, 17052 indices spanning [68584,85635] "wiener" 
[23:22:55.445] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 1, Qtot=765898830 Qloss=-50454247, 13208 indices spanning [85636,98843] "gauss" 
[23:22:55.872] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 2, Qtot=955300735 Qloss=-58951724, 17439 indices spanning [98844,116282] "wiener" 
[23:22:56.279] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 2, Qtot=923349889 Qloss=-27122683, 13531 indices spanning [116283,129813] "gauss" 
[23:22:56.279] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 produce 129814 traces: 50099 wiener7, 0 decon_charge7, 39236 gauss7, frame tag: sigproc 
[23:22:56.279] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 output frame: frame: ident=101222 time=27 tick=512 with 129814 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":39236 [0] "mp2_roi7":25869 [0] "mp3_roi7":14610 [0] "wiener7":50099 [50099] ] cmm:[ ] 
[23:23:00.004] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[23:23:00.005] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=101222 time=27 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=101222 time=27 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[23:23:00.005] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 input frame: frame: ident=101222 time=27 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[23:23:00.005] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 init nticks=8704 tbinmin=0 tbinmax=8704 
[23:23:00.032] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 load plane index: 0, ntraces=1536, input bad regions: 0 
[23:23:02.371] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 load plane index: 1, ntraces=1536, input bad regions: 0 
[23:23:04.743] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      8.5741e-05      154.368       228.031       165.325       67.4398         7     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.1535e-05    9.35156e-05   0.000209363   8.0591e-05    4.81067e-05       7     
produce:tpcrawdecoder:PDVDTPCReader               50.4464       67.1505       78.0995       69.3263       9.25701         7     
produce:triggerrawdecoder:PDVDTriggerReader4      0.53659      0.824029       2.01348      0.650104      0.488564         7     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000397055   0.000457051   0.000773572   0.000402827   0.000129402       7     
produce:ophit:OpHitFinder                        6.371e-05    0.000151827   0.000609706   7.6944e-05    0.000187019       7     
produce:opflash:OpFlashFinderVerticalDrift      5.6446e-05    9.8864e-05    0.000337363   5.7277e-05    9.74071e-05       7     
produce:wclsdatavd:WireCellToolkit                53.7086       65.6481       99.0401       60.3209       15.7356         6     
produce:gaushit:GausHitFinder                    0.803418       1.22472       1.56806       1.22044      0.264066         6     
produce:nhitsfilter:NumberOfHitsFilter          0.000254097   0.000320651   0.000525016   0.000287795   9.30656e-05       6     
produce:reco3d:SpacePointSolver                   7.55559       12.5923       17.809        13.0731       3.4627          6     
produce:hitpdune:DisambigFromSpacePoints         0.104108      0.181168      0.274422      0.181607      0.0569456        6     
produce:pandora:StandardPandora                   15.7361       27.7887       44.1304       25.9494       8.95229         6     
produce:pandoraTrack:LArPandoraTrackCreation      0.77141       1.33978       2.39258       1.23092       0.5075          6     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0223042     0.0367684     0.0555779     0.0379704     0.0105917        6     
[art]:TriggerResults:TriggerResultInserter      2.0368e-05    3.84457e-05   8.0902e-05    3.0312e-05    2.11403e-05       6     
end_path:out1:RootOutput                         6.442e-06    1.06098e-05   2.5388e-05     7.479e-06    6.71783e-06       6     
end_path:out1:RootOutput(write)                   4.48963       4.8541        5.43316       4.87168      0.300543         6     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.92 MB
  Peak resident set size usage (VmHWM): 6706.23 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 02-Nov-2025 23:24:18 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 101222
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-04 01:28:12 UTC       justIN version: 01.05.01