justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40580.0@dunegpschedd02.fnal.gov

Jobsub ID40580.0@dunegpschedd02.fnal.gov
Workflow ID2656
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 19:36:15
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-09-16 21:33:17
From worker nodeHostnamewn-sate-036.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 19:36:56
Input filesvd-protodune:np02vd_raw_run039275_0471_df-s03-d0_dw_0_20250902T062014.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-16 21:33:17
Saved logsjustin-logs:40580.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

or:chsel6> input frame: ident=169906 time=143 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=169906 time=143 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[23:31:28.209] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 input frame: frame: ident=169906 time=143 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[23:31:28.209] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 init nticks=10000 tbinmin=0 tbinmax=10000 
[23:31:28.248] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[23:31:30.398] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[23:31:32.653] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 
[23:31:38.088] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 0, Qtot=158171228 Qloss=-20582702, 4367 indices spanning [17988,22354] "wiener" 
[23:31:38.299] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 0, Qtot=149391548 Qloss=-15536810, 3523 indices spanning [22355,25877] "gauss" 
[23:31:38.788] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 1, Qtot=141784869 Qloss=-21916595, 4600 indices spanning [25878,30477] "wiener" 
[23:31:39.013] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 1, Qtot=129581651 Qloss=-16535415, 3516 indices spanning [30478,33993] "gauss" 
[23:31:39.393] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 2, Qtot=131394340 Qloss=-11570365, 4805 indices spanning [33994,38798] "wiener" 
[23:31:39.769] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 save plane index: 2, Qtot=125951015 Qloss=-5743162, 4049 indices spanning [38799,42847] "gauss" 
[23:31:39.769] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 produce 42848 traces: 13772 wiener6, 0 decon_charge6, 11088 gauss6, frame tag: sigproc 
[23:31:39.769] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=34 output frame: frame: ident=169906 time=143 tick=512 with 42848 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":11088 [0] "mp2_roi6":14059 [0] "mp3_roi6":3929 [0] "wiener6":13772 [13772] ] cmm:[ ] 
[23:31:39.871] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[23:31:39.872] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=169906 time=143 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=169906 time=143 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[23:31:39.872] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 input frame: frame: ident=169906 time=143 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[23:31:39.872] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 init nticks=10000 tbinmin=0 tbinmax=10000 
[23:31:39.909] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[23:31:42.049] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[23:31:44.282] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 
[23:31:50.851] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 save plane index: 0, Qtot=267191245 Qloss=-21633874, 5645 indices spanning [26394,32038] "wiener" 
[23:31:51.063] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 save plane index: 0, Qtot=250232912 Qloss=-16050599, 4487 indices spanning [32039,36525] "gauss" 
[23:31:51.614] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 save plane index: 1, Qtot=170847581 Qloss=-22205095, 5927 indices spanning [36526,42452] "wiener" 
[23:31:51.841] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 save plane index: 1, Qtot=154369524 Qloss=-14328353, 4447 indices spanning [42453,46899] "gauss" 
[23:31:52.225] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 save plane index: 2, Qtot=142099371 Qloss=-15818897, 4933 indices spanning [46900,51832] "wiener" 
[23:31:52.602] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 save plane index: 2, Qtot=134060563 Qloss=-7102106, 4042 indices spanning [51833,55874] "gauss" 
[23:31:52.602] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 produce 55875 traces: 16505 wiener5, 0 decon_charge5, 12976 gauss5, frame tag: sigproc 
[23:31:52.602] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=34 output frame: frame: ident=169906 time=143 tick=512 with 55875 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":12976 [0] "mp2_roi5":21128 [0] "mp3_roi5":5266 [0] "wiener5":16505 [16505] ] cmm:[ ] 
[23:31:52.826] W [  glue  ] <ChannelSelector:chsel4> Untagged summary not supported, summary will be dropped. 
[23:31:52.827] D [  glue  ] <ChannelSelector:chsel4> input frame: ident=169906 time=143 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=169906 time=143 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[23:31:52.827] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=34 input frame: frame: ident=169906 time=143 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[23:31:52.827] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=34 init nticks=10000 tbinmin=0 tbinmax=10000 
[23:31:52.869] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=34 load plane index: 0, ntraces=1536, input bad regions: 0 
[23:31:54.987] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=34 load plane index: 1, ntraces=1536, input bad regions: 0 
[23:31:57.187] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=34 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      0.000104216     367.895       562.648       341.591       134.532        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.899e-05    0.000103712   0.000272993   9.01495e-05   4.41526e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               71.987        112.41        183.57        98.4862       35.0428        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0351028     0.0354984     0.0372267     0.0353796    0.000469331      18     
produce:pdvddaphne:DAPHNEReaderPDVD               10.6537       17.2662       27.4329       16.0098       4.24398        18     
produce:ophit:OpHitFinder                        0.0739966     0.0960821     0.112091      0.0970893    0.00852722       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0113699     0.0199475     0.0249719     0.0211418    0.00411986       18     
produce:wclsdatavd:WireCellToolkit                82.7175       103.206       119.433       107.256       9.87455        17     
produce:gaushit:GausHitFinder                     1.10546       1.85444       2.67012       1.8718       0.464756        17     
produce:nhitsfilter:NumberOfHitsFilter          0.000395082   0.000723549   0.00118868    0.000752233   0.000190772      17     
produce:reco3d:SpacePointSolver                   10.9735       21.9755       33.4883       21.8933       7.55938        17     
produce:hitpdune:DisambigFromSpacePoints         0.161318      0.347902      0.636616      0.325831       0.12716        17     
produce:pandora:StandardPandora                   48.5152       124.325       224.96        98.1296       63.2934        17     
produce:pandoraTrack:LArPandoraTrackCreation      1.41292       4.28566       10.1666       4.16784       2.17375        17     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0323256     0.0539663     0.0728105     0.0520666     0.0129706       17     
[art]:TriggerResults:TriggerResultInserter      2.3925e-05    7.99291e-05   0.000147938   7.2376e-05    3.57598e-05      17     
end_path:out1:RootOutput                         4.889e-06    2.52407e-05   5.7648e-05    2.8423e-05    1.45021e-05      17     
end_path:out1:RootOutput(write)                   6.18968       6.65495       7.23901       6.65269      0.310351        17     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.93 MB
  Peak resident set size usage (VmHWM): 6718.08 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 23:32:57 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39275 subRun: 1 event: 169906
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-19 01:44:22 UTC       justIN version: 01.05.00