justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 239244.143@dunegpschedd01.fnal.gov

Jobsub ID239244.143@dunegpschedd01.fnal.gov
Workflow ID9410
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-28 12:16:10
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-10-28 12:52:10
From worker nodeHostnamewn-lot-033.farm.nikhef.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-10-28 12:28:40
Input filesvd-protodune:np02vd_raw_run040267_0434_df-s03-d1_dw_0_20251026T064053.hdf5
Outputting started 
Output files
Finished2025-10-28 12:52:10
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ue  ] <ChannelSelector:chsel7> input frame: ident=80377 time=43 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=80377 time=43 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:47:25.261] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 input frame: frame: ident=80377 time=43 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:47:25.262] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 init nticks=8000 tbinmin=0 tbinmax=8000 
[13:47:25.283] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 load plane index: 0, ntraces=1536, input bad regions: 0 
[13:47:26.618] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 load plane index: 1, ntraces=1536, input bad regions: 0 
[13:47:28.097] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 load plane index: 2, ntraces=1536, input bad regions: 0 
[13:47:49.105] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 save plane index: 0, Qtot=30748366530 Qloss=-4062338598, 5335 indices spanning [13021,18355] "wiener" 
[13:47:49.261] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 save plane index: 0, Qtot=29291439831 Qloss=-3978314220, 4790 indices spanning [18356,23145] "gauss" 
[13:47:49.716] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 save plane index: 1, Qtot=13667118916 Qloss=-6578579005, 5833 indices spanning [23146,28978] "wiener" 
[13:47:49.874] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 save plane index: 1, Qtot=12188262436 Qloss=-6678234988, 5410 indices spanning [28979,34388] "gauss" 
[13:47:50.182] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 save plane index: 2, Qtot=4168028710 Qloss=-3876802244, 13965 indices spanning [34389,48353] "wiener" 
[13:47:50.499] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 save plane index: 2, Qtot=3473438636 Qloss=-3188937367, 15120 indices spanning [48354,63473] "gauss" 
[13:47:50.499] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 produce 63474 traces: 25133 wiener7, 0 decon_charge7, 25320 gauss7, frame tag: sigproc 
[13:47:50.499] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=20 output frame: frame: ident=80377 time=43 tick=512 with 63474 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":25320 [0] "mp2_roi7":9993 [0] "mp3_roi7":3028 [0] "wiener7":25133 [25133] ] cmm:[ ] 
[13:47:53.641] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[13:47:53.641] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=80377 time=43 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=80377 time=43 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:47:53.641] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 input frame: frame: ident=80377 time=43 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:47:53.641] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 init nticks=8000 tbinmin=0 tbinmax=8000 
[13:47:53.662] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 load plane index: 0, ntraces=1536, input bad regions: 0 
[13:47:55.317] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 load plane index: 1, ntraces=1536, input bad regions: 0 
[13:47:57.029] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 load plane index: 2, ntraces=1536, input bad regions: 0 
[13:47:59.240] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 save plane index: 0, Qtot=102383884 Qloss=-15703022, 2624 indices spanning [9116,11739] "wiener" 
[13:47:59.382] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 save plane index: 0, Qtot=96784996 Qloss=-13017635, 1973 indices spanning [11740,13712] "gauss" 
[13:47:59.693] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 save plane index: 1, Qtot=93026809 Qloss=-13825371, 3170 indices spanning [13713,16882] "wiener" 
[13:47:59.870] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 save plane index: 1, Qtot=85625152 Qloss=-9815839, 2403 indices spanning [16883,19285] "gauss" 
[13:48:00.196] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 save plane index: 2, Qtot=88323008 Qloss=-6322892, 2562 indices spanning [19286,21847] "wiener" 
[13:48:00.518] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 save plane index: 2, Qtot=85208596 Qloss=-3141969, 2128 indices spanning [21848,23975] "gauss" 
[13:48:00.518] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 produce 23976 traces: 8356 wiener6, 0 decon_charge6, 6504 gauss6, frame tag: sigproc 
[13:48:00.518] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=20 output frame: frame: ident=80377 time=43 tick=512 with 23976 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":6504 [0] "mp2_roi6":6904 [0] "mp3_roi6":2212 [0] "wiener6":8356 [8356] ] cmm:[ ] 
[13:48:00.533] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[13:48:00.534] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=80377 time=43 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=80377 time=43 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:48:00.534] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=20 input frame: frame: ident=80377 time=43 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:48:00.534] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=20 init nticks=8000 tbinmin=0 tbinmax=8000 
[13:48:00.555] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=20 load plane index: 0, ntraces=1536, input bad regions: 0 
[13:48:02.155] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=20 load plane index: 1, ntraces=1536, input bad regions: 0 
[13:48:03.960] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=20 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.8348e-05      95.5485       141.691       95.6008       34.0845        11     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      4.9433e-05    7.23335e-05   0.000123321    6.934e-05    1.75033e-05      11     
produce:tpcrawdecoder:PDVDTPCReader               14.9488       19.1242       39.4032       16.7846       6.5469         11     
produce:triggerrawdecoder:PDVDTriggerReader4     0.290927      0.493759       1.5047       0.332569       0.34005        11     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000280115   0.00031801    0.000603141   0.000290214   9.03887e-05      11     
produce:ophit:OpHitFinder                       4.5055e-05    8.22645e-05   0.000429445    4.771e-05    0.000109793      11     
produce:opflash:OpFlashFinderVerticalDrift      3.3904e-05    5.5976e-05    0.000256841   3.5667e-05    6.35338e-05      11     
produce:wclsdatavd:WireCellToolkit                44.058        51.8055       76.1587       47.2815       10.3112        10     
produce:gaushit:GausHitFinder                    0.675631      0.886878       1.14234       0.83084      0.140823        10     
produce:nhitsfilter:NumberOfHitsFilter          0.000203902   0.000251493   0.000396003   0.000225998   5.6299e-05       10     
produce:reco3d:SpacePointSolver                   7.3934        9.66211       13.4093       9.02435       1.92693        10     
produce:hitpdune:DisambigFromSpacePoints         0.0989075     0.142639      0.206639      0.127086      0.032906        10     
produce:pandora:StandardPandora                   10.8452       20.4916       34.5179       19.6631       6.4778         10     
produce:pandoraTrack:LArPandoraTrackCreation     0.407575      0.727914      0.985519      0.668532       0.16431        10     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0219598     0.0271763     0.0341039     0.0263716     0.004605        10     
[art]:TriggerResults:TriggerResultInserter      1.3696e-05    1.79236e-05   4.4773e-05    1.4913e-05    8.96726e-06      10     
end_path:out1:RootOutput                         3.086e-06    4.7569e-06     1.644e-05    3.4915e-06    3.89842e-06      10     
end_path:out1:RootOutput(write)                   3.63692       3.84585       4.18592       3.82045      0.149481        10     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.85 MB
  Peak resident set size usage (VmHWM): 6700.76 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 28-Oct-2025 13:49:22 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40267 subRun: 1 event: 80377
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 09:04:54 UTC       justIN version: 01.05.01