justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241227.9@dunegpschedd01.fnal.gov

Jobsub ID241227.9@dunegpschedd01.fnal.gov
Workflow ID9373
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-30 23:48:17
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-10-31 01:21:32
From worker nodeHostnamewn-sate-046.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-30 23:49:43
Input filesvd-protodune:np02vd_raw_run040266_0265_df-s04-d0_dw_0_20251025T043531.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 01:21:32
Saved logsjustin-logs:241227.9-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

race sets:[ ] cmm:[ ] 
[02:18:44.553] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=44 init nticks=8064 tbinmin=0 tbinmax=8064 
[02:18:44.579] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=44 load plane index: 0, ntraces=1536, input bad regions: 0 
[02:18:46.634] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=44 load plane index: 1, ntraces=1536, input bad regions: 0 
[02:18:48.704] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=44 load plane index: 2, ntraces=1536, input bad regions: 0 
[02:18:56.296] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=44 save plane index: 0, Qtot=433792980 Qloss=-51179553, 8234 indices spanning [32582,40815] "wiener" 
[02:18:56.516] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=44 save plane index: 0, Qtot=388733366 Qloss=-42151345, 6107 indices spanning [40816,46922] "gauss" 
[02:18:57.419] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=44 save plane index: 1, Qtot=495085681 Qloss=-33716241, 8131 indices spanning [46923,55053] "wiener" 
[02:18:57.634] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=44 save plane index: 1, Qtot=403105617 Qloss=-27803191, 6222 indices spanning [55054,61275] "gauss" 
[02:18:58.067] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=44 save plane index: 2, Qtot=356763317 Qloss=-18423041, 10229 indices spanning [61276,71504] "wiener" 
[02:18:58.497] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=44 save plane index: 2, Qtot=344642073 Qloss=-6156788, 7313 indices spanning [71505,78817] "gauss" 
[02:18:58.497] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=44 produce 78818 traces: 26594 wiener2, 0 decon_charge2, 19642 gauss2, frame tag: sigproc 
[02:18:58.497] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=44 output frame: frame: ident=49036 time=90 tick=500 with 78818 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":19642 [0] "mp2_roi2":25561 [0] "mp3_roi2":7021 [0] "wiener2":26594 [26594] ] cmm:[ ] 
[02:18:58.907] W [  glue  ] <ChannelSelector:chsel1> Untagged summary not supported, summary will be dropped. 
[02:18:58.907] D [  glue  ] <ChannelSelector:chsel1> input frame: ident=49036 time=90 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=49036 time=90 tick=512 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:18:58.908] D [  aux   ] <Resampler:resmp1> first ch=617 Ts=512 Ns=7875 Ns_pad=8000 Nrat=125 Tr=500 Nr=8192 Nout=8064 padding:linear 
[02:18:59.108] D [  aux   ] <Resampler:resmp1> resample 1536 traces at call=44 
[02:18:59.108] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 input frame: frame: ident=49036 time=90 tick=500 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:18:59.108] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 init nticks=8064 tbinmin=0 tbinmax=8064 
[02:18:59.137] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 load plane index: 0, ntraces=1536, input bad regions: 0 
[02:19:01.194] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 load plane index: 1, ntraces=1536, input bad regions: 0 
[02:19:03.226] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 load plane index: 2, ntraces=1536, input bad regions: 0 
[02:19:30.503] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 save plane index: 0, Qtot=1105306795 Qloss=-59185747, 11349 indices spanning [40025,51373] "wiener" 
[02:19:30.727] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 save plane index: 0, Qtot=970201999 Qloss=-55351490, 9002 indices spanning [51374,60375] "gauss" 
[02:19:31.548] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 save plane index: 1, Qtot=1271038080 Qloss=-106871754, 11011 indices spanning [60376,71386] "wiener" 
[02:19:31.763] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 save plane index: 1, Qtot=1105607082 Qloss=-104554678, 8602 indices spanning [71387,79988] "gauss" 
[02:19:32.189] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 save plane index: 2, Qtot=1138462640 Qloss=-19254313, 11634 indices spanning [79989,91622] "wiener" 
[02:19:32.606] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 save plane index: 2, Qtot=1127002433 Qloss=-7666261, 8293 indices spanning [91623,99915] "gauss" 
[02:19:32.606] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 produce 99916 traces: 33994 wiener1, 0 decon_charge1, 25897 gauss1, frame tag: sigproc 
[02:19:32.606] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=44 output frame: frame: ident=49036 time=90 tick=500 with 99916 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss1":25897 [0] "mp2_roi1":30622 [0] "mp3_roi1":9403 [0] "wiener1":33994 [33994] ] cmm:[ ] 
[02:19:35.592] W [  glue  ] <ChannelSelector:chsel0> Untagged summary not supported, summary will be dropped. 
[02:19:35.593] D [  glue  ] <ChannelSelector:chsel0> input frame: ident=49036 time=90 tick=512 with 12288 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=49036 time=90 tick=512 with 1536 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:19:35.598] D [  aux   ] <Resampler:resmp0> first ch=1388 Ts=512 Ns=7875 Ns_pad=8000 Nrat=125 Tr=500 Nr=8192 Nout=8064 padding:linear 
[02:19:35.799] D [  aux   ] <Resampler:resmp0> resample 1536 traces at call=44 
[02:19:35.799] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 input frame: frame: ident=49036 time=90 tick=500 with 1536 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:19:35.799] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 init nticks=8064 tbinmin=0 tbinmax=8064 
[02:19:35.828] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 load plane index: 0, ntraces=1536, input bad regions: 0 
[02:19:37.810] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 load plane index: 1, ntraces=1536, input bad regions: 0 
[02:19:39.908] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=44 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      5.7067e-05      220.065       350.473       212.818       72.8934        23     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.37e-05     7.03352e-05   0.000163738   6.6084e-05    2.13469e-05      23     
produce:tpcrawdecoder:PDVDTPCReader               29.0989       92.5603       179.119       91.7107       31.9282        23     
produce:triggerrawdecoder:PDVDTriggerReader4     0.482335      0.729608       1.56911       0.64322      0.249711        23     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000357883   0.000390901   0.000669317   0.000376246   6.07418e-05      23     
produce:ophit:OpHitFinder                       5.2479e-05     8.072e-05    0.000476325   5.9743e-05    8.47539e-05      23     
produce:opflash:OpFlashFinderVerticalDrift      4.6087e-05    6.19593e-05   0.000281138   5.0365e-05    4.72452e-05      23     
produce:wclsdatavd:WireCellToolkit                52.8223       78.8677       126.19        68.4512       21.5199        22     
produce:gaushit:GausHitFinder                    0.693072       1.32493       2.21735       1.25842      0.321157        22     
produce:nhitsfilter:NumberOfHitsFilter          0.000207009   0.000337174    0.0005326    0.000325236   8.1585e-05       22     
produce:reco3d:SpacePointSolver                   6.40655       13.8145       27.5584       13.2211       4.48527        22     
produce:hitpdune:DisambigFromSpacePoints         0.0771649      0.20477      0.513947       0.18127      0.0905831       22     
produce:pandora:StandardPandora                   10.0405       36.0116       122.169       26.3534       23.0074        22     
produce:pandoraTrack:LArPandoraTrackCreation     0.297696       1.37345       3.64146       1.25567      0.732073        22     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0214981     0.0375142     0.0508675     0.0367694    0.00766703       22     
[art]:TriggerResults:TriggerResultInserter      1.6581e-05    2.77944e-05   5.9983e-05    2.2898e-05    1.11395e-05      22     
end_path:out1:RootOutput                         6.252e-06    8.65541e-06   2.0769e-05     6.989e-06    3.71196e-06      22     
end_path:out1:RootOutput(write)                   4.42087       4.97759       5.67008       4.97594      0.324632        22     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.86 MB
  Peak resident set size usage (VmHWM): 6713.32 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 02:21:03 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 49036
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:26:20 UTC       justIN version: 01.05.01