justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236272.2@dunegpschedd02.fnal.gov

Jobsub ID236272.2@dunegpschedd02.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-02 10:01:22
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-11-02 10:44:21
From worker nodeHostnamewn-sate-036.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-02 10:01:35
Input filesvd-protodune:np02vd_raw_run040266_0363_df-s04-d1_dw_0_20251025T071956.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-02 10:44:21
Saved logsjustin-logs:236272.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

"orig3" ] 0 tagged trace sets:[ ] cmm:[ ] 
[11:41:47.937] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 init nticks=9911 tbinmin=0 tbinmax=9911 
[11:41:47.969] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 load plane index: 0, ntraces=1536, input bad regions: 0 
[11:41:52.978] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 load plane index: 1, ntraces=1536, input bad regions: 0 
[11:41:58.146] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 load plane index: 2, ntraces=1536, input bad regions: 0 
[11:42:07.749] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 save plane index: 0, Qtot=422520443 Qloss=-55237517, 6419 indices spanning [28621,35039] "wiener" 
[11:42:08.329] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 save plane index: 0, Qtot=366951554 Qloss=-50060155, 5035 indices spanning [35040,40074] "gauss" 
[11:42:09.368] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 save plane index: 1, Qtot=344323530 Qloss=-24553996, 7445 indices spanning [40075,47519] "wiener" 
[11:42:09.976] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 save plane index: 1, Qtot=263944263 Qloss=-18435384, 5639 indices spanning [47520,53158] "gauss" 
[11:42:10.882] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 save plane index: 2, Qtot=231611773 Qloss=-18713676, 8090 indices spanning [53159,61248] "wiener" 
[11:42:11.781] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 save plane index: 2, Qtot=218603469 Qloss=-5464523, 5973 indices spanning [61249,67221] "gauss" 
[11:42:11.781] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 produce 67222 traces: 21954 wiener3, 0 decon_charge3, 16647 gauss3, frame tag: sigproc 
[11:42:11.781] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=28 output frame: frame: ident=66973 time=64 tick=500 with 67222 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":16647 [0] "mp2_roi3":23065 [0] "mp3_roi3":5556 [0] "wiener3":21954 [21954] ] cmm:[ ] 
[11:42:12.036] W [  glue  ] <ChannelSelector:chsel2> Untagged summary not supported, summary will be dropped. 
[11:42:12.036] D [  glue  ] <ChannelSelector:chsel2> input frame: ident=66973 time=64 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=66973 time=64 tick=512 with 1536 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[11:42:12.037] D [  aux   ] <Resampler:resmp2> first ch=3406 Ts=512 Ns=9679 Ns_pad=9750 Nrat=125 Tr=500 Nr=9984 Nout=9911 padding:linear 
[11:42:12.333] D [  aux   ] <Resampler:resmp2> resample 1536 traces at call=28 
[11:42:12.333] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 input frame: frame: ident=66973 time=64 tick=500 with 1536 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[11:42:12.333] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 init nticks=9911 tbinmin=0 tbinmax=9911 
[11:42:12.372] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 load plane index: 0, ntraces=1536, input bad regions: 0 
[11:42:17.629] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 load plane index: 1, ntraces=1536, input bad regions: 0 
[11:42:22.784] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 load plane index: 2, ntraces=1536, input bad regions: 0 
[11:42:29.721] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 save plane index: 0, Qtot=119281491 Qloss=-17388054, 2634 indices spanning [12887,15520] "wiener" 
[11:42:30.302] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 save plane index: 0, Qtot=113357839 Qloss=-14800482, 2172 indices spanning [15521,17692] "gauss" 
[11:42:31.179] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 save plane index: 1, Qtot=112248848 Qloss=-17840662, 2903 indices spanning [17693,20595] "wiener" 
[11:42:31.754] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 save plane index: 1, Qtot=102037903 Qloss=-14836576, 2151 indices spanning [20596,22746] "gauss" 
[11:42:32.641] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 save plane index: 2, Qtot=95809783 Qloss=-6082536, 3941 indices spanning [22747,26687] "wiener" 
[11:42:33.522] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 save plane index: 2, Qtot=92315519 Qloss=-2538706, 3107 indices spanning [26688,29794] "gauss" 
[11:42:33.522] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 produce 29795 traces: 9478 wiener2, 0 decon_charge2, 7430 gauss2, frame tag: sigproc 
[11:42:33.522] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=28 output frame: frame: ident=66973 time=64 tick=500 with 29795 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":7430 [0] "mp2_roi2":10770 [0] "mp3_roi2":2117 [0] "wiener2":9478 [9478] ] cmm:[ ] 
[11:42:33.558] W [  glue  ] <ChannelSelector:chsel1> Untagged summary not supported, summary will be dropped. 
[11:42:33.558] D [  glue  ] <ChannelSelector:chsel1> input frame: ident=66973 time=64 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=66973 time=64 tick=512 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[11:42:33.559] D [  aux   ] <Resampler:resmp1> first ch=617 Ts=512 Ns=9679 Ns_pad=9750 Nrat=125 Tr=500 Nr=9984 Nout=9911 padding:linear 
[11:42:33.854] D [  aux   ] <Resampler:resmp1> resample 1536 traces at call=28 
[11:42:33.854] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=28 input frame: frame: ident=66973 time=64 tick=500 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[11:42:33.854] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=28 init nticks=9911 tbinmin=0 tbinmax=9911 
[11:42:33.889] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=28 load plane index: 0, ntraces=1536, input bad regions: 0 
[11:42:39.075] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=28 load plane index: 1, ntraces=1536, input bad regions: 0 
[11:42:44.326] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=28 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      9.8685e-05      153.024       280.592       143.134       59.1339        15     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.1705e-05    7.85608e-05   0.000174898   7.0372e-05    2.69747e-05      15     
produce:tpcrawdecoder:PDVDTPCReader               12.5783       13.2722       14.4543       13.0284      0.602357        15     
produce:triggerrawdecoder:PDVDTriggerReader4     0.517901      0.526368      0.543439      0.521832     0.00902023       15     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000373391   0.000422902   0.000728949   0.000401203   8.29377e-05      15     
produce:ophit:OpHitFinder                       6.1055e-05    0.00010683    0.000522331   6.9139e-05    0.00011335       15     
produce:opflash:OpFlashFinderVerticalDrift      5.5455e-05    8.12445e-05   0.000317215   6.0293e-05    6.41839e-05      15     
produce:wclsdatavd:WireCellToolkit                69.302        85.8411       117.48        76.7314       17.7271        14     
produce:gaushit:GausHitFinder                    0.986477       1.29248       1.81966       1.22855      0.268202        14     
produce:nhitsfilter:NumberOfHitsFilter          0.000213361   0.000303169   0.000575541   0.000283528   8.20793e-05      14     
produce:reco3d:SpacePointSolver                   7.43847       12.6143       18.1025       12.0725       3.12803        14     
produce:hitpdune:DisambigFromSpacePoints         0.108983      0.196076      0.350899      0.167436      0.0721303       14     
produce:pandora:StandardPandora                   16.9502       43.4478       161.257       29.6875       36.3869        14     
produce:pandoraTrack:LArPandoraTrackCreation     0.679521       1.6989        3.0987        1.57083      0.784605        14     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0265951     0.0366691     0.049904      0.0380556     0.0073041       14     
[art]:TriggerResults:TriggerResultInserter      4.2129e-05    5.22224e-05   0.00011171    4.73135e-05   1.69422e-05      14     
end_path:out1:RootOutput                         6.121e-06    1.17048e-05   3.6318e-05    1.0284e-05    6.96263e-06      14     
end_path:out1:RootOutput(write)                   4.57924       5.04636       5.79028       4.94244      0.329744        14     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.8 MB
  Peak resident set size usage (VmHWM): 6698.31 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 02-Nov-2025 11:43:58 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 66973
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-04 07:30:38 UTC       justIN version: 01.05.01