justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 32472.5@dunegpschedd02.fnal.gov

Jobsub ID32472.5@dunegpschedd02.fnal.gov
Workflow ID2135
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-02 21:43:08
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-09-02 23:03:30
From worker nodeHostnamewn-la-01.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-02 21:51:11
Input filesvd-protodune:np02vd_raw_run039275_0447_df-s03-d1_dw_0_20250902T052648.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-02 23:03:30
Saved logsjustin-logs:32472.5-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

roi6":11527 [0] "mp3_roi6":3232 [0] "wiener6":13427 [13427] ] cmm:[ ] input 7: frame: ident=161267 time=111 tick=512 with 34563 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":9700 [0] "mp2_roi7":9811 [0] "mp3_roi7":3302 [0] "wiener7":11750 [11750] ] cmm:[ ] output: frame: ident=161267 time=111 tick=500 with 191715 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":3316 [0] "gauss1":5301 [0] "gauss2":2872 [0] "gauss3":2637 [0] "gauss4":10290 [0] "gauss5":8478 [0] "gauss6":11030 [0] "gauss7":9700 [0] "wiener0":4063 [4063] "wiener1":6286 [6286] "wiener2":3589 [3589] "wiener3":3381 [3381] "wiener4":12635 [12635] "wiener5":10248 [10248] "wiener6":13427 [13427] "wiener7":11750 [11750] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 65379 traces, 65379 summary
Retagger: tagging trace set: gauss with 53624 traces, 0 summary
[01:00:59.112] D [  glue  ] sink frame: #161267 @0.000111 with 191715 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[01:00:59.113] D [  glue  ] <FrameFanout:nfsp> call=35: see EOS 
[01:00:59.113] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=17 
[01:00:59.113] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=35 anode=7 
[01:00:59.113] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=17 
[01:00:59.113] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=35 anode=6 
[01:00:59.113] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=17 
[01:00:59.113] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=35 anode=5 
[01:00:59.113] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=17 
[01:00:59.113] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=35 anode=4 
[01:00:59.113] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=17 
[01:00:59.113] D [  aux   ] <Resampler:resmp3> EOS at call=35 
[01:00:59.113] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=35 anode=3 
[01:00:59.113] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=17 
[01:00:59.113] D [  aux   ] <Resampler:resmp2> EOS at call=35 
[01:00:59.113] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=35 anode=2 
[01:00:59.113] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=17 
[01:00:59.113] D [  aux   ] <Resampler:resmp1> EOS at call=35 
[01:00:59.113] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=35 anode=1 
[01:00:59.113] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=17 
[01:00:59.113] D [  aux   ] <Resampler:resmp0> EOS at call=35 
[01:00:59.113] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=35 anode=0 
[01:00:59.113] D [  glue  ] <FrameFanin:nfsp> EOS at call=35 with 8 
[01:00:59.113] D [  glue  ] frame sink sees EOS
[01:00:59.113] D [ pgraph ] <Pgrapher:> graph execution complete 
[01:00:59.113] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.06 sec
[01:00:59.113] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.89 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.7 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.47 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.98 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.8 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.73 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.49 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.71 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.7 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.69 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.68 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:00:59.114] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[01:00:59.114] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[01:00:59.114] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[01:00:59.114] I [ timer  ] Timer: Total node execution : 92.94999985583127 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 53624 traces tagged "gauss"
FrameSaver: q=1.13932e+07 n=1271882 tag=gauss
wclsFrameSaver: saving 65379 traces tagged "wiener"
FrameSaver: q=1.23243e+07 n=1225764 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 393 XUs and 490 XVs -> 28 XUVs
C:0 T:1 323 XUs and 311 XVs -> 17 XUVs
C:0 T:2 875 XUs and 977 XVs -> 23 XUVs
C:0 T:3 547 XUs and 689 XVs -> 34 XUVs
C:0 T:4 566 XUs and 585 XVs -> 35 XUVs
C:0 T:5 239 XUs and 218 XVs -> 7 XUVs
C:0 T:6 652 XUs and 713 XVs -> 40 XUVs
C:0 T:7 120 XUs and 134 XVs -> 10 XUVs
C:0 T:8 6331 XUs and 8031 XVs -> 708 XUVs
C:0 T:9 7951 XUs and 18044 XVs -> 2331 XUVs
C:0 T:10 2431 XUs and 5969 XVs -> 835 XUVs
C:0 T:11 14798 XUs and 32972 XVs -> 2899 XUVs
C:0 T:12 16007 XUs and 46281 XVs -> 3086 XUVs
C:0 T:13 2510 XUs and 5222 XVs -> 648 XUVs
C:0 T:14 2885 XUs and 5546 XVs -> 368 XUVs
C:0 T:15 19157 XUs and 37470 XVs -> 3791 XUVs
14860 XUVs total
2564 collection wire objects
14860 potential space points
Neighbour search...
1321940 tests to find 550840 neighbours
Iterating with no regularization...
Begin: 9.56842e+08
0 8.33338e+08
1 8.23996e+08
2 8.2269e+08
3 8.22211e+08
Now with regularization...
Begin: 8.09515e+08
0 8.09127e+08
03-Sep-2025 01:03:02 CEST  Closed output file "np02vd_raw_run039275_0447_df-s03-d1_dw_0_20250902T052648_reco_stage1_20250902T230302_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        175.081       236.711       413.039       213.833       65.062         18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.4737e-05    8.18699e-05   0.000251687   7.1993e-05    4.13576e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               16.9601       29.6614       46.857        27.7809       7.87116        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0360736     0.0382451     0.0475595     0.0374033    0.00279171       18     
produce:pdvddaphne:DAPHNEReaderPDVD               6.16434       7.24486       9.44171       7.12762      0.873812        18     
produce:ophit:OpHitFinder                        0.0412482     0.0627873     0.0887664     0.0628821    0.00971218       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00584057     0.0117048     0.0167312     0.0115739    0.00272501       18     
produce:wclsdatavd:WireCellToolkit                87.9979       98.1201       110.941       99.6652       6.56364        18     
produce:gaushit:GausHitFinder                    0.828421       1.29205       1.9339        1.20457      0.310916        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000228723   0.000561676   0.00130504    0.000498793   0.000258082      18     
produce:reco3d:SpacePointSolver                   10.052        17.6399       33.0966       15.855        6.14411        18     
produce:hitpdune:DisambigFromSpacePoints         0.121812      0.252476      0.560163      0.215368      0.112826        18     
produce:pandora:StandardPandora                   23.6617       75.0209       223.638       53.8935        58.47         18     
produce:pandoraTrack:LArPandoraTrackCreation     0.839119       1.78693       3.90908       1.51801      0.798272        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0226349     0.0335193     0.0595435     0.0302979     0.0103124       18     
[art]:TriggerResults:TriggerResultInserter      2.5108e-05    3.75435e-05   9.1948e-05    3.4517e-05    1.44642e-05      18     
end_path:out1:RootOutput                         5.609e-06    7.91978e-06   2.7342e-05     6.715e-06    4.78939e-06      18     
end_path:out1:RootOutput(write)                   4.85241       5.45229       7.49511       5.12444      0.801989        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5242.83 MB
  Peak resident set size usage (VmHWM): 3356.29 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0447_df-s03-d1_dw_0_20250902T052648_reco_stage1_20250902T230302_keepup.root
\tHists: np02vd_raw_run039275_0447_df-s03-d1_dw_0_20250902T052648_reco_stage1_20250902T230302_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-11-05 14:43:55 UTC       justIN version: 01.05.01