justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 32473.7@dunegpschedd02.fnal.gov

Jobsub ID32473.7@dunegpschedd02.fnal.gov
Workflow ID2135
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-02 21:47:08
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2025-09-02 23:21:13
From worker nodeHostnameheplnc160.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-02 21:50:50
Input filesvd-protodune:np02vd_raw_run039275_0448_df-s01-d2_dw_0_20250902T052910.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-02 23:21:13
Saved logsjustin-logs:32473.7-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

"mp2_roi6":10601 [0] "mp3_roi6":2987 [0] "wiener6":11229 [11229] ] cmm:[ ] input 7: frame: ident=161640 time=113 tick=512 with 21179 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":6354 [0] "mp2_roi7":6131 [0] "mp3_roi7":1258 [0] "wiener7":7436 [7436] ] cmm:[ ] output: frame: ident=161640 time=113 tick=500 with 203686 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":4424 [0] "gauss1":9604 [0] "gauss2":1841 [0] "gauss3":6189 [0] "gauss4":11383 [0] "gauss5":7643 [0] "gauss6":8818 [0] "gauss7":6354 [0] "wiener0":5473 [5473] "wiener1":11625 [11625] "wiener2":2261 [2261] "wiener3":7596 [7596] "wiener4":15103 [15103] "wiener5":9152 [9152] "wiener6":11229 [11229] "wiener7":7436 [7436] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 69875 traces, 69875 summary
Retagger: tagging trace set: gauss with 56256 traces, 0 summary
[00:18:26.178] D [  glue  ] sink frame: #161640 @0.000113 with 203686 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[00:18:26.178] D [  glue  ] <FrameFanout:nfsp> call=35: see EOS 
[00:18:26.178] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=17 
[00:18:26.178] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=35 anode=7 
[00:18:26.178] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=17 
[00:18:26.178] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=35 anode=6 
[00:18:26.178] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=17 
[00:18:26.178] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=35 anode=5 
[00:18:26.178] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=17 
[00:18:26.178] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=35 anode=4 
[00:18:26.178] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=17 
[00:18:26.178] D [  aux   ] <Resampler:resmp3> EOS at call=35 
[00:18:26.178] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=35 anode=3 
[00:18:26.178] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=17 
[00:18:26.178] D [  aux   ] <Resampler:resmp2> EOS at call=35 
[00:18:26.178] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=35 anode=2 
[00:18:26.179] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=17 
[00:18:26.179] D [  aux   ] <Resampler:resmp1> EOS at call=35 
[00:18:26.179] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=35 anode=1 
[00:18:26.179] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=17 
[00:18:26.179] D [  aux   ] <Resampler:resmp0> EOS at call=35 
[00:18:26.179] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=35 anode=0 
[00:18:26.179] D [  glue  ] <FrameFanin:nfsp> EOS at call=35 with 8 
[00:18:26.179] D [  glue  ] frame sink sees EOS
[00:18:26.179] D [ pgraph ] <Pgrapher:> graph execution complete 
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.94 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.85 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.61 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.58 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.18 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.81 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.37 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.81 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.28 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.26 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.25 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.24 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.06 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.03 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[00:18:26.179] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[00:18:26.179] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[00:18:26.179] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[00:18:26.179] I [ timer  ] Timer: Total node execution : 90.30000054091215 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 56256 traces tagged "gauss"
FrameSaver: q=9.28616e+06 n=1192996 tag=gauss
wclsFrameSaver: saving 69875 traces tagged "wiener"
FrameSaver: q=1.00005e+07 n=1142324 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 286 XUs and 660 XVs -> 6 XUVs
C:0 T:1 789 XUs and 881 XVs -> 21 XUVs
C:0 T:2 1977 XUs and 2133 XVs -> 78 XUVs
C:0 T:3 2001 XUs and 2192 XVs -> 70 XUVs
C:0 T:4 213 XUs and 222 XVs -> 5 XUVs
C:0 T:5 159 XUs and 272 XVs -> 10 XUVs
C:0 T:6 1156 XUs and 1234 XVs -> 50 XUVs
C:0 T:7 971 XUs and 1415 XVs -> 66 XUVs
C:0 T:8 33966 XUs and 55525 XVs -> 11310 XUVs
C:0 T:9 5083 XUs and 9092 XVs -> 1239 XUVs
C:0 T:10 3117 XUs and 3898 XVs -> 862 XUVs
C:0 T:11 3702 XUs and 3479 XVs -> 263 XUVs
C:0 T:12 4147 XUs and 6136 XVs -> 788 XUVs
C:0 T:13 5429 XUs and 7284 XVs -> 553 XUVs
C:0 T:14 411 XUs and 752 XVs -> 22 XUVs
C:0 T:15 2353 XUs and 2995 XVs -> 335 XUVs
15678 XUVs total
2480 collection wire objects
15678 potential space points
Neighbour search...
2387262 tests to find 1025446 neighbours
Iterating with no regularization...
Begin: 5.39873e+08
0 4.75648e+08
1 4.72871e+08
2 4.7237e+08
3 4.72197e+08
Now with regularization...
Begin: 4.60569e+08
0 4.60369e+08
03-Sep-2025 00:20:41 BST  Closed output file "np02vd_raw_run039275_0448_df-s01-d2_dw_0_20250902T052910_reco_stage1_20250902T232041_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        170.068       296.369       513.573       255.175       100.733        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      8.9737e-05    0.000144621   0.000328935   0.000141104   5.88347e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               16.025        20.1783       26.5646       19.4273       2.45667        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0353625     0.0371979     0.0519105     0.0359557    0.00376164       18     
produce:pdvddaphne:DAPHNEReaderPDVD               5.30406       5.86196       6.8871        5.91106      0.356737        18     
produce:ophit:OpHitFinder                        0.0652724     0.0774122     0.0903911     0.0767877    0.00635654       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0109135     0.019932      0.0317523     0.0197014    0.00489408       18     
produce:wclsdatavd:WireCellToolkit                86.7238       101.204       127.487       100.092       9.5664         18     
produce:gaushit:GausHitFinder                     1.20043       2.18116       3.25049       2.17483       0.52363        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000270166   0.000530095   0.000831907   0.000548186   0.000136664      18     
produce:reco3d:SpacePointSolver                   8.34406       24.5684       42.6361       24.3317       8.9732         18     
produce:hitpdune:DisambigFromSpacePoints         0.123722      0.457556      0.912435      0.417263      0.209202        18     
produce:pandora:StandardPandora                   26.3246       131.752       334.302       100.953       85.3367        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.719798       2.03747       3.97706       2.15259      0.820482        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0260972     0.0567744     0.0767337     0.0550698     0.013553        18     
[art]:TriggerResults:TriggerResultInserter      2.4015e-05    4.14687e-05   0.000120786   3.74705e-05   2.08252e-05      18     
end_path:out1:RootOutput                         5.41e-06     1.31644e-05   4.4463e-05    1.11505e-05   7.89847e-06      18     
end_path:out1:RootOutput(write)                   7.03286       7.87075       8.68962       7.86481      0.430472        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5230.4 MB
  Peak resident set size usage (VmHWM): 3250.34 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0448_df-s01-d2_dw_0_20250902T052910_reco_stage1_20250902T232041_keepup.root
\tHists: np02vd_raw_run039275_0448_df-s01-d2_dw_0_20250902T052910_reco_stage1_20250902T232042_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-11-05 14:41:27 UTC       justIN version: 01.05.01