justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 32473.4@dunegpschedd02.fnal.gov

Jobsub ID32473.4@dunegpschedd02.fnal.gov
Workflow ID2135
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-02 21:47:08
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2025-09-02 23:22:38
From worker nodeHostnameheplnc153.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-02 21:51:04
Input filesvd-protodune:np02vd_raw_run039275_0448_df-s04-d0_dw_0_20250902T052900.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-02 23:22:38
Saved logsjustin-logs:32473.4-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ident=161630 time=122 tick=512 with 27739 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":7568 [0] "mp2_roi7":8156 [0] "mp3_roi7":2367 [0] "wiener7":9648 [9648] ] cmm:[ ] output: frame: ident=161630 time=122 tick=500 with 244118 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":2980 [0] "gauss1":3136 [0] "gauss2":6349 [0] "gauss3":8239 [0] "gauss4":13541 [0] "gauss5":12331 [0] "gauss6":11498 [0] "gauss7":7568 [0] "wiener0":3700 [3700] "wiener1":3785 [3785] "wiener2":7616 [7616] "wiener3":9834 [9834] "wiener4":17043 [17043] "wiener5":15204 [15204] "wiener6":14491 [14491] "wiener7":9648 [9648] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 81321 traces, 81321 summary
Retagger: tagging trace set: gauss with 65642 traces, 0 summary
[00:18:32.235] D [  glue  ] sink frame: #161630 @0.000122 with 244118 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[00:18:32.235] D [  glue  ] <FrameFanout:nfsp> call=35: see EOS 
[00:18:32.236] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=17 
[00:18:32.236] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=35 anode=7 
[00:18:32.236] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=17 
[00:18:32.236] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=35 anode=6 
[00:18:32.236] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=17 
[00:18:32.236] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=35 anode=5 
[00:18:32.236] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=17 
[00:18:32.236] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=35 anode=4 
[00:18:32.236] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=17 
[00:18:32.236] D [  aux   ] <Resampler:resmp3> EOS at call=35 
[00:18:32.236] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=35 anode=3 
[00:18:32.236] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=17 
[00:18:32.236] D [  aux   ] <Resampler:resmp2> EOS at call=35 
[00:18:32.236] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=35 anode=2 
[00:18:32.236] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=17 
[00:18:32.236] D [  aux   ] <Resampler:resmp1> EOS at call=35 
[00:18:32.236] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=35 anode=1 
[00:18:32.236] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=17 
[00:18:32.236] D [  aux   ] <Resampler:resmp0> EOS at call=35 
[00:18:32.236] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=35 anode=0 
[00:18:32.236] D [  glue  ] <FrameFanin:nfsp> EOS at call=35 with 8 
[00:18:32.236] D [  glue  ] frame sink sees EOS
[00:18:32.236] D [ pgraph ] <Pgrapher:> graph execution complete 
[00:18:32.236] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.63 sec
[00:18:32.236] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.41 sec
[00:18:32.236] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.95 sec
[00:18:32.236] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.02 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.59 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.39 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.77 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.59 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.27 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.25 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.03 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.97 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:18:32.237] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[00:18:32.237] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[00:18:32.237] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[00:18:32.237] I [ timer  ] Timer: Total node execution : 83.9300008378923 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 65642 traces tagged "gauss"
FrameSaver: q=1.23168e+07 n=1461353 tag=gauss
wclsFrameSaver: saving 81321 traces tagged "wiener"
FrameSaver: q=1.32087e+07 n=1406369 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 610 XUs and 710 XVs -> 36 XUVs
C:0 T:1 365 XUs and 534 XVs -> 20 XUVs
C:0 T:2 225 XUs and 398 XVs -> 18 XUVs
C:0 T:3 420 XUs and 570 XVs -> 26 XUVs
C:0 T:4 802 XUs and 1204 XVs -> 44 XUVs
C:0 T:5 1434 XUs and 1452 XVs -> 69 XUVs
C:0 T:6 1245 XUs and 1616 XVs -> 59 XUVs
C:0 T:7 2068 XUs and 2435 XVs -> 89 XUVs
C:0 T:8 19274 XUs and 43114 XVs -> 6039 XUVs
C:0 T:9 10234 XUs and 17387 XVs -> 1298 XUVs
C:0 T:10 5787 XUs and 7947 XVs -> 1003 XUVs
C:0 T:11 12541 XUs and 23341 XVs -> 2515 XUVs
C:0 T:12 12770 XUs and 24697 XVs -> 4049 XUVs
C:0 T:13 6796 XUs and 11547 XVs -> 1342 XUVs
C:0 T:14 5224 XUs and 8198 XVs -> 1330 XUVs
C:0 T:15 2072 XUs and 3573 XVs -> 623 XUVs
18560 XUVs total
3017 collection wire objects
18560 potential space points
Neighbour search...
2023902 tests to find 894238 neighbours
Iterating with no regularization...
Begin: 6.3642e+08
0 5.53593e+08
1 5.4713e+08
2 5.45872e+08
3 5.45395e+08
Now with regularization...
Begin: 5.28843e+08
0 5.28425e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
03-Sep-2025 00:22:11 BST  Closed output file "np02vd_raw_run039275_0448_df-s04-d0_dw_0_20250902T052900_reco_stage1_20250902T232211_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        200.28        300.724       774.787       266.49        131.289        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.099e-05    0.000146859   0.000271478   0.000152475   4.3228e-05       18     
produce:tpcrawdecoder:PDVDTPCReader               14.0999       15.0149       15.9409       15.0469      0.377413        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0352914     0.0362501     0.0452191     0.0355322    0.00223157       18     
produce:pdvddaphne:DAPHNEReaderPDVD               4.7924        5.41877       6.09819       5.54105      0.358141        18     
produce:ophit:OpHitFinder                        0.0658335     0.0823688     0.0955532     0.0826458    0.00806523       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0156869     0.0224451     0.0334666     0.0232433    0.00457889       18     
produce:wclsdatavd:WireCellToolkit                79.7887       91.1813       103.473       91.9246       6.49089        18     
produce:gaushit:GausHitFinder                     1.52005       2.20356       3.32287       2.04579      0.469298        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000351228   0.000533815   0.000930503   0.000495097   0.000136181      18     
produce:reco3d:SpacePointSolver                   14.964        25.0551       43.225         24.56        7.67653        18     
produce:hitpdune:DisambigFromSpacePoints          0.2109       0.451178      0.834956      0.404584      0.179338        18     
produce:pandora:StandardPandora                   58.1115       151.532       617.942       120.682       125.098        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.980093       1.90058       4.37813       1.57308      0.848232        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0329342     0.0536326     0.0808019     0.050506      0.0130505       18     
[art]:TriggerResults:TriggerResultInserter      2.4727e-05    4.87731e-05   0.000162665   4.1548e-05    2.93572e-05      18     
end_path:out1:RootOutput                         7.714e-06    1.34335e-05   5.4452e-05    1.07805e-05   1.00595e-05      18     
end_path:out1:RootOutput(write)                   7.1224        7.71929       8.58786       7.80286      0.403663        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5076.89 MB
  Peak resident set size usage (VmHWM): 3126.14 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0448_df-s04-d0_dw_0_20250902T052900_reco_stage1_20250902T232211_keepup.root
\tHists: np02vd_raw_run039275_0448_df-s04-d0_dw_0_20250902T052900_reco_stage1_20250902T232211_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-11-05 14:39:41 UTC       justIN version: 01.05.01