justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236799.35@dunegpschedd02.fnal.gov

Jobsub ID236799.35@dunegpschedd02.fnal.gov
Workflow ID9538
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-03 17:55:07
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat2025-11-03 18:41:35
From worker nodeHostnameheplnc165.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 17:56:28
Input filesvd-protodune:np02vd_raw_run040346_3765_df-s04-d0_dw_0_20251103T171041.hdf5
JobscriptExit code0
Real time42m (2566s)
CPU time37m (2263s = 88%)
Max RSS bytes3295076352 (3142 MiB)
Outputting started2025-11-03 18:39:15
Output filesvd-protodune-det-reco:np02vd_raw_run040346_3765_df-s04-d0_dw_0_20251103T171041_reco_stage1_20251103T183904_keepup.root
Finished2025-11-03 18:41:35
Saved logsjustin-logs:236799.35-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

c:anode0sigproc0> EOS at call=47 anode=0 
[18:37:45.972] D [  glue  ] <FrameFanin:nfsp> EOS at call=47 with 8 
[18:37:45.972] D [  glue  ] frame sink sees EOS
[18:37:45.972] D [ pgraph ] <Pgrapher:> graph execution complete 
[18:37:45.972] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.32 sec
[18:37:45.972] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.99 sec
[18:37:45.972] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.19 sec
[18:37:45.972] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.06 sec
[18:37:45.972] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.49 sec
[18:37:45.972] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.46 sec
[18:37:45.972] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.39 sec
[18:37:45.972] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.76 sec
[18:37:45.972] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.58 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.58 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.57 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.52 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:37:45.973] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[18:37:45.973] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[18:37:45.973] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[18:37:45.973] I [ timer  ] Timer: Total node execution : 89.96999978646636 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 53167 traces tagged "gauss"
FrameSaver: q=2.12715e+07 n=1334972 tag=gauss
wclsFrameSaver: saving 67934 traces tagged "wiener"
FrameSaver: q=2.24929e+07 n=1275251 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 520 XUs and 895 XVs -> 49 XUVs
C:0 T:1 742 XUs and 782 XVs -> 26 XUVs
C:0 T:2 17568 XUs and 22227 XVs -> 1258 XUVs
C:0 T:3 808 XUs and 1114 XVs -> 23 XUVs
C:0 T:4 208 XUs and 220 XVs -> 6 XUVs
C:0 T:5 257 XUs and 474 XVs -> 15 XUVs
C:0 T:6 869 XUs and 1170 XVs -> 61 XUVs
C:0 T:7 943 XUs and 1780 XVs -> 43 XUVs
C:0 T:8 1796 XUs and 1200 XVs -> 98 XUVs
C:0 T:9 928 XUs and 1546 XVs -> 49 XUVs
C:0 T:10 16746 XUs and 19931 XVs -> 1210 XUVs
C:0 T:11 2070 XUs and 1850 XVs -> 264 XUVs
C:0 T:12 1037 XUs and 2153 XVs -> 89 XUVs
C:0 T:13 3029 XUs and 2896 XVs -> 100 XUVs
C:0 T:14 1219 XUs and 1205 XVs -> 177 XUVs
C:0 T:15 1871 XUs and 1787 XVs -> 77 XUVs
3545 XUVs total
2253 collection wire objects
3545 potential space points
Neighbour search...
75087 tests to find 22476 neighbours
Iterating with no regularization...
Begin: 4.94675e+10
0 4.87243e+10
1 4.86922e+10
Now with regularization...
Begin: 4.78268e+10
0 4.78256e+10
03-Nov-2025 18:39:04 GMT  Closed output file "np02vd_raw_run040346_3765_df-s04-d0_dw_0_20251103T171041_reco_stage1_20251103T183904_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        77.0697       104.776       185.405       98.7287       26.8093        24     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      4.8682e-05    7.88379e-05   0.000259258   6.29185e-05   4.15688e-05      24     
produce:tpcrawdecoder:PDVDTPCReader               11.2231       12.8619       16.5906       12.2489       1.35878        24     
produce:triggerrawdecoder:PDVDTriggerReader4      0.52373      0.542451      0.597948      0.537348      0.0191352       24     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000269567   0.000384285   0.000796509   0.000383211   0.000108639      24     
produce:ophit:OpHitFinder                       4.6638e-05    9.67693e-05   0.000636417   7.5597e-05    0.000114219      24     
produce:opflash:OpFlashFinderVerticalDrift       3.745e-05    7.17196e-05   0.000394202   6.01635e-05   6.88422e-05      24     
produce:wclsdatavd:WireCellToolkit                38.4422       52.2895       93.076        47.9581       14.1349        24     
produce:gaushit:GausHitFinder                     0.71523       1.00356       1.81836      0.926873      0.223647        24     
produce:nhitsfilter:NumberOfHitsFilter          0.000173917   0.000244379   0.000511792   0.000221837   7.22787e-05      24     
produce:reco3d:SpacePointSolver                   6.33827       9.92165       17.6238       9.37115       2.57511        24     
produce:hitpdune:DisambigFromSpacePoints         0.0828414     0.145069      0.310047      0.130197      0.0530195       24     
produce:pandora:StandardPandora                   11.3705       23.3651       66.4371       20.3341       12.216         24     
produce:pandoraTrack:LArPandoraTrackCreation     0.470355      0.827367       1.99093      0.732872      0.333435        24     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0172188     0.0254574     0.0442734     0.0241642    0.00622896       24     
[art]:TriggerResults:TriggerResultInserter      1.4016e-05    2.52895e-05   6.8519e-05    2.44315e-05   1.11218e-05      24     
end_path:out1:RootOutput                         3.817e-06    7.63125e-06   2.6149e-05     6.467e-06    4.47939e-06      24     
end_path:out1:RootOutput(write)                   3.22021       3.74987       5.42305       3.50586      0.573872        24     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5384.38 MB
  Peak resident set size usage (VmHWM): 3295.08 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040346_3765_df-s04-d0_dw_0_20251103T171041_reco_stage1_20251103T183904_keepup.root
\tHists: np02vd_raw_run040346_3765_df-s04-d0_dw_0_20251103T171041_reco_stage1_20251103T183904_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040346_3765_df-s04-d0_dw_0_20251103T171041_reco_stage1_20251103T183904_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040346_3765_df-s04-d0_dw_0_20251103T171041_reco_stage1_20251103T183904_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762195145.0,
    "core.end_time": 1762195145.0,
    "core.events": [
      347552,
      347556,
      347560,
      347564,
      347568,
      347572,
      347576,
      347580,
      347584,
      347588,
      347592,
      347596,
      347600,
      347604,
      347608,
      347612,
      347616,
      347620,
      347624,
      347628,
      347632,
      347636,
      347640,
      347644
    ],
    "core.event_count": 24,
    "core.first_event_number": 347552,
    "core.last_event_number": 347644,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3765_df-s04-d0_dw_0_20251103T171041.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040346_3765_df-s04-d0_dw_0_20251103T171041_reco_stage1_20251103T183904_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762195145.0,
    "core.end_time": 1762195145.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3765_df-s04-d0_dw_0_20251103T171041.hdf5"
    }
  ]
}
justIN time: 2025-11-05 07:15:03 UTC       justIN version: 01.05.01