justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40870.157@dunegpschedd02.fnal.gov

Jobsub ID40870.157@dunegpschedd02.fnal.gov
Workflow ID2740
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-17 08:51:00
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2025-09-17 10:51:01
From worker nodeHostnamehd79.dice.priv
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-17 09:18:27
Input filesvd-protodune:np02vd_raw_run039388_0534_df-s05-d0_dw_0_20250917T002552.hdf5
JobscriptExit code0
Real time1h (5239s)
CPU time1h (4805s = 91%)
Max RSS bytes3205500928 (3057 MiB)
Outputting started2025-09-17 10:45:46
Output filesvd-protodune-det-reco:np02vd_raw_run039388_0534_df-s05-d0_dw_0_20250917T002552_reco_stage1_20250917T104535_keepup.root
Finished2025-09-17 10:51:01
Saved logsjustin-logs:40870.157-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

98] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=37 anode=0 
[10:44:17.698] D [  glue  ] <FrameFanin:nfsp> EOS at call=37 with 8 
[10:44:17.698] D [  glue  ] frame sink sees EOS
[10:44:17.698] D [ pgraph ] <Pgrapher:> graph execution complete 
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.2 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.16 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.1 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.1 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.96 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.88 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.75 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.14 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.27 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.27 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.25 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.24 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:44:17.698] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:44:17.699] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[10:44:17.699] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:44:17.699] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:44:17.699] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[10:44:17.699] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:44:17.699] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[10:44:17.699] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[10:44:17.699] I [ timer  ] Timer: Total node execution : 93.36000088602304 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 44144 traces tagged "gauss"
FrameSaver: q=6.7051e+06 n=905774 tag=gauss
wclsFrameSaver: saving 55188 traces tagged "wiener"
FrameSaver: q=7.09693e+06 n=868380 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1066 XUs and 1193 XVs -> 34 XUVs
C:0 T:1 525 XUs and 794 XVs -> 22 XUVs
C:0 T:2 1195 XUs and 610 XVs -> 86 XUVs
C:0 T:3 2366 XUs and 2263 XVs -> 217 XUVs
C:0 T:4 942 XUs and 842 XVs -> 48 XUVs
C:0 T:5 1252 XUs and 994 XVs -> 140 XUVs
C:0 T:6 2193 XUs and 2033 XVs -> 213 XUVs
C:0 T:7 459 XUs and 470 XVs -> 42 XUVs
C:0 T:8 2452 XUs and 2510 XVs -> 163 XUVs
C:0 T:9 1163 XUs and 1789 XVs -> 55 XUVs
C:0 T:10 1917 XUs and 2625 XVs -> 102 XUVs
C:0 T:11 847 XUs and 1208 XVs -> 69 XUVs
C:0 T:12 1130 XUs and 1754 XVs -> 62 XUVs
C:0 T:13 840 XUs and 921 XVs -> 28 XUVs
C:0 T:14 1074 XUs and 1135 XVs -> 79 XUVs
C:0 T:15 365 XUs and 385 XVs -> 40 XUVs
1400 XUVs total
1115 collection wire objects
1400 potential space points
Neighbour search...
15600 tests to find 8686 neighbours
Iterating with no regularization...
Begin: 2.61849e+08
0 2.55008e+08
1 2.54858e+08
Now with regularization...
Begin: 2.51205e+08
0 2.51199e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
17-Sep-2025 10:45:35 UTC  Closed output file "np02vd_raw_run039388_0534_df-s05-d0_dw_0_20250917T002552_reco_stage1_20250917T104535_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        166.967       271.975       446.114       237.082       76.0404        19     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.1074e-05    0.000121998   0.000299635   0.00010113    5.76451e-05      19     
produce:tpcrawdecoder:PDVDTPCReader               17.8736       22.2195       27.7795       21.638        2.52725        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.312697      0.319946      0.339555      0.317023     0.00727612       19     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000333069   0.000497215   0.00139723    0.000419382   0.000255275      19     
produce:ophit:OpHitFinder                       0.000155052   0.000323954   0.00103582    0.000271032   0.000195181      19     
produce:opflash:OpFlashFinderVerticalDrift       5.817e-05    9.55894e-05   0.000383083   7.2206e-05    7.18319e-05      19     
produce:wclsdatavd:WireCellToolkit                91.055        109.216       142.823       104.334       13.2701        19     
produce:gaushit:GausHitFinder                     1.35962       2.14208       3.1372        2.03593      0.455253        19     
produce:nhitsfilter:NumberOfHitsFilter          0.000302902   0.000461344   0.000843873   0.00042965    0.000121783      19     
produce:reco3d:SpacePointSolver                   11.0044       20.1628       32.8937       18.087        6.11144        19     
produce:hitpdune:DisambigFromSpacePoints         0.152329      0.342851       0.7493        0.28012      0.163783        19     
produce:pandora:StandardPandora                   33.5524       108.331       244.377       76.1056       59.2976        19     
produce:pandoraTrack:LArPandoraTrackCreation      1.01101       1.90559       3.13729       1.75177      0.566576        19     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0362125     0.0480666     0.0717944     0.0461342    0.00880515       19     
[art]:TriggerResults:TriggerResultInserter      2.1331e-05    3.71592e-05   7.0594e-05    3.2642e-05    1.35628e-05      19     
end_path:out1:RootOutput                         4.388e-06    1.14621e-05   2.8153e-05     1.077e-05    5.11119e-06      19     
end_path:out1:RootOutput(write)                   6.61828       7.23602       8.32886       7.03708      0.466134        19     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5166.08 MB
  Peak resident set size usage (VmHWM): 3205.5 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039388_0534_df-s05-d0_dw_0_20250917T002552_reco_stage1_20250917T104535_keepup.root
\tHists: np02vd_raw_run039388_0534_df-s05-d0_dw_0_20250917T002552_reco_stage1_20250917T104535_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039388_0534_df-s05-d0_dw_0_20250917T002552_reco_stage1_20250917T104535_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039388_0534_df-s05-d0_dw_0_20250917T002552_reco_stage1_20250917T104535_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758105936.0,
    "core.end_time": 1758105936.0,
    "core.events": [
      41088,
      41092,
      41096,
      41100,
      41104,
      41108,
      41112,
      41116,
      41120,
      41124,
      41128,
      41132,
      41136,
      41140,
      41144,
      41148,
      41152,
      41156,
      41160
    ],
    "core.event_count": 19,
    "core.first_event_number": 41088,
    "core.last_event_number": 41160,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39388
    ],
    "core.runs_subruns": [
      3938800001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039388_0534_df-s05-d0_dw_0_20250917T002552.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039388_0534_df-s05-d0_dw_0_20250917T002552_reco_stage1_20250917T104535_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758105936.0,
    "core.end_time": 1758105936.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39388
    ],
    "core.runs_subruns": [
      3938800001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039388_0534_df-s05-d0_dw_0_20250917T002552.hdf5"
    }
  ]
}
justIN time: 2025-09-19 00:44:35 UTC       justIN version: 01.05.00