justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 41739.3@dunegpschedd02.fnal.gov

Jobsub ID41739.3@dunegpschedd02.fnal.gov
Workflow ID2867
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-18 07:48:12
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-09-18 11:28:02
From worker nodeHostnamewn2210251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-18 07:49:34
Input filesvd-protodune:np02vd_raw_run039433_0378_df-s05-d0_dw_0_20250918T065335.hdf5
JobscriptExit code0
Real time3h (12997s)
CPU time3h (12099s = 93%)
Max RSS bytes3127775232 (2982 MiB)
Outputting started2025-09-18 11:26:12
Output filesvd-protodune-det-reco:np02vd_raw_run039433_0378_df-s05-d0_dw_0_20250918T065335_reco_stage1_20250918T112558_keepup.root
Finished2025-09-18 11:28:02
Saved logsjustin-logs:41739.3-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oc ] <OmnibusSigProc:anode0sigproc0> EOS at call=97 anode=0 
[12:25:35.964] D [  glue  ] <FrameFanin:nfsp> EOS at call=97 with 8 
[12:25:35.964] D [  glue  ] frame sink sees EOS
[12:25:35.964] D [ pgraph ] <Pgrapher:> graph execution complete 
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.51 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.55 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.01 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.98 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.45 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.44 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.44 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.43 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[12:25:35.964] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[12:25:35.964] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[12:25:35.964] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[12:25:35.964] I [ timer  ] Timer: Total node execution : 27.85000065714121 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 14575 traces tagged "gauss"
FrameSaver: q=2.58199e+06 n=373828 tag=gauss
wclsFrameSaver: saving 18829 traces tagged "wiener"
FrameSaver: q=2.72303e+06 n=360778 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 2521 XUs and 2284 XVs -> 208 XUVs
C:0 T:1 814 XUs and 1026 XVs -> 62 XUVs
C:0 T:2 10101 XUs and 7400 XVs -> 553 XUVs
C:0 T:3 487 XUs and 394 XVs -> 24 XUVs
C:0 T:4 177 XUs and 128 XVs -> 1 XUVs
C:0 T:5 1092 XUs and 931 XVs -> 275 XUVs
C:0 T:6 333 XUs and 363 XVs -> 16 XUVs
C:0 T:7 234 XUs and 168 XVs -> 13 XUVs
1152 XUVs total
592 collection wire objects
1152 potential space points
Neighbour search...
70742 tests to find 29384 neighbours
Iterating with no regularization...
Begin: 1.6218e+09
0 1.5363e+09
1 1.53561e+09
Now with regularization...
Begin: 1.52734e+09
0 1.52729e+09
18-Sep-2025 12:25:58 BST  Closed output file "np02vd_raw_run039433_0378_df-s05-d0_dw_0_20250918T065335_reco_stage1_20250918T112558_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        53.5939       262.317       8590.78       78.0346       1202.86        49     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.4149e-05    0.000155627   0.00084086    0.000127289   0.00011084       49     
produce:tpcrawdecoder:PDVDTPCReader               9.03433       10.7002       24.4203       10.157        2.29948        49     
produce:triggerrawdecoder:PDVDTriggerReader4     0.372067      0.391042      0.418449      0.388717      0.0120323       49     
produce:pdvddaphne:DAPHNEReaderPDVD               3.47768       4.45649       5.70674       4.48942      0.379839        49     
produce:ophit:OpHitFinder                        0.0419129     0.0572777     0.0874161     0.0559015    0.00872313       49     
produce:opflash:OpFlashFinderVerticalDrift      0.00509722     0.0123948     0.0250578     0.0119094    0.00431149       49     
produce:wclsdatavd:WireCellToolkit                26.1283       36.8164       85.4532       32.4176       11.2962        49     
produce:gaushit:GausHitFinder                    0.444091      0.801126       2.01604      0.745969      0.318474        49     
produce:nhitsfilter:NumberOfHitsFilter          0.000100669   0.000228804   0.000726374   0.000195186   0.000106983      49     
produce:reco3d:SpacePointSolver                   2.49304       6.95669       53.0318       5.70757       7.25202        49     
produce:hitpdune:DisambigFromSpacePoints         0.0250679     0.136304       1.57599      0.0746372     0.225438        49     
produce:pandora:StandardPandora                   5.83136       198.574       8425.02       17.4552       1187.9         49     
produce:pandoraTrack:LArPandoraTrackCreation     0.113236      0.411819       4.26536      0.253618      0.603241        49     
produce:pandoraGnocalo:GnocchiCalorimetry       0.00888554     0.0191721     0.0489261     0.0167493    0.00767969       49     
[art]:TriggerResults:TriggerResultInserter      1.9557e-05    3.83701e-05   8.1142e-05    3.6658e-05    1.16583e-05      49     
end_path:out1:RootOutput                         3.787e-06    1.23229e-05   6.1887e-05     9.999e-06    9.68313e-06      49     
end_path:out1:RootOutput(write)                   2.39019       2.94774       4.0793        2.80347      0.430409        49     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5055.53 MB
  Peak resident set size usage (VmHWM): 3127.78 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039433_0378_df-s05-d0_dw_0_20250918T065335_reco_stage1_20250918T112558_keepup.root
\tHists: np02vd_raw_run039433_0378_df-s05-d0_dw_0_20250918T065335_reco_stage1_20250918T112559_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039433_0378_df-s05-d0_dw_0_20250918T065335_reco_stage1_20250918T112558_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039433_0378_df-s05-d0_dw_0_20250918T065335_reco_stage1_20250918T112558_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758194760.0,
    "core.end_time": 1758194760.0,
    "core.events": [
      108540,
      108546,
      108552,
      108558,
      108564,
      108570,
      108576,
      108582,
      108588,
      108594,
      108600,
      108606,
      108612,
      108618,
      108624,
      108630,
      108636,
      108642,
      108648,
      108654,
      108660,
      108666,
      108672,
      108678,
      108684,
      108690,
      108696,
      108702,
      108708,
      108714,
      108720,
      108726,
      108732,
      108738,
      108744,
      108750,
      108756,
      108762,
      108768,
      108774,
      108780,
      108786,
      108792,
      108798,
      108804,
      108810,
      108816,
      108822,
      108828
    ],
    "core.event_count": 49,
    "core.first_event_number": 108540,
    "core.last_event_number": 108828,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39433
    ],
    "core.runs_subruns": [
      3943300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039433_0378_df-s05-d0_dw_0_20250918T065335.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039433_0378_df-s05-d0_dw_0_20250918T065335_reco_stage1_20250918T112559_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758194760.0,
    "core.end_time": 1758194760.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39433
    ],
    "core.runs_subruns": [
      3943300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039433_0378_df-s05-d0_dw_0_20250918T065335.hdf5"
    }
  ]
}
justIN time: 2025-09-18 15:56:35 UTC       justIN version: 01.05.00