justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 37300.5@dunegpschedd02.fnal.gov

Jobsub ID37300.5@dunegpschedd02.fnal.gov
Workflow ID2365
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-10 01:51:47
SiteUK_Glasgow
EntryCLAS12_T3_UK_ScotGrid_GLA_ce04_scitok
Last heartbeat2025-09-10 04:38:07
From worker nodeHostnamewn-d21-026.beowulf.cluster
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-10 01:53:02
Input filesvd-protodune:np02vd_raw_run039349_0026_df-s05-d3_dw_0_20250909T123213.hdf5
JobscriptExit code0
Real time2h (9438s)
CPU time36m (2182s = 23%)
Max RSS bytes2255720448 (2151 MiB)
Outputting started2025-09-10 04:30:23
Output filesvd-protodune-det-reco:np02vd_raw_run039349_0026_df-s05-d3_dw_0_20250909T123213_reco_stage1_20250910T043012_keepup.root
Finished2025-09-10 04:38:07
Saved logsjustin-logs:37300.5-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

EOS at call=55 with 8 
[05:29:47.635] D [  glue  ] frame sink sees EOS
[05:29:47.635] D [ pgraph ] <Pgrapher:> graph execution complete 
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.39 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.94 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.94 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.48 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.47 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.46 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.36 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.31 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.27 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.26 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.26 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.26 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[05:29:47.635] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[05:29:47.635] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[05:29:47.635] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[05:29:47.635] I [ timer  ] Timer: Total node execution : 30.41999988630414 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 27103 traces tagged "gauss"
FrameSaver: q=4.05818e+06 n=549761 tag=gauss
wclsFrameSaver: saving 32779 traces tagged "wiener"
FrameSaver: q=4.31012e+06 n=526534 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 455 XUs and 560 XVs -> 16 XUVs
C:0 T:1 7 XUs and 12 XVs -> 0 XUVs
C:0 T:2 473 XUs and 564 XVs -> 17 XUVs
C:0 T:3 477 XUs and 331 XVs -> 21 XUVs
C:0 T:4 156 XUs and 230 XVs -> 7 XUVs
C:0 T:5 138 XUs and 204 XVs -> 7 XUVs
C:0 T:6 655 XUs and 833 XVs -> 30 XUVs
C:0 T:7 975 XUs and 748 XVs -> 34 XUVs
C:0 T:8 6882 XUs and 17733 XVs -> 1695 XUVs
C:0 T:9 699 XUs and 733 XVs -> 39 XUVs
C:0 T:10 417 XUs and 648 XVs -> 25 XUVs
C:0 T:11 678 XUs and 1042 XVs -> 55 XUVs
C:0 T:12 380 XUs and 584 XVs -> 46 XUVs
C:0 T:13 611 XUs and 700 XVs -> 41 XUVs
C:0 T:14 439 XUs and 530 XVs -> 18 XUVs
C:0 T:15 4192 XUs and 4950 XVs -> 626 XUVs
2677 XUVs total
681 collection wire objects
2677 potential space points
Neighbour search...
204483 tests to find 92248 neighbours
Iterating with no regularization...
Begin: 1.64723e+08
0 1.53649e+08
1 1.5287e+08
2 1.52707e+08
3 1.52628e+08
Now with regularization...
Begin: 1.49487e+08
0 1.49417e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
10-Sep-2025 05:30:12 BST  Closed output file "np02vd_raw_run039349_0026_df-s05-d3_dw_0_20250909T123213_reco_stage1_20250910T043012_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        163.384       334.486       3880.84       205.85        682.829        28     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.5521e-05    0.000118499   0.000199344   0.000103234   3.84065e-05      28     
produce:tpcrawdecoder:PDVDTPCReader               86.8042       249.688       3789.09       119.849       681.358        28     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0563774     0.0641448     0.149898      0.0570278     0.0177547       28     
produce:pdvddaphne:DAPHNEReaderPDVD               8.62194        13.85        23.7805       13.8897       2.93653        28     
produce:ophit:OpHitFinder                        0.0267288     0.0360052     0.0422787     0.0357645    0.00342869       28     
produce:opflash:OpFlashFinderVerticalDrift      0.00464424    0.00895085     0.0133377    0.00872719    0.00203179       28     
produce:wclsdatavd:WireCellToolkit                30.1371       32.516        37.8032       32.7128       1.55868        28     
produce:gaushit:GausHitFinder                    0.522699      0.720763       0.90779      0.691802      0.114127        28     
produce:nhitsfilter:NumberOfHitsFilter          0.000135705   0.000225208   0.000370896   0.000214678   5.88791e-05      28     
produce:reco3d:SpacePointSolver                   4.55099       7.4951        11.5358       7.1029        1.81544        28     
produce:hitpdune:DisambigFromSpacePoints         0.0506913     0.106004      0.168903      0.105777      0.0277186       28     
produce:pandora:StandardPandora                   14.1094       26.5074       40.5768       25.8485       7.92441        28     
produce:pandoraTrack:LArPandoraTrackCreation     0.186251      0.347675      0.552925      0.339825      0.0911411       28     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0103034     0.0175327     0.0240423     0.0175739    0.00387777       28     
[art]:TriggerResults:TriggerResultInserter      1.0961e-05    1.83267e-05   5.1667e-05    1.66005e-05   8.28218e-06      28     
end_path:out1:RootOutput                         2.675e-06    6.03011e-06   1.3275e-05     5.39e-06     2.43879e-06      28     
end_path:out1:RootOutput(write)                   2.86649       3.09273       3.39498       3.08835      0.125401        28     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4239.03 MB
  Peak resident set size usage (VmHWM): 2255.72 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039349_0026_df-s05-d3_dw_0_20250909T123213_reco_stage1_20250910T043012_keepup.root
\tHists: np02vd_raw_run039349_0026_df-s05-d3_dw_0_20250909T123213_reco_stage1_20250910T043012_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039349_0026_df-s05-d3_dw_0_20250909T123213_reco_stage1_20250910T043012_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039349_0026_df-s05-d3_dw_0_20250909T123213_reco_stage1_20250910T043012_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757478613.0,
    "core.end_time": 1757478613.0,
    "core.events": [
      14437,
      14457,
      14477,
      14497,
      14517,
      14537,
      14557,
      14577,
      14597,
      14617,
      14637,
      14657,
      14677,
      14697,
      14717,
      14737,
      14757,
      14777,
      14797,
      14817,
      14837,
      14857,
      14877,
      14897,
      14917,
      14937,
      14957,
      14977
    ],
    "core.event_count": 28,
    "core.first_event_number": 14437,
    "core.last_event_number": 14977,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39349
    ],
    "core.runs_subruns": [
      3934900001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039349_0026_df-s05-d3_dw_0_20250909T123213.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039349_0026_df-s05-d3_dw_0_20250909T123213_reco_stage1_20250910T043012_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757478613.0,
    "core.end_time": 1757478613.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39349
    ],
    "core.runs_subruns": [
      3934900001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039349_0026_df-s05-d3_dw_0_20250909T123213.hdf5"
    }
  ]
}
justIN time: 2025-09-19 02:30:37 UTC       justIN version: 01.05.00