justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237115.23@dunegpschedd02.fnal.gov

Jobsub ID237115.23@dunegpschedd02.fnal.gov
Workflow ID9538
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-04 06:23:46
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2025-11-04 08:49:40
From worker nodeHostnameb9p02p7290.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-04 06:25:29
Input filesvd-protodune:np02vd_raw_run040346_3632_df-s04-d2_dw_0_20251103T151123.hdf5
JobscriptExit code0
Real time2h (8515s)
CPU time2h (8251s = 96%)
Max RSS bytes4231880704 (4035 MiB)
Outputting started2025-11-04 08:47:25
Output filesvd-protodune-det-reco:np02vd_raw_run040346_3632_df-s04-d2_dw_0_20251103T151123_reco_stage1_20251104T084714_keepup.root
Finished2025-11-04 08:49:40
Saved logsjustin-logs:237115.23-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

sigproc0> EOS at call=45 anode=0 
[08:16:30.122] D [  glue  ] <FrameFanin:nfsp> EOS at call=45 with 8 
[08:16:30.123] D [  glue  ] frame sink sees EOS
[08:16:30.123] D [ pgraph ] <Pgrapher:> graph execution complete 
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 39.13 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 25.84 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.41 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.76 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.74 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.64 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.62 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.26 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.26 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.25 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.25 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.24 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[08:16:30.123] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[08:16:30.123] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[08:16:30.123] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[08:16:30.123] I [ timer  ] Timer: Total node execution : 136.47000150941312 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 66659 traces tagged "gauss"
FrameSaver: q=2.7303e+07 n=1639071 tag=gauss
wclsFrameSaver: saving 87873 traces tagged "wiener"
FrameSaver: q=3.11704e+07 n=1580165 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 20915 XUs and 22268 XVs -> 1545 XUVs
C:0 T:1 116965 XUs and 82534 XVs -> 9836 XUVs
C:0 T:2 146893 XUs and 123460 XVs -> 13564 XUVs
C:0 T:3 9185 XUs and 6344 XVs -> 332 XUVs
C:0 T:4 788 XUs and 1096 XVs -> 29 XUVs
C:0 T:5 2212 XUs and 1175 XVs -> 256 XUVs
C:0 T:6 550 XUs and 898 XVs -> 36 XUVs
C:0 T:7 117 XUs and 139 XVs -> 1 XUVs
C:0 T:8 294 XUs and 440 XVs -> 37 XUVs
C:0 T:9 161 XUs and 321 XVs -> 10 XUVs
C:0 T:10 619 XUs and 738 XVs -> 46 XUVs
C:0 T:11 1338 XUs and 1611 XVs -> 212 XUVs
C:0 T:12 751 XUs and 809 XVs -> 46 XUVs
C:0 T:13 687 XUs and 943 XVs -> 67 XUVs
C:0 T:14 1367 XUs and 1988 XVs -> 420 XUVs
C:0 T:15 738 XUs and 998 XVs -> 145 XUVs
26582 XUVs total
6396 collection wire objects
26582 potential space points
Neighbour search...
2040134 tests to find 421404 neighbours
Iterating with no regularization...
Begin: 1.59308e+11
0 1.49189e+11
1 1.47669e+11
2 1.47544e+11
Now with regularization...
Begin: 1.46933e+11
0 1.46896e+11
04-Nov-2025 09:47:14 CET  Closed output file "np02vd_raw_run040346_3632_df-s04-d2_dw_0_20251103T151123_reco_stage1_20251104T084714_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        80.163        367.758       5590.36       114.393       1114.91        23     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.9411e-05    0.000134222   0.000771728   8.6021e-05    0.000142871      23     
produce:tpcrawdecoder:PDVDTPCReader               5.17782       6.17187        7.293        6.07495       0.56687        23     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0113761     0.0287509     0.069022      0.022308      0.0185224       23     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000326313   0.00042188    0.000759488   0.000397144   0.000106344      23     
produce:ophit:OpHitFinder                       6.6371e-05    0.000139278   0.000648546   0.000118151   0.000111311      23     
produce:opflash:OpFlashFinderVerticalDrift      5.2671e-05     9.423e-05    0.000415244    8.136e-05    6.96588e-05      23     
produce:wclsdatavd:WireCellToolkit                52.3385       73.5167       140.714       62.9612       22.0256        23     
produce:gaushit:GausHitFinder                     0.84173       1.35839       2.1909        1.29711      0.341133        23     
produce:nhitsfilter:NumberOfHitsFilter          0.000164811   0.000292164   0.000497845   0.000262873   7.31353e-05      23     
produce:reco3d:SpacePointSolver                   4.8248        12.2717        38.03        10.8608       6.29296        23     
produce:hitpdune:DisambigFromSpacePoints         0.0644698     0.230065       1.05059      0.182598      0.192247        23     
produce:pandora:StandardPandora                   11.5936       268.164       5395.46       26.1045       1093.93        23     
produce:pandoraTrack:LArPandoraTrackCreation     0.302975      0.929093       2.31092      0.844897      0.451839        23     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0155767     0.0342412     0.0498548      0.03535     0.00881244       23     
[art]:TriggerResults:TriggerResultInserter       2.023e-05    3.2202e-05    7.6131e-05    3.0771e-05    1.15366e-05      23     
end_path:out1:RootOutput                         3.95e-06     9.33361e-06    2.633e-05     9.27e-06     4.08036e-06      23     
end_path:out1:RootOutput(write)                   3.69283       5.0066        5.9266        5.01682      0.637596        23     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6310.22 MB
  Peak resident set size usage (VmHWM): 4231.88 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040346_3632_df-s04-d2_dw_0_20251103T151123_reco_stage1_20251104T084714_keepup.root
\tHists: np02vd_raw_run040346_3632_df-s04-d2_dw_0_20251103T151123_reco_stage1_20251104T084714_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040346_3632_df-s04-d2_dw_0_20251103T151123_reco_stage1_20251104T084714_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040346_3632_df-s04-d2_dw_0_20251103T151123_reco_stage1_20251104T084714_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762246035.0,
    "core.end_time": 1762246035.0,
    "core.events": [
      335318,
      335322,
      335326,
      335330,
      335334,
      335338,
      335342,
      335346,
      335350,
      335354,
      335358,
      335362,
      335366,
      335370,
      335374,
      335378,
      335382,
      335386,
      335390,
      335394,
      335398,
      335402,
      335406
    ],
    "core.event_count": 23,
    "core.first_event_number": 335318,
    "core.last_event_number": 335406,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3632_df-s04-d2_dw_0_20251103T151123.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040346_3632_df-s04-d2_dw_0_20251103T151123_reco_stage1_20251104T084714_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762246035.0,
    "core.end_time": 1762246035.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3632_df-s04-d2_dw_0_20251103T151123.hdf5"
    }
  ]
}
justIN time: 2025-11-05 08:41:12 UTC       justIN version: 01.05.01