justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236578.179@dunegpschedd02.fnal.gov

Jobsub ID236578.179@dunegpschedd02.fnal.gov
Workflow ID9538
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-03 08:42:38
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce16-multicore
Last heartbeat2025-11-03 12:12:21
From worker nodeHostnamehnode78.pic.es
cpuinfoAMD EPYC 7443P 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 10:21:49
Input filesvd-protodune:np02vd_raw_run040346_3247_df-s04-d1_dw_0_20251103T093132.hdf5
JobscriptExit code0
Real time1h (6514s)
CPU time38m (2336s = 35%)
Max RSS bytes3184185344 (3036 MiB)
Outputting started2025-11-03 12:10:24
Output filesvd-protodune-det-reco:np02vd_raw_run040346_3247_df-s04-d1_dw_0_20251103T093132_reco_stage1_20251103T121014_keepup.root
Finished2025-11-03 12:12:21
Saved logsjustin-logs:236578.179-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] <Resampler:resmp0> EOS at call=43 
[13:09:51.030] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=43 anode=0 
[13:09:51.030] D [  glue  ] <FrameFanin:nfsp> EOS at call=43 with 8 
[13:09:51.030] D [  glue  ] frame sink sees EOS
[13:09:51.030] D [ pgraph ] <Pgrapher:> graph execution complete 
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.84 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.79 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.75 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.64 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.26 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.15 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.05 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.83 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.04 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.04 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.02 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::Aux::Resampler : 1 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[13:09:51.030] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[13:09:51.030] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[13:09:51.030] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[13:09:51.030] I [ timer  ] Timer: Total node execution : 55.43000032380223 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 33393 traces tagged "gauss"
FrameSaver: q=7.05575e+06 n=660771 tag=gauss
wclsFrameSaver: saving 41756 traces tagged "wiener"
FrameSaver: q=7.50961e+06 n=622825 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 562 XUs and 624 XVs -> 44 XUVs
C:0 T:1 782 XUs and 885 XVs -> 37 XUVs
C:0 T:2 286 XUs and 262 XVs -> 11 XUVs
C:0 T:3 1769 XUs and 2275 XVs -> 88 XUVs
C:0 T:4 127 XUs and 141 XVs -> 8 XUVs
C:0 T:5 241 XUs and 351 XVs -> 18 XUVs
C:0 T:6 529 XUs and 1057 XVs -> 38 XUVs
C:0 T:7 28 XUs and 32 XVs -> 0 XUVs
C:0 T:8 1078 XUs and 994 XVs -> 48 XUVs
C:0 T:9 1526 XUs and 772 XVs -> 55 XUVs
C:0 T:10 113 XUs and 158 XVs -> 5 XUVs
C:0 T:11 1676 XUs and 1328 XVs -> 93 XUVs
C:0 T:12 578 XUs and 702 XVs -> 54 XUVs
C:0 T:13 467 XUs and 707 XVs -> 54 XUVs
C:0 T:14 838 XUs and 1048 XVs -> 69 XUVs
C:0 T:15 574 XUs and 812 XVs -> 45 XUVs
667 XUVs total
547 collection wire objects
667 potential space points
Neighbour search...
3997 tests to find 1976 neighbours
Iterating with no regularization...
Begin: 9.64496e+08
0 9.54984e+08
1 9.54914e+08
Now with regularization...
Begin: 9.49097e+08
0 9.49095e+08
03-Nov-2025 13:10:14 CET  Closed output file "np02vd_raw_run040346_3247_df-s04-d1_dw_0_20251103T093132_reco_stage1_20251103T121014_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        179.354       293.225       753.355       249.96        144.013        22     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.1061e-05    6.75748e-05   0.000120453   6.40165e-05    1.588e-05       22     
produce:tpcrawdecoder:PDVDTPCReader               78.0647       172.76        568.448       140.916       107.862        22     
produce:triggerrawdecoder:PDVDTriggerReader4      1.07284       6.9368        125.406       1.22823       25.8529        22     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000283476   0.000342427   0.000722007   0.000320613   8.63375e-05      22     
produce:ophit:OpHitFinder                       5.3751e-05    8.38219e-05   0.000523562   6.1581e-05    9.63042e-05      22     
produce:opflash:OpFlashFinderVerticalDrift      4.0931e-05    6.30233e-05   0.000354648   4.9161e-05    6.38381e-05      22     
produce:wclsdatavd:WireCellToolkit                37.5164       73.7056       316.076       59.6359       54.9227        22     
produce:gaushit:GausHitFinder                    0.566577       1.04165       1.67498       1.05922      0.333225        22     
produce:nhitsfilter:NumberOfHitsFilter          0.000140404   0.00023264    0.000318537   0.000238701   4.70863e-05      22     
produce:reco3d:SpacePointSolver                   5.82192       9.64922       13.9414       8.7532        2.43055        22     
produce:hitpdune:DisambigFromSpacePoints         0.0688771     0.147907      0.285954      0.140614      0.0633659       22     
produce:pandora:StandardPandora                   10.6792       23.714        51.5072       21.9091       11.2864        22     
produce:pandoraTrack:LArPandoraTrackCreation      0.40885      0.763244       1.40523      0.744889      0.277824        22     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0181116     0.0309229     0.0443537     0.0328387    0.00856398       22     
[art]:TriggerResults:TriggerResultInserter       1.185e-05    2.04522e-05   4.0411e-05    1.90005e-05   6.88773e-06      22     
end_path:out1:RootOutput                         2.97e-06     4.50459e-06    1.475e-05     3.935e-06    2.36911e-06      22     
end_path:out1:RootOutput(write)                   3.32384       4.44818       5.87688       4.68811      0.656934        22     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5215.17 MB
  Peak resident set size usage (VmHWM): 3184.19 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040346_3247_df-s04-d1_dw_0_20251103T093132_reco_stage1_20251103T121014_keepup.root
\tHists: np02vd_raw_run040346_3247_df-s04-d1_dw_0_20251103T093132_reco_stage1_20251103T121014_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040346_3247_df-s04-d1_dw_0_20251103T093132_reco_stage1_20251103T121014_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040346_3247_df-s04-d1_dw_0_20251103T093132_reco_stage1_20251103T121014_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762171815.0,
    "core.end_time": 1762171815.0,
    "core.events": [
      299841,
      299845,
      299849,
      299853,
      299857,
      299861,
      299865,
      299869,
      299873,
      299877,
      299881,
      299885,
      299889,
      299893,
      299897,
      299901,
      299905,
      299909,
      299913,
      299917,
      299921,
      299925
    ],
    "core.event_count": 22,
    "core.first_event_number": 299841,
    "core.last_event_number": 299925,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3247_df-s04-d1_dw_0_20251103T093132.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040346_3247_df-s04-d1_dw_0_20251103T093132_reco_stage1_20251103T121014_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762171815.0,
    "core.end_time": 1762171815.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3247_df-s04-d1_dw_0_20251103T093132.hdf5"
    }
  ]
}
justIN time: 2025-11-04 01:31:04 UTC       justIN version: 01.05.01