justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 42218.157@dunegpschedd01.fnal.gov

Jobsub ID42218.157@dunegpschedd01.fnal.gov
Workflow ID2325
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-07 10:17:55
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce01
Last heartbeat2025-09-07 11:24:49
From worker nodeHostnameccwcondor0055
cpuinfoAMD EPYC 9334 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-07 10:18:44
Input filesvd-protodune:np02vd_raw_run039324_1772_df-s03-d1_dw_0_20250907T052607.hdf5
JobscriptExit code0
Real time1h (3867s)
CPU time1h (3637s = 94%)
Max RSS bytes3061985280 (2920 MiB)
Outputting started2025-09-07 11:23:11
Output filesvd-protodune-det-reco:np02vd_raw_run039324_1772_df-s03-d1_dw_0_20250907T052607_reco_stage1_20250907T112259_keepup.root
Finished2025-09-07 11:24:49
Saved logsjustin-logs:42218.157-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e  ] frame sink sees EOS
[13:21:43.704] D [ pgraph ] <Pgrapher:> graph execution complete 
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.16 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.87 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.16 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.9 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.86 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.83 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.51 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.11 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.37 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.34 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.31 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.29 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:21:43.704] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[13:21:43.704] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[13:21:43.704] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[13:21:43.704] I [ timer  ] Timer: Total node execution : 44.75000009685755 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 45077 traces tagged "gauss"
FrameSaver: q=8.30599e+06 n=954941 tag=gauss
wclsFrameSaver: saving 56776 traces tagged "wiener"
FrameSaver: q=8.96307e+06 n=905515 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 364 XUs and 498 XVs -> 24 XUVs
C:0 T:1 1287 XUs and 1656 XVs -> 75 XUVs
C:0 T:2 988 XUs and 888 XVs -> 36 XUVs
C:0 T:3 384 XUs and 599 XVs -> 23 XUVs
C:0 T:4 122 XUs and 158 XVs -> 5 XUVs
C:0 T:5 435 XUs and 675 XVs -> 30 XUVs
C:0 T:6 216 XUs and 318 XVs -> 13 XUVs
C:0 T:7 381 XUs and 333 XVs -> 19 XUVs
C:0 T:8 31738 XUs and 66045 XVs -> 11924 XUVs
C:0 T:9 9422 XUs and 19402 XVs -> 3020 XUVs
C:0 T:10 6398 XUs and 9177 XVs -> 1663 XUVs
C:0 T:11 1678 XUs and 2113 XVs -> 81 XUVs
C:0 T:12 4971 XUs and 6032 XVs -> 441 XUVs
C:0 T:13 1781 XUs and 976 XVs -> 97 XUVs
C:0 T:14 1803 XUs and 2544 XVs -> 76 XUVs
C:0 T:15 1711 XUs and 910 XVs -> 90 XUVs
17617 XUVs total
1974 collection wire objects
17617 potential space points
Neighbour search...
2886587 tests to find 1205086 neighbours
Iterating with no regularization...
Begin: 4.45308e+08
0 3.77885e+08
1 3.7343e+08
2 3.7237e+08
3 3.71953e+08
4 3.71742e+08
Now with regularization...
Begin: 3.61846e+08
0 3.61629e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
07-Sep-2025 13:22:59 CEST  Closed output file "np02vd_raw_run039324_1772_df-s03-d1_dw_0_20250907T052607_reco_stage1_20250907T112259_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        67.7892       146.462       281.346       124.355       58.7649        26     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.8263e-05    8.95859e-05   0.000140582   8.56635e-05   1.51958e-05      26     
produce:tpcrawdecoder:PDVDTPCReader               5.49119       6.85953       14.4771       6.2247        1.89502        26     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00736103     0.010218      0.0313307    0.00769751    0.00546724       26     
produce:pdvddaphne:DAPHNEReaderPDVD               2.30635       2.67749       3.26411       2.72077      0.223721        26     
produce:ophit:OpHitFinder                        0.0249906     0.0465436     0.100057      0.0457502     0.0117775       26     
produce:opflash:OpFlashFinderVerticalDrift      0.00291154     0.0124242     0.0258112     0.0127021    0.00442489       26     
produce:wclsdatavd:WireCellToolkit                 35.87        48.974        82.2312       45.8976       10.7527        26     
produce:gaushit:GausHitFinder                    0.576383       1.34139       1.88974       1.33625      0.347359        26     
produce:nhitsfilter:NumberOfHitsFilter          0.000180912   0.000294394   0.000617769   0.000285129   8.30197e-05      26     
produce:reco3d:SpacePointSolver                   4.99381       13.1747       21.1555       12.5539       4.47158        26     
produce:hitpdune:DisambigFromSpacePoints         0.0696676     0.236207      0.436868      0.211465      0.101396        26     
produce:pandora:StandardPandora                   13.128        67.5357       189.245       47.7001       47.7604        26     
produce:pandoraTrack:LArPandoraTrackCreation     0.299889      0.818415       2.10715      0.738281       0.45399        26     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0164853     0.0305457     0.0431139     0.0319225    0.00746309       26     
[art]:TriggerResults:TriggerResultInserter      1.6625e-05    3.37543e-05   7.1788e-05    2.95795e-05   1.26394e-05      26     
end_path:out1:RootOutput                         4.327e-06    9.76665e-06   3.5423e-05     8.824e-06    6.26965e-06      26     
end_path:out1:RootOutput(write)                   3.99726       4.69077       6.89448       4.52037      0.758395        26     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4991.02 MB
  Peak resident set size usage (VmHWM): 3061.99 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_1772_df-s03-d1_dw_0_20250907T052607_reco_stage1_20250907T112259_keepup.root
\tHists: np02vd_raw_run039324_1772_df-s03-d1_dw_0_20250907T052607_reco_stage1_20250907T112259_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_1772_df-s03-d1_dw_0_20250907T052607_reco_stage1_20250907T112259_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039324_1772_df-s03-d1_dw_0_20250907T052607_reco_stage1_20250907T112259_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757244180.0,
    "core.end_time": 1757244180.0,
    "core.events": [
      928767,
      928787,
      928807,
      928827,
      928847,
      928867,
      928887,
      928907,
      928927,
      928947,
      928967,
      928987,
      929007,
      929027,
      929047,
      929067,
      929087,
      929107,
      929127,
      929147,
      929167,
      929187,
      929207,
      929227,
      929247,
      929267
    ],
    "core.event_count": 26,
    "core.first_event_number": 928767,
    "core.last_event_number": 929267,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39324
    ],
    "core.runs_subruns": [
      3932400001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039324_1772_df-s03-d1_dw_0_20250907T052607.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039324_1772_df-s03-d1_dw_0_20250907T052607_reco_stage1_20250907T112259_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757244180.0,
    "core.end_time": 1757244180.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39324
    ],
    "core.runs_subruns": [
      3932400001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039324_1772_df-s03-d1_dw_0_20250907T052607.hdf5"
    }
  ]
}
justIN time: 2025-09-19 07:31:54 UTC       justIN version: 01.05.00