justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237284.168@dunegpschedd02.fnal.gov

Jobsub ID237284.168@dunegpschedd02.fnal.gov
Workflow ID9584
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-04 13:48:09
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2025-11-04 16:21:47
From worker nodeHostnamewn-f01-14-01-02.cr.cnaf.infn.it
cpuinfoIntel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-04 13:50:01
Input filesvd-protodune:np02vd_raw_run040348_1064_df-s04-d2_dw_0_20251104T095621.hdf5
JobscriptExit code0
Real time2h (8973s)
CPU time2h (8700s = 96%)
Max RSS bytes5877149696 (5604 MiB)
Outputting started2025-11-04 16:19:35
Output filesvd-protodune-det-reco:np02vd_raw_run040348_1064_df-s04-d2_dw_0_20251104T095621_reco_stage1_20251104T161924_keepup.root
Finished2025-11-04 16:21:47
Saved logsjustin-logs:237284.168-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

3.703] D [  glue  ] <FrameFanin:nfsp> EOS at call=45 with 8 
[17:18:23.703] D [  glue  ] frame sink sees EOS
[17:18:23.703] D [ pgraph ] <Pgrapher:> graph execution complete 
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 16.25 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 15.49 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 15.37 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.95 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.76 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.31 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.94 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.62 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.6 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.59 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.59 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.58 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.05 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:18:23.703] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:18:23.703] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[17:18:23.703] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[17:18:23.703] I [ timer  ] Timer: Total node execution : 121.1699995342642 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 42283 traces tagged "gauss"
FrameSaver: q=1.23962e+07 n=981427 tag=gauss
wclsFrameSaver: saving 53502 traces tagged "wiener"
FrameSaver: q=1.32953e+07 n=940830 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 10267 XUs and 8094 XVs -> 595 XUVs
C:0 T:1 197 XUs and 331 XVs -> 14 XUVs
C:0 T:2 1343 XUs and 1541 XVs -> 45 XUVs
C:0 T:3 722 XUs and 949 XVs -> 29 XUVs
C:0 T:4 79 XUs and 194 XVs -> 0 XUVs
C:0 T:5 356 XUs and 604 XVs -> 9 XUVs
C:0 T:6 331 XUs and 975 XVs -> 15 XUVs
C:0 T:7 781 XUs and 932 XVs -> 29 XUVs
C:0 T:8 610 XUs and 739 XVs -> 34 XUVs
C:0 T:9 2738 XUs and 5608 XVs -> 1045 XUVs
C:0 T:10 3593 XUs and 7613 XVs -> 1097 XUVs
C:0 T:11 7284 XUs and 12978 XVs -> 2333 XUVs
C:0 T:12 13608 XUs and 21638 XVs -> 3283 XUVs
C:0 T:13 625 XUs and 943 XVs -> 155 XUVs
C:0 T:14 410 XUs and 505 XVs -> 58 XUVs
C:0 T:15 1854 XUs and 1942 XVs -> 332 XUVs
9073 XUVs total
1746 collection wire objects
9073 potential space points
Neighbour search...
991467 tests to find 448644 neighbours
Iterating with no regularization...
Begin: 8.65088e+09
0 7.80113e+09
1 7.7499e+09
2 7.74523e+09
Now with regularization...
Begin: 7.5911e+09
0 7.5895e+09
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
04-Nov-2025 17:19:24 CET  Closed output file "np02vd_raw_run040348_1064_df-s04-d2_dw_0_20251104T095621_reco_stage1_20251104T161924_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        139.06        387.576        2493         218.868       507.871        23     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.7557e-05    0.000116228   0.000224893   0.00010094    4.00412e-05      23     
produce:tpcrawdecoder:PDVDTPCReader               10.2039       12.6519       15.5975       12.3555        1.28          23     
produce:triggerrawdecoder:PDVDTriggerReader4     0.308028      0.327243      0.392912      0.323131       0.0197         23     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000413807   0.000610244   0.00121616    0.000592542   0.00014494       23     
produce:ophit:OpHitFinder                       6.5476e-05    0.000152941   0.00107548    0.000108945   0.000198184      23     
produce:opflash:OpFlashFinderVerticalDrift      5.6504e-05    0.000107981   0.000595517   8.4979e-05    0.000104871      23     
produce:wclsdatavd:WireCellToolkit                83.6431       139.568       243.229       136.744       41.5378        23     
produce:gaushit:GausHitFinder                     1.09249       2.1202        4.75537       1.99621      0.848136        23     
produce:nhitsfilter:NumberOfHitsFilter           0.0002837    0.000494061   0.000888189   0.000469425   0.000165258      23     
produce:reco3d:SpacePointSolver                   8.86192       18.373        55.6667       15.4073       10.9222        23     
produce:hitpdune:DisambigFromSpacePoints         0.143817      0.546589       2.92473      0.328193      0.641068        23     
produce:pandora:StandardPandora                   26.1868       203.697       2200.8        44.0808       469.315        23     
produce:pandoraTrack:LArPandoraTrackCreation      1.02567       3.24509       13.8597       2.37613       2.77327        23     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0289305     0.0561101     0.0781002     0.058191      0.0146481       23     
[art]:TriggerResults:TriggerResultInserter      2.4982e-05    4.29175e-05   0.000131448    3.553e-05    2.11134e-05      23     
end_path:out1:RootOutput                         7.977e-06    1.2667e-05    5.1297e-05    1.0088e-05    8.94854e-06      23     
end_path:out1:RootOutput(write)                   4.20629       6.92577       10.3514       7.19528       2.02422        23     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7783.57 MB
  Peak resident set size usage (VmHWM): 5877.15 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040348_1064_df-s04-d2_dw_0_20251104T095621_reco_stage1_20251104T161924_keepup.root
\tHists: np02vd_raw_run040348_1064_df-s04-d2_dw_0_20251104T095621_reco_stage1_20251104T161924_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040348_1064_df-s04-d2_dw_0_20251104T095621_reco_stage1_20251104T161924_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040348_1064_df-s04-d2_dw_0_20251104T095621_reco_stage1_20251104T161924_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762273165.0,
    "core.end_time": 1762273165.0,
    "core.events": [
      98378,
      98382,
      98386,
      98390,
      98394,
      98398,
      98402,
      98406,
      98410,
      98414,
      98418,
      98422,
      98426,
      98430,
      98434,
      98438,
      98442,
      98446,
      98450,
      98454,
      98458,
      98462,
      98466
    ],
    "core.event_count": 23,
    "core.first_event_number": 98378,
    "core.last_event_number": 98466,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40348
    ],
    "core.runs_subruns": [
      4034800001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040348_1064_df-s04-d2_dw_0_20251104T095621.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040348_1064_df-s04-d2_dw_0_20251104T095621_reco_stage1_20251104T161924_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762273165.0,
    "core.end_time": 1762273165.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40348
    ],
    "core.runs_subruns": [
      4034800001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040348_1064_df-s04-d2_dw_0_20251104T095621.hdf5"
    }
  ]
}
justIN time: 2025-11-05 09:17:05 UTC       justIN version: 01.05.01