justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 47409.101@dunegpschedd01.fnal.gov

Jobsub ID47409.101@dunegpschedd01.fnal.gov
Workflow ID2740
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-17 08:45:00
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2025-09-17 09:57:32
From worker nodeHostnameb9p20p3034.cern.ch
cpuinfoIntel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-17 08:46:34
Input filesvd-protodune:np02vd_raw_run039388_0524_df-s05-d1_dw_0_20250917T001415.hdf5
JobscriptExit code0
Real time1h (4187s)
CPU time1h (4084s = 97%)
Max RSS bytes2922819584 (2787 MiB)
Outputting started2025-09-17 09:56:22
Output filesvd-protodune-det-reco:np02vd_raw_run039388_0524_df-s05-d1_dw_0_20250917T001415_reco_stage1_20250917T095608_keepup.root
Finished2025-09-17 09:57:32
Saved logsjustin-logs:47409.101-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[11:52:32.979] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=37 anode=0 
[11:52:32.979] D [  glue  ] <FrameFanin:nfsp> EOS at call=37 with 8 
[11:52:32.979] D [  glue  ] frame sink sees EOS
[11:52:32.979] D [ pgraph ] <Pgrapher:> graph execution complete 
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 17.22 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 16.62 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.81 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.91 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.47 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.66 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.43 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.44 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.16 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.06 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.05 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.03 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:52:32.979] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[11:52:32.979] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[11:52:32.979] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[11:52:32.979] I [ timer  ] Timer: Total node execution : 112.95000025071204 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 62776 traces tagged "gauss"
FrameSaver: q=1.41443e+07 n=1502065 tag=gauss
wclsFrameSaver: saving 77390 traces tagged "wiener"
FrameSaver: q=1.51985e+07 n=1449802 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 927 XUs and 1164 XVs -> 53 XUVs
C:0 T:1 385 XUs and 698 XVs -> 39 XUVs
C:0 T:2 563 XUs and 516 XVs -> 16 XUVs
C:0 T:3 182 XUs and 110 XVs -> 4 XUVs
C:0 T:4 523 XUs and 968 XVs -> 30 XUVs
C:0 T:5 1106 XUs and 1427 XVs -> 72 XUVs
C:0 T:6 3013 XUs and 3889 XVs -> 72 XUVs
C:0 T:7 1074 XUs and 1343 XVs -> 51 XUVs
C:0 T:8 9542 XUs and 23831 XVs -> 2448 XUVs
C:0 T:9 15590 XUs and 26893 XVs -> 3564 XUVs
C:0 T:10 22108 XUs and 44186 XVs -> 5851 XUVs
C:0 T:11 7793 XUs and 15346 XVs -> 2304 XUVs
C:0 T:12 12151 XUs and 24166 XVs -> 2048 XUVs
C:0 T:13 1800 XUs and 1883 XVs -> 151 XUVs
C:0 T:14 2609 XUs and 2575 XVs -> 117 XUVs
C:0 T:15 3289 XUs and 4546 XVs -> 798 XUVs
17618 XUVs total
2628 collection wire objects
17618 potential space points
Neighbour search...
2588534 tests to find 1051844 neighbours
Iterating with no regularization...
Begin: 2.55366e+09
0 2.31039e+09
1 2.29208e+09
2 2.28945e+09
3 2.28857e+09
Now with regularization...
Begin: 2.27141e+09
0 2.27081e+09
17-Sep-2025 11:56:08 CEST  Closed output file "np02vd_raw_run039388_0524_df-s05-d1_dw_0_20250917T001415_reco_stage1_20250917T095608_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        148.966       217.214       337.34        200.984       46.787         19     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.0068e-05    8.7206e-05    0.000286364   7.1718e-05    4.92517e-05      19     
produce:tpcrawdecoder:PDVDTPCReader               7.34856       8.67201       11.1404       8.71686      0.769191        19     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00560229    0.00774987     0.0255965    0.00650535     0.0042961       19     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000342547   0.000603427   0.00135525    0.000472058   0.00032162       19     
produce:ophit:OpHitFinder                       0.000109861   0.000229509   0.00116698    0.000168578   0.000225999      19     
produce:opflash:OpFlashFinderVerticalDrift       4.605e-05    9.83084e-05   0.000531291   7.9273e-05    0.000103377      19     
produce:wclsdatavd:WireCellToolkit                88.9195       102.115       126.984       98.6433       9.93341        19     
produce:gaushit:GausHitFinder                     1.0788        1.93456       3.75229       1.80865      0.625423        19     
produce:nhitsfilter:NumberOfHitsFilter          0.000331342   0.000547007   0.00135359    0.000465729   0.000221605      19     
produce:reco3d:SpacePointSolver                   8.14371       18.4342       34.0085       16.7211       5.65079        19     
produce:hitpdune:DisambigFromSpacePoints         0.131532      0.345967      0.890082      0.311748      0.163795        19     
produce:pandora:StandardPandora                   34.6458       77.7883       172.682       67.6335       33.6334        19     
produce:pandoraTrack:LArPandoraTrackCreation      0.57886       2.00365       4.35746       1.72294      0.889008        19     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0194195     0.0382179     0.0604767     0.0380318    0.00896984       19     
[art]:TriggerResults:TriggerResultInserter       3.034e-05    3.97775e-05    8.184e-05    3.6244e-05    1.15369e-05      19     
end_path:out1:RootOutput                         4.924e-06    9.47837e-06   2.6039e-05     8.32e-06     4.91428e-06      19     
end_path:out1:RootOutput(write)                   5.06859       5.81918       8.03321       5.51809      0.695092        19     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4868.7 MB
  Peak resident set size usage (VmHWM): 2922.82 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039388_0524_df-s05-d1_dw_0_20250917T001415_reco_stage1_20250917T095608_keepup.root
\tHists: np02vd_raw_run039388_0524_df-s05-d1_dw_0_20250917T001415_reco_stage1_20250917T095608_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039388_0524_df-s05-d1_dw_0_20250917T001415_reco_stage1_20250917T095608_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039388_0524_df-s05-d1_dw_0_20250917T001415_reco_stage1_20250917T095608_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758102969.0,
    "core.end_time": 1758102969.0,
    "core.events": [
      40385,
      40389,
      40393,
      40397,
      40401,
      40405,
      40409,
      40413,
      40417,
      40421,
      40425,
      40429,
      40433,
      40437,
      40441,
      40445,
      40449,
      40453,
      40457
    ],
    "core.event_count": 19,
    "core.first_event_number": 40385,
    "core.last_event_number": 40457,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39388
    ],
    "core.runs_subruns": [
      3938800001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039388_0524_df-s05-d1_dw_0_20250917T001415.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039388_0524_df-s05-d1_dw_0_20250917T001415_reco_stage1_20250917T095608_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758102969.0,
    "core.end_time": 1758102969.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39388
    ],
    "core.runs_subruns": [
      3938800001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039388_0524_df-s05-d1_dw_0_20250917T001415.hdf5"
    }
  ]
}
justIN time: 2025-09-19 13:32:47 UTC       justIN version: 01.05.00