justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 41150.140@dunegpschedd02.fnal.gov

Jobsub ID41150.140@dunegpschedd02.fnal.gov
Workflow ID2815
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-17 18:11:32
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk_multicore
Last heartbeat2025-09-17 20:40:37
From worker nodeHostnamewn-sate-028.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-17 19:28:57
Input filesvd-protodune:np02vd_raw_run039388_0446_df-s05-d3_dw_0_20250916T223516.hdf5
JobscriptExit code0
Real time1h (4234s)
CPU time1h (4017s = 94%)
Max RSS bytes3469443072 (3308 MiB)
Outputting started2025-09-17 20:39:31
Output filesvd-protodune-det-reco:np02vd_raw_run039388_0446_df-s05-d3_dw_0_20250916T223516_reco_stage1_20250917T203922_keepup.root
Finished2025-09-17 20:40:37
Saved logsjustin-logs:41150.140-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

see EOS at call=18 
[22:38:08.850] D [  aux   ] <Resampler:resmp0> EOS at call=37 
[22:38:08.850] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=37 anode=0 
[22:38:08.850] D [  glue  ] <FrameFanin:nfsp> EOS at call=37 with 8 
[22:38:08.850] D [  glue  ] frame sink sees EOS
[22:38:08.850] D [ pgraph ] <Pgrapher:> graph execution complete 
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.99 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.14 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.11 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.73 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.62 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.29 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.2 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.09 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.05 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.05 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.04 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.04 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[22:38:08.850] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[22:38:08.850] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[22:38:08.850] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[22:38:08.850] I [ timer  ] Timer: Total node execution : 74.38999895006418 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 43256 traces tagged "gauss"
FrameSaver: q=6.38911e+06 n=869910 tag=gauss
wclsFrameSaver: saving 52544 traces tagged "wiener"
FrameSaver: q=6.71106e+06 n=834609 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1153 XUs and 1559 XVs -> 74 XUVs
C:0 T:1 520 XUs and 522 XVs -> 25 XUVs
C:0 T:2 1249 XUs and 1298 XVs -> 47 XUVs
C:0 T:3 261 XUs and 285 XVs -> 20 XUVs
C:0 T:4 453 XUs and 693 XVs -> 18 XUVs
C:0 T:5 204 XUs and 340 XVs -> 6 XUVs
C:0 T:6 182 XUs and 258 XVs -> 15 XUVs
C:0 T:7 2538 XUs and 2573 XVs -> 113 XUVs
C:0 T:8 847 XUs and 1158 XVs -> 69 XUVs
C:0 T:9 526 XUs and 591 XVs -> 40 XUVs
C:0 T:10 622 XUs and 583 XVs -> 38 XUVs
C:0 T:11 504 XUs and 938 XVs -> 33 XUVs
C:0 T:12 4326 XUs and 3860 XVs -> 154 XUVs
C:0 T:13 1642 XUs and 1395 XVs -> 93 XUVs
C:0 T:14 2318 XUs and 2137 XVs -> 70 XUVs
C:0 T:15 966 XUs and 1075 XVs -> 90 XUVs
905 XUVs total
815 collection wire objects
905 potential space points
Neighbour search...
3617 tests to find 1700 neighbours
Iterating with no regularization...
Begin: 2.67422e+08
0 2.63807e+08
1 2.63738e+08
Now with regularization...
Begin: 2.61048e+08
0 2.61046e+08
17-Sep-2025 22:39:22 CEST  Closed output file "np02vd_raw_run039388_0446_df-s05-d3_dw_0_20250916T223516_reco_stage1_20250917T203922_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        159.907       220.128       324.334       222.265       46.8039        19     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.9752e-05    7.77834e-05   0.00015951    6.5603e-05    2.39352e-05      19     
produce:tpcrawdecoder:PDVDTPCReader               14.6541       15.0541       15.4801       15.057       0.209073        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.291611      0.294978      0.311218      0.293632     0.00438965       19     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000306446   0.000344024   0.00061691    0.000327235   6.66709e-05      19     
produce:ophit:OpHitFinder                       0.000124604   0.000198729   0.000774696   0.000165801   0.000142256      19     
produce:opflash:OpFlashFinderVerticalDrift      5.1397e-05    7.55356e-05   0.000323218   5.8961e-05    5.95543e-05      19     
produce:wclsdatavd:WireCellToolkit                75.0327       84.834        108.203       83.2999       8.70673        19     
produce:gaushit:GausHitFinder                     1.09655       1.62575       1.98309       1.57772      0.281104        19     
produce:nhitsfilter:NumberOfHitsFilter          0.000288793   0.000387621   0.000681942   0.000387158   8.23316e-05      19     
produce:reco3d:SpacePointSolver                   11.8908       19.4483       27.3497       18.8178       5.09321        19     
produce:hitpdune:DisambigFromSpacePoints         0.147438      0.292602      0.544058      0.285244      0.110486        19     
produce:pandora:StandardPandora                   37.2246       91.0191       164.331       88.453         35.99         19     
produce:pandoraTrack:LArPandoraTrackCreation     0.761966       1.68109       2.81662       1.51885      0.645114        19     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0252012     0.0373643     0.0498019     0.0370323    0.00734137       19     
[art]:TriggerResults:TriggerResultInserter      1.7402e-05    2.61623e-05   7.2295e-05    2.2532e-05    1.17541e-05      19     
end_path:out1:RootOutput                         3.616e-06    7.84111e-06   2.5528e-05     7.725e-06    4.53894e-06      19     
end_path:out1:RootOutput(write)                   5.45113       5.7883        6.15378       5.81848      0.208296        19     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5355.75 MB
  Peak resident set size usage (VmHWM): 3469.44 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039388_0446_df-s05-d3_dw_0_20250916T223516_reco_stage1_20250917T203922_keepup.root
\tHists: np02vd_raw_run039388_0446_df-s05-d3_dw_0_20250916T223516_reco_stage1_20250917T203922_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039388_0446_df-s05-d3_dw_0_20250916T223516_reco_stage1_20250917T203922_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039388_0446_df-s05-d3_dw_0_20250916T223516_reco_stage1_20250917T203922_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758141563.0,
    "core.end_time": 1758141563.0,
    "core.events": [
      34319,
      34323,
      34327,
      34331,
      34335,
      34339,
      34343,
      34347,
      34351,
      34355,
      34359,
      34363,
      34367,
      34371,
      34375,
      34379,
      34383,
      34387,
      34391
    ],
    "core.event_count": 19,
    "core.first_event_number": 34319,
    "core.last_event_number": 34391,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39388
    ],
    "core.runs_subruns": [
      3938800001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039388_0446_df-s05-d3_dw_0_20250916T223516.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039388_0446_df-s05-d3_dw_0_20250916T223516_reco_stage1_20250917T203922_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758141563.0,
    "core.end_time": 1758141563.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39388
    ],
    "core.runs_subruns": [
      3938800001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039388_0446_df-s05-d3_dw_0_20250916T223516.hdf5"
    }
  ]
}
justIN time: 2025-09-18 23:45:21 UTC       justIN version: 01.05.00