justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 41739.10@dunegpschedd02.fnal.gov

Jobsub ID41739.10@dunegpschedd02.fnal.gov
Workflow ID2867
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-18 07:48:12
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-09-18 08:17:35
From worker nodeHostnamewn-da-01.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-18 07:48:53
Input filesvd-protodune:np02vd_raw_run039433_0396_df-s05-d0_dw_0_20250918T073812.hdf5
JobscriptExit code0
Real time26m (1597s)
CPU time15m (939s = 58%)
Max RSS bytes2385305600 (2274 MiB)
Outputting started2025-09-18 08:15:31
Output filesvd-protodune-det-reco:np02vd_raw_run039433_0396_df-s05-d0_dw_0_20250918T073812_reco_stage1_20250918T081516_keepup.root
Finished2025-09-18 08:17:35
Saved logsjustin-logs:41739.10-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] <OmnibusSigProc:anode2sigproc2> EOS at call=21 anode=2 
[10:15:00.081] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=10 
[10:15:00.081] D [  aux   ] <Resampler:resmp1> EOS at call=21 
[10:15:00.081] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=21 anode=1 
[10:15:00.081] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=10 
[10:15:00.081] D [  aux   ] <Resampler:resmp0> EOS at call=21 
[10:15:00.081] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=21 anode=0 
[10:15:00.081] D [  glue  ] <FrameFanin:nfsp> EOS at call=21 with 8 
[10:15:00.081] D [  glue  ] frame sink sees EOS
[10:15:00.082] D [ pgraph ] <Pgrapher:> graph execution complete 
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 15.67 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.94 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.72 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.53 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.19 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.19 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.18 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.18 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[10:15:00.082] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[10:15:00.082] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[10:15:00.082] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[10:15:00.082] I [ timer  ] Timer: Total node execution : 60.60999966599047 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 16187 traces tagged "gauss"
FrameSaver: q=3.43215e+06 n=407186 tag=gauss
wclsFrameSaver: saving 20398 traces tagged "wiener"
FrameSaver: q=3.71666e+06 n=395391 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 417 XUs and 513 XVs -> 18 XUVs
C:0 T:1 290 XUs and 211 XVs -> 13 XUVs
C:0 T:2 432 XUs and 529 XVs -> 20 XUVs
C:0 T:3 71 XUs and 99 XVs -> 19 XUVs
C:0 T:4 1042 XUs and 1607 XVs -> 43 XUVs
C:0 T:5 14409 XUs and 15588 XVs -> 1225 XUVs
C:0 T:6 478 XUs and 339 XVs -> 25 XUVs
C:0 T:7 3419 XUs and 2777 XVs -> 239 XUVs
1602 XUVs total
841 collection wire objects
1602 potential space points
Neighbour search...
46358 tests to find 12226 neighbours
Iterating with no regularization...
Begin: 6.55289e+09
0 6.46425e+09
1 6.45649e+09
2 6.45586e+09
Now with regularization...
Begin: 6.29995e+09
0 6.29979e+09
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
18-Sep-2025 10:15:16 CEST  Closed output file "np02vd_raw_run039433_0396_df-s05-d0_dw_0_20250918T073812_reco_stage1_20250918T081516_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        98.5948       139.795       319.963       116.063       60.5347        11     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.6948e-05    7.55332e-05   0.000175754   6.1767e-05    3.3207e-05       11     
produce:tpcrawdecoder:PDVDTPCReader               42.477        50.744        69.0436       47.4265       8.09814        11     
produce:triggerrawdecoder:PDVDTriggerReader4     0.314761      0.454732      0.845746      0.399186      0.151274        11     
produce:pdvddaphne:DAPHNEReaderPDVD               9.68376       11.7534       14.3551       11.5547       1.42548        11     
produce:ophit:OpHitFinder                        0.0255459      0.04102      0.0601594     0.0423255    0.00873139       11     
produce:opflash:OpFlashFinderVerticalDrift      0.00205918    0.00887692     0.0160035    0.00850512    0.00349422       11     
produce:wclsdatavd:WireCellToolkit                26.2591       45.7025       64.9082       41.2159       11.8085        11     
produce:gaushit:GausHitFinder                    0.229507      0.543587       1.01788      0.495512      0.203752        11     
produce:nhitsfilter:NumberOfHitsFilter          8.2768e-05    0.000135507   0.000290763   0.000113366   6.15703e-05      11     
produce:reco3d:SpacePointSolver                   1.40879       3.78758       8.05104       3.58584       1.74978        11     
produce:hitpdune:DisambigFromSpacePoints         0.0214811     0.0559472     0.167053      0.0480749     0.0374946       11     
produce:pandora:StandardPandora                   2.92915       24.403        175.75        8.60425       48.0523        11     
produce:pandoraTrack:LArPandoraTrackCreation     0.0788199     0.218572       0.60452      0.193059      0.132903        11     
produce:pandoraGnocalo:GnocchiCalorimetry       0.00566262     0.0116731     0.0208836     0.0115719    0.00408711       11     
[art]:TriggerResults:TriggerResultInserter      1.4958e-05    1.9723e-05    4.8162e-05    1.7003e-05    9.14847e-06      11     
end_path:out1:RootOutput                         3.857e-06     5.648e-06    1.7504e-05     4.468e-06    3.76406e-06      11     
end_path:out1:RootOutput(write)                   1.81642       2.06166       2.68997       2.00076      0.236871        11     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4268.03 MB
  Peak resident set size usage (VmHWM): 2385.31 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039433_0396_df-s05-d0_dw_0_20250918T073812_reco_stage1_20250918T081516_keepup.root
\tHists: np02vd_raw_run039433_0396_df-s05-d0_dw_0_20250918T073812_reco_stage1_20250918T081516_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039433_0396_df-s05-d0_dw_0_20250918T073812_reco_stage1_20250918T081516_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039433_0396_df-s05-d0_dw_0_20250918T073812_reco_stage1_20250918T081516_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758183317.0,
    "core.end_time": 1758183317.0,
    "core.events": [
      113670,
      113676,
      113682,
      113688,
      113694,
      113700,
      113706,
      113712,
      113718,
      113724,
      113730
    ],
    "core.event_count": 11,
    "core.first_event_number": 113670,
    "core.last_event_number": 113730,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39433
    ],
    "core.runs_subruns": [
      3943300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039433_0396_df-s05-d0_dw_0_20250918T073812.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039433_0396_df-s05-d0_dw_0_20250918T073812_reco_stage1_20250918T081516_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758183317.0,
    "core.end_time": 1758183317.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39433
    ],
    "core.runs_subruns": [
      3943300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039433_0396_df-s05-d0_dw_0_20250918T073812.hdf5"
    }
  ]
}
justIN time: 2025-09-18 19:24:52 UTC       justIN version: 01.05.00