justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 48298.0@dunegpschedd01.fnal.gov

Jobsub ID48298.0@dunegpschedd01.fnal.gov
Workflow ID2867
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-18 09:26:16
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce01
Last heartbeat2025-09-18 12:02:21
From worker nodeHostnamedune001-9163787.0-lcg2560.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-18 09:27:30
Input filesvd-protodune:np02vd_raw_run039433_0366_df-s05-d2_dw_0_20250918T062307.hdf5
JobscriptExit code0
Real time2h (9177s)
CPU time2h (8667s = 94%)
Max RSS bytes4112519168 (3922 MiB)
Outputting started2025-09-18 12:00:29
Output filesvd-protodune-det-reco:np02vd_raw_run039433_0366_df-s05-d2_dw_0_20250918T062307_reco_stage1_20250918T120011_keepup.root
Finished2025-09-18 12:02:21
Saved logsjustin-logs:48298.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oc:anode0sigproc0> EOS at call=95 anode=0 
[10:50:14.066] D [  glue  ] <FrameFanin:nfsp> EOS at call=95 with 8 
[10:50:14.066] D [  glue  ] frame sink sees EOS
[10:50:14.066] D [ pgraph ] <Pgrapher:> graph execution complete 
[10:50:14.066] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 51.72 sec
[10:50:14.066] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.2 sec
[10:50:14.066] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.11 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.01 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.31 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.3 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.29 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.28 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[10:50:14.067] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[10:50:14.067] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[10:50:14.067] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[10:50:14.067] I [ timer  ] Timer: Total node execution : 91.26000092178583 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 35148 traces tagged "gauss"
FrameSaver: q=1.1421e+07 n=981087 tag=gauss
wclsFrameSaver: saving 45452 traces tagged "wiener"
FrameSaver: q=1.23198e+07 n=940780 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 270 XUs and 665 XVs -> 12 XUVs
C:0 T:1 115 XUs and 164 XVs -> 5 XUVs
C:0 T:2 573 XUs and 526 XVs -> 9 XUVs
C:0 T:3 178 XUs and 251 XVs -> 10 XUVs
C:0 T:4 3344 XUs and 3328 XVs -> 153 XUVs
C:0 T:5 183310 XUs and 239756 XVs -> 19623 XUVs
C:0 T:6 1336 XUs and 1127 XVs -> 65 XUVs
C:0 T:7 9160 XUs and 10559 XVs -> 480 XUVs
20357 XUVs total
4645 collection wire objects
20357 potential space points
Neighbour search...
823177 tests to find 163236 neighbours
Iterating with no regularization...
Begin: 8.84867e+10
0 8.29128e+10
1 8.21571e+10
2 8.21476e+10
Now with regularization...
Begin: 8.19478e+10
0 8.19462e+10
18-Sep-2025 12:00:11 UTC  Closed output file "np02vd_raw_run039433_0366_df-s05-d2_dw_0_20250918T062307_reco_stage1_20250918T120011_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        55.4734       189.056       4300.54       86.6543       601.573        48     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      8.3601e-05    0.000143248   0.000380927   0.000114217   6.06126e-05      48     
produce:tpcrawdecoder:PDVDTPCReader               7.4816        8.1772        9.95187       8.08582      0.502893        48     
produce:triggerrawdecoder:PDVDTriggerReader4     0.392042      0.398202      0.425609      0.397078     0.00534702       48     
produce:pdvddaphne:DAPHNEReaderPDVD               3.3328        3.97375       5.32634       3.94484      0.332126        48     
produce:ophit:OpHitFinder                        0.0438283     0.0604638     0.0887922     0.0603878    0.00757095       48     
produce:opflash:OpFlashFinderVerticalDrift      0.00606823     0.0158448     0.0444888     0.0152463    0.00596508       48     
produce:wclsdatavd:WireCellToolkit                29.4698       46.2294       92.5619       39.559        14.7693        48     
produce:gaushit:GausHitFinder                    0.371741      0.848882       1.88438      0.805263      0.257174        48     
produce:nhitsfilter:NumberOfHitsFilter          0.000104412   0.000239929   0.00133223    0.000208074   0.000179029      48     
produce:reco3d:SpacePointSolver                   1.70912       7.49328       34.7537       7.14604       4.86824        48     
produce:hitpdune:DisambigFromSpacePoints         0.0232932     0.131366      0.807938      0.0914964     0.123299        48     
produce:pandora:StandardPandora                   3.06214       117.945       4148.87       20.4219       589.094        48     
produce:pandoraTrack:LArPandoraTrackCreation     0.0806948     0.465159       2.43939      0.369113      0.358926        48     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0105157     0.0246482     0.0619116     0.0236611    0.00909152       48     
[art]:TriggerResults:TriggerResultInserter       2.598e-05    4.08354e-05   0.000157112   3.53805e-05   2.01599e-05      48     
end_path:out1:RootOutput                         4.709e-06    1.40947e-05   0.000166593    9.82e-06     2.26232e-05      48     
end_path:out1:RootOutput(write)                   2.50584       3.26861       6.85709       3.18456      0.621044        48     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6181.84 MB
  Peak resident set size usage (VmHWM): 4112.52 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039433_0366_df-s05-d2_dw_0_20250918T062307_reco_stage1_20250918T120011_keepup.root
\tHists: np02vd_raw_run039433_0366_df-s05-d2_dw_0_20250918T062307_reco_stage1_20250918T120011_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039433_0366_df-s05-d2_dw_0_20250918T062307_reco_stage1_20250918T120011_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039433_0366_df-s05-d2_dw_0_20250918T062307_reco_stage1_20250918T120011_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758196813.0,
    "core.end_time": 1758196813.0,
    "core.events": [
      104990,
      104996,
      105002,
      105008,
      105014,
      105020,
      105026,
      105032,
      105038,
      105044,
      105050,
      105056,
      105062,
      105068,
      105074,
      105080,
      105086,
      105092,
      105098,
      105104,
      105110,
      105116,
      105122,
      105128,
      105134,
      105140,
      105146,
      105152,
      105158,
      105164,
      105170,
      105176,
      105182,
      105188,
      105194,
      105200,
      105206,
      105212,
      105218,
      105224,
      105230,
      105236,
      105242,
      105248,
      105254,
      105260,
      105266,
      105272
    ],
    "core.event_count": 48,
    "core.first_event_number": 104990,
    "core.last_event_number": 105272,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39433
    ],
    "core.runs_subruns": [
      3943300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039433_0366_df-s05-d2_dw_0_20250918T062307.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039433_0366_df-s05-d2_dw_0_20250918T062307_reco_stage1_20250918T120011_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758196813.0,
    "core.end_time": 1758196813.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39433
    ],
    "core.runs_subruns": [
      3943300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039433_0366_df-s05-d2_dw_0_20250918T062307.hdf5"
    }
  ]
}
justIN time: 2025-09-18 23:46:32 UTC       justIN version: 01.05.00