justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 243188.7@dunegpschedd01.fnal.gov

Jobsub ID243188.7@dunegpschedd01.fnal.gov
Workflow ID9584
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-04 21:38:33
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce01
Last heartbeat2025-11-05 00:59:54
From worker nodeHostnamedune001-162932.0-lcg2699.gridpp.rl.ac.uk
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-04 21:39:25
Input filesvd-protodune:np02vd_raw_run040348_1456_df-s04-d2_dw_0_20251104T152426.hdf5
JobscriptExit code0
Real time3h (11930s)
CPU time3h (11278s = 94%)
Max RSS bytes6421696512 (6124 MiB)
Outputting started2025-11-05 00:58:16
Output filesvd-protodune-det-reco:np02vd_raw_run040348_1456_df-s04-d2_dw_0_20251104T152426_reco_stage1_20251105T005803_keepup.root
Finished2025-11-05 00:59:54
Saved logsjustin-logs:243188.7-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] <OmnibusSigProc:anode0sigproc0> EOS at call=45 anode=0 
[00:57:05.870] D [  glue  ] <FrameFanin:nfsp> EOS at call=45 with 8 
[00:57:05.870] D [  glue  ] frame sink sees EOS
[00:57:05.870] D [ pgraph ] <Pgrapher:> graph execution complete 
[00:57:05.870] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.06 sec
[00:57:05.870] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.37 sec
[00:57:05.870] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.93 sec
[00:57:05.870] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.71 sec
[00:57:05.870] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.55 sec
[00:57:05.870] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.44 sec
[00:57:05.870] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.41 sec
[00:57:05.870] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.93 sec
[00:57:05.870] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.25 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.24 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.22 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.2 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.03 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:57:05.871] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[00:57:05.871] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[00:57:05.871] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[00:57:05.871] I [ timer  ] Timer: Total node execution : 57.37000009045005 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 46659 traces tagged "gauss"
FrameSaver: q=1.52405e+07 n=1080822 tag=gauss
wclsFrameSaver: saving 58997 traces tagged "wiener"
FrameSaver: q=1.62731e+07 n=1039792 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 255 XUs and 360 XVs -> 7 XUVs
C:0 T:1 267 XUs and 221 XVs -> 6 XUVs
C:0 T:2 1214 XUs and 1055 XVs -> 88 XUVs
C:0 T:3 2569 XUs and 1668 XVs -> 248 XUVs
C:0 T:4 2180 XUs and 1442 XVs -> 160 XUVs
C:0 T:5 1121 XUs and 776 XVs -> 87 XUVs
C:0 T:6 1395 XUs and 895 XVs -> 124 XUVs
C:0 T:7 5303 XUs and 5401 XVs -> 440 XUVs
C:0 T:8 314 XUs and 349 XVs -> 44 XUVs
C:0 T:9 665 XUs and 1051 XVs -> 58 XUVs
C:0 T:10 816 XUs and 710 XVs -> 31 XUVs
C:0 T:11 8873 XUs and 14791 XVs -> 4181 XUVs
C:0 T:12 2933 XUs and 6982 XVs -> 741 XUVs
C:0 T:13 13052 XUs and 22397 XVs -> 2677 XUVs
C:0 T:14 12968 XUs and 22936 XVs -> 6371 XUVs
C:0 T:15 1261 XUs and 849 XVs -> 57 XUVs
15320 XUVs total
1569 collection wire objects
15320 potential space points
Neighbour search...
3274818 tests to find 1260384 neighbours
Iterating with no regularization...
Begin: 1.70604e+10
0 1.68869e+10
1 1.68781e+10
Now with regularization...
Begin: 1.65348e+10
0 1.65343e+10
05-Nov-2025 00:58:03 UTC  Closed output file "np02vd_raw_run040348_1456_df-s04-d2_dw_0_20251104T152426_reco_stage1_20251105T005803_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        98.2278       515.801       8515.47       147.223        1706          23     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.4041e-05    0.000117518   0.000295505   0.000111739   4.34983e-05      23     
produce:tpcrawdecoder:PDVDTPCReader               14.7822       20.5948       58.5046       18.4637       8.75794        23     
produce:triggerrawdecoder:PDVDTriggerReader4     0.691475       0.70843      0.770044      0.700513      0.0201728       23     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000310057   0.000472177   0.00117932    0.000414505   0.000179145      23     
produce:ophit:OpHitFinder                       6.7832e-05    0.000416203   0.00576055    0.000157137   0.00114404       23     
produce:opflash:OpFlashFinderVerticalDrift      4.5779e-05    0.00015135    0.000914117   8.1052e-05    0.000200912      23     
produce:wclsdatavd:WireCellToolkit                44.0492       81.4684       168.933       67.6861       33.0484        23     
produce:gaushit:GausHitFinder                    0.757069       1.63108       5.40427       1.29641       1.0878         23     
produce:nhitsfilter:NumberOfHitsFilter          0.000174723   0.000362104   0.000865364   0.000309296   0.000190864      23     
produce:reco3d:SpacePointSolver                   7.3251        16.8103       116.156       11.9601       21.9819        22     
produce:hitpdune:DisambigFromSpacePoints         0.114662      0.349819       3.12486      0.208509       0.61208        22     
produce:pandora:StandardPandora                   13.9454       404.912       8161.75       31.9458       1692.78        22     
produce:pandoraTrack:LArPandoraTrackCreation     0.412318       2.62876       34.8323       0.98551       7.05061        22     
produce:pandoraGnocalo:GnocchiCalorimetry         0.01671      0.041528      0.154388      0.0361055     0.027187        22     
[art]:TriggerResults:TriggerResultInserter      3.5614e-05    6.29304e-05   9.8218e-05    5.9239e-05    1.66718e-05      23     
end_path:out1:RootOutput                         6.22e-06     1.64998e-05   4.8944e-05    1.4131e-05    8.29169e-06      23     
end_path:out1:RootOutput(write)                   3.63417       5.02601       7.61626       4.89514      0.823188        23     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8694.58 MB
  Peak resident set size usage (VmHWM): 6421.7 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040348_1456_df-s04-d2_dw_0_20251104T152426_reco_stage1_20251105T005803_keepup.root
\tHists: np02vd_raw_run040348_1456_df-s04-d2_dw_0_20251104T152426_reco_stage1_20251105T005803_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040348_1456_df-s04-d2_dw_0_20251104T152426_reco_stage1_20251105T005803_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040348_1456_df-s04-d2_dw_0_20251104T152426_reco_stage1_20251105T005803_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762304284.0,
    "core.end_time": 1762304284.0,
    "core.events": [
      134522,
      134526,
      134530,
      134534,
      134538,
      134542,
      134546,
      134550,
      134554,
      134558,
      134562,
      134566,
      134570,
      134574,
      134578,
      134582,
      134586,
      134590,
      134594,
      134598,
      134602,
      134606,
      134610
    ],
    "core.event_count": 23,
    "core.first_event_number": 134522,
    "core.last_event_number": 134610,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40348
    ],
    "core.runs_subruns": [
      4034800001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040348_1456_df-s04-d2_dw_0_20251104T152426.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040348_1456_df-s04-d2_dw_0_20251104T152426_reco_stage1_20251105T005803_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762304284.0,
    "core.end_time": 1762304284.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40348
    ],
    "core.runs_subruns": [
      4034800001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040348_1456_df-s04-d2_dw_0_20251104T152426.hdf5"
    }
  ]
}
justIN time: 2025-11-05 07:30:34 UTC       justIN version: 01.05.01