justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 49050.178@dunegpschedd01.fnal.gov

Jobsub ID49050.178@dunegpschedd01.fnal.gov
Workflow ID2920
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-18 20:08:50
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce05
Last heartbeat2025-09-18 23:17:07
From worker nodeHostnamedune001-7806722.0-lcg2709.gridpp.rl.ac.uk
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-18 22:24:24
Input filesvd-protodune:np02vd_raw_run039499_0083_df-s05-d4_dw_0_20250918T202956.hdf5
JobscriptExit code0
Real time51m (3089s)
CPU time44m (2652s = 85%)
Max RSS bytes2647326720 (2524 MiB)
Outputting started2025-09-18 23:15:53
Output filesvd-protodune-det-reco:np02vd_raw_run039499_0083_df-s05-d4_dw_0_20250918T202956_reco_stage1_20250918T231544_keepup.root
Finished2025-09-18 23:17:07
Saved logsjustin-logs:49050.178-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[23:15:26.050] D [  aux   ] <Resampler:resmp0> EOS at call=81 
[23:15:26.050] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=81 anode=0 
[23:15:26.050] D [  glue  ] <FrameFanin:nfsp> EOS at call=81 with 8 
[23:15:26.050] D [  glue  ] frame sink sees EOS
[23:15:26.050] D [ pgraph ] <Pgrapher:> graph execution complete 
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.44 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.86 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.15 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.9 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.22 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.2 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.18 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.17 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[23:15:26.050] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[23:15:26.050] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[23:15:26.050] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[23:15:26.050] I [ timer  ] Timer: Total node execution : 23.140000391751528 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 20220 traces tagged "gauss"
FrameSaver: q=3.20771e+06 n=467917 tag=gauss
wclsFrameSaver: saving 26722 traces tagged "wiener"
FrameSaver: q=3.4291e+06 n=446582 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 823 XUs and 767 XVs -> 38 XUVs
C:0 T:1 328 XUs and 273 XVs -> 18 XUVs
C:0 T:2 8685 XUs and 9257 XVs -> 600 XUVs
C:0 T:3 528 XUs and 641 XVs -> 35 XUVs
C:0 T:4 2307 XUs and 2453 XVs -> 127 XUVs
C:0 T:5 1413 XUs and 1436 XVs -> 75 XUVs
C:0 T:6 14 XUs and 14 XVs -> 0 XUVs
C:0 T:7 445 XUs and 359 XVs -> 26 XUVs
919 XUVs total
675 collection wire objects
919 potential space points
Neighbour search...
14881 tests to find 4260 neighbours
Iterating with no regularization...
Begin: 2.61513e+09
0 2.59965e+09
1 2.59933e+09
Now with regularization...
Begin: 2.52516e+09
0 2.52515e+09
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
18-Sep-2025 23:15:44 UTC  Closed output file "np02vd_raw_run039499_0083_df-s05-d4_dw_0_20250918T202956_reco_stage1_20250918T231544_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        37.1143       73.3778       581.119       51.7849       85.1509        41     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.6235e-05    0.000104121   0.000210388   9.0797e-05    4.0164e-05       41     
produce:tpcrawdecoder:PDVDTPCReader               6.95583       7.43979       9.14089       7.31385      0.480247        41     
produce:triggerrawdecoder:PDVDTriggerReader4      0.38816      0.396215      0.453176       0.3907       0.0137789       41     
produce:pdvddaphne:DAPHNEReaderPDVD               1.96112       2.53958       2.98571       2.54776      0.218276        41     
produce:ophit:OpHitFinder                        0.0262444     0.0359117     0.0449072     0.0359433    0.00472739       41     
produce:opflash:OpFlashFinderVerticalDrift      0.00299351    0.00731781     0.0124985    0.00729259    0.00230799       41     
produce:wclsdatavd:WireCellToolkit                19.6965       25.8008       39.494        22.9368       5.84164        41     
produce:gaushit:GausHitFinder                    0.186055      0.464599      0.970902      0.443554      0.161058        41     
produce:nhitsfilter:NumberOfHitsFilter          7.6055e-05    0.000137079   0.000308215   0.000118649   5.18245e-05      41     
produce:reco3d:SpacePointSolver                   1.01938       4.67843       18.5638       4.03671       2.86378        41     
produce:hitpdune:DisambigFromSpacePoints         0.0146336     0.0583094     0.279492      0.0441384     0.047389        41     
produce:pandora:StandardPandora                   2.75147       29.7486       529.736       10.0999       82.2039        41     
produce:pandoraTrack:LArPandoraTrackCreation     0.0624497     0.239612      0.807368      0.212505      0.154503        41     
produce:pandoraGnocalo:GnocchiCalorimetry       0.00391628     0.0116284     0.0255817     0.0111601     0.0045609       41     
[art]:TriggerResults:TriggerResultInserter      1.5063e-05    2.88731e-05   9.1468e-05    2.4638e-05    1.41691e-05      41     
end_path:out1:RootOutput                         3.576e-06    7.29476e-06   2.0531e-05     7.06e-06     2.90218e-06      41     
end_path:out1:RootOutput(write)                   1.62492       1.93716       2.38427       1.91306      0.164484        41     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4982.46 MB
  Peak resident set size usage (VmHWM): 2647.33 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039499_0083_df-s05-d4_dw_0_20250918T202956_reco_stage1_20250918T231544_keepup.root
\tHists: np02vd_raw_run039499_0083_df-s05-d4_dw_0_20250918T202956_reco_stage1_20250918T231544_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039499_0083_df-s05-d4_dw_0_20250918T202956_reco_stage1_20250918T231544_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039499_0083_df-s05-d4_dw_0_20250918T202956_reco_stage1_20250918T231544_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758237345.0,
    "core.end_time": 1758237345.0,
    "core.events": [
      20266,
      20272,
      20278,
      20284,
      20290,
      20296,
      20302,
      20308,
      20314,
      20320,
      20326,
      20332,
      20338,
      20344,
      20350,
      20356,
      20362,
      20368,
      20374,
      20380,
      20386,
      20392,
      20398,
      20404,
      20410,
      20416,
      20422,
      20428,
      20434,
      20440,
      20446,
      20452,
      20458,
      20464,
      20470,
      20476,
      20482,
      20488,
      20494,
      20500,
      20506
    ],
    "core.event_count": 41,
    "core.first_event_number": 20266,
    "core.last_event_number": 20506,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39499
    ],
    "core.runs_subruns": [
      3949900001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039499_0083_df-s05-d4_dw_0_20250918T202956.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039499_0083_df-s05-d4_dw_0_20250918T202956_reco_stage1_20250918T231544_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758237345.0,
    "core.end_time": 1758237345.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39499
    ],
    "core.runs_subruns": [
      3949900001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039499_0083_df-s05-d4_dw_0_20250918T202956.hdf5"
    }
  ]
}
justIN time: 2025-09-19 05:51:37 UTC       justIN version: 01.05.00