justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236601.88@dunegpschedd02.fnal.gov

Jobsub ID236601.88@dunegpschedd02.fnal.gov
Workflow ID9538
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-03 10:42:44
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce01
Last heartbeat2025-11-03 12:57:42
From worker nodeHostnamedune001-153795.0-lcg2474.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 10:58:35
Input filesvd-protodune:np02vd_raw_run040346_3320_df-s04-d3_dw_0_20251103T103313.hdf5
JobscriptExit code0
Real time1h (7047s)
CPU time1h (5014s = 71%)
Max RSS bytes4125102080 (3934 MiB)
Outputting started2025-11-03 12:56:04
Output filesvd-protodune-det-reco:np02vd_raw_run040346_3320_df-s04-d3_dw_0_20251103T103313_reco_stage1_20251103T125549_keepup.root
Finished2025-11-03 12:57:42
Saved logsjustin-logs:236601.88-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

with 8 
[12:55:01.364] D [  glue  ] frame sink sees EOS
[12:55:01.364] D [ pgraph ] <Pgrapher:> graph execution complete 
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.48 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.17 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.8 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.77 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.34 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.18 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.1 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.65 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.3 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.3 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.29 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.24 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:55:01.364] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[12:55:01.364] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[12:55:01.364] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[12:55:01.364] I [ timer  ] Timer: Total node execution : 69.6900012101978 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 41556 traces tagged "gauss"
FrameSaver: q=1.05627e+07 n=855069 tag=gauss
wclsFrameSaver: saving 52837 traces tagged "wiener"
FrameSaver: q=1.12531e+07 n=811007 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 396 XUs and 438 XVs -> 22 XUVs
C:0 T:1 454 XUs and 458 XVs -> 14 XUVs
C:0 T:2 542 XUs and 487 XVs -> 47 XUVs
C:0 T:3 273 XUs and 368 XVs -> 19 XUVs
C:0 T:4 854 XUs and 914 XVs -> 62 XUVs
C:0 T:5 1901 XUs and 2873 XVs -> 213 XUVs
C:0 T:6 1976 XUs and 1323 XVs -> 64 XUVs
C:0 T:7 5502 XUs and 6907 XVs -> 412 XUVs
C:0 T:8 517 XUs and 722 XVs -> 29 XUVs
C:0 T:9 827 XUs and 1630 XVs -> 87 XUVs
C:0 T:10 353 XUs and 661 XVs -> 45 XUVs
C:0 T:11 978 XUs and 1374 XVs -> 35 XUVs
C:0 T:12 791 XUs and 1600 XVs -> 77 XUVs
C:0 T:13 853 XUs and 825 XVs -> 71 XUVs
C:0 T:14 1070 XUs and 1825 XVs -> 101 XUVs
C:0 T:15 1104 XUs and 1530 XVs -> 89 XUVs
1387 XUVs total
1070 collection wire objects
1387 potential space points
Neighbour search...
20115 tests to find 8126 neighbours
Iterating with no regularization...
Begin: 1.40575e+11
0 1.40531e+11
Now with regularization...
Begin: 1.40344e+11
0 1.40343e+11
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
03-Nov-2025 12:55:49 UTC  Closed output file "np02vd_raw_run040346_3320_df-s04-d3_dw_0_20251103T103313_reco_stage1_20251103T125549_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        174.244        303.2        821.623       262.854       143.391        23     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      8.7044e-05    0.000131233   0.000270119   0.000115157   3.97709e-05      23     
produce:tpcrawdecoder:PDVDTPCReader               43.6675       89.0636       191.187       88.0489       26.2824        23     
produce:triggerrawdecoder:PDVDTriggerReader4     0.601419      0.757602       1.17973      0.697041       0.13748        23     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000399953   0.000840211   0.00358208    0.00055194    0.000859088      23     
produce:ophit:OpHitFinder                       7.5833e-05    0.000330764   0.00327302    0.000187844   0.000631095      23     
produce:opflash:OpFlashFinderVerticalDrift      5.9241e-05    0.000122073   0.000850512   8.4038e-05    0.000156622      23     
produce:wclsdatavd:WireCellToolkit                70.9173       94.7571       141.134       92.1174       20.9631        23     
produce:gaushit:GausHitFinder                     1.01411       1.60393       2.99255       1.46259      0.449984        23     
produce:nhitsfilter:NumberOfHitsFilter          0.000322587   0.000476713   0.00099809    0.000443225   0.00014699       23     
produce:reco3d:SpacePointSolver                   6.39517       14.8211       31.7759       12.8962       6.54196        23     
produce:hitpdune:DisambigFromSpacePoints         0.108221      0.300943      0.714687       0.1946       0.182428        23     
produce:pandora:StandardPandora                   16.2278       93.815        590.609       35.798        131.495        23     
produce:pandoraTrack:LArPandoraTrackCreation     0.541905       2.18542       8.87356       1.2827        2.11371        23     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0270777     0.049158      0.0831009     0.0491407     0.0146792       23     
[art]:TriggerResults:TriggerResultInserter      3.4184e-05    6.53353e-05   0.000163658   6.0134e-05    2.55289e-05      23     
end_path:out1:RootOutput                         4.999e-06    1.4246e-05    5.6847e-05    1.0399e-05    1.09287e-05      23     
end_path:out1:RootOutput(write)                   5.03253       5.74509       6.82152       5.62989      0.504112        23     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6328.8 MB
  Peak resident set size usage (VmHWM): 4157.73 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040346_3320_df-s04-d3_dw_0_20251103T103313_reco_stage1_20251103T125549_keepup.root
\tHists: np02vd_raw_run040346_3320_df-s04-d3_dw_0_20251103T103313_reco_stage1_20251103T125549_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040346_3320_df-s04-d3_dw_0_20251103T103313_reco_stage1_20251103T125549_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040346_3320_df-s04-d3_dw_0_20251103T103313_reco_stage1_20251103T125549_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762174551.0,
    "core.end_time": 1762174551.0,
    "core.events": [
      306383,
      306387,
      306391,
      306395,
      306399,
      306403,
      306407,
      306411,
      306415,
      306419,
      306423,
      306427,
      306431,
      306435,
      306439,
      306443,
      306447,
      306451,
      306455,
      306459,
      306463,
      306467,
      306471
    ],
    "core.event_count": 23,
    "core.first_event_number": 306383,
    "core.last_event_number": 306471,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3320_df-s04-d3_dw_0_20251103T103313.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040346_3320_df-s04-d3_dw_0_20251103T103313_reco_stage1_20251103T125549_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762174551.0,
    "core.end_time": 1762174551.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3320_df-s04-d3_dw_0_20251103T103313.hdf5"
    }
  ]
}
justIN time: 2025-11-04 01:31:04 UTC       justIN version: 01.05.01