justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 41697.1@dunegpschedd02.fnal.gov

Jobsub ID41697.1@dunegpschedd02.fnal.gov
Workflow ID2867
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-18 06:46:09
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce04
Last heartbeat2025-09-18 07:54:27
From worker nodeHostnamedune001-7744427.0-lcg2693.gridpp.rl.ac.uk
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-18 06:47:29
Input filesvd-protodune:np02vd_raw_run039433_0354_df-s05-d0_dw_0_20250918T055409.hdf5
JobscriptExit code0
Real time1h (3923s)
CPU time57m (3468s = 88%)
Max RSS bytes2736836608 (2610 MiB)
Outputting started2025-09-18 07:52:53
Output filesvd-protodune-det-reco:np02vd_raw_run039433_0354_df-s05-d0_dw_0_20250918T055409_reco_stage1_20250918T075240_keepup.root
Finished2025-09-18 07:54:27
Saved logsjustin-logs:41697.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

me sink sees EOS
[07:52:20.757] D [ pgraph ] <Pgrapher:> graph execution complete 
[07:52:20.757] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.69 sec
[07:52:20.757] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.16 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.78 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.68 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.44 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.42 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.42 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.3 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:52:20.758] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[07:52:20.758] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[07:52:20.758] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[07:52:20.758] I [ timer  ] Timer: Total node execution : 25.95999992452562 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 17330 traces tagged "gauss"
FrameSaver: q=3.02147e+06 n=402359 tag=gauss
wclsFrameSaver: saving 21623 traces tagged "wiener"
FrameSaver: q=3.20368e+06 n=387005 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 7311 XUs and 13144 XVs -> 541 XUVs
C:0 T:1 679 XUs and 610 XVs -> 24 XUVs
C:0 T:2 1805 XUs and 2985 XVs -> 99 XUVs
C:0 T:3 101 XUs and 98 XVs -> 2 XUVs
C:0 T:4 606 XUs and 484 XVs -> 29 XUVs
C:0 T:5 435 XUs and 746 XVs -> 26 XUVs
C:0 T:6 420 XUs and 696 XVs -> 21 XUVs
C:0 T:7 632 XUs and 775 XVs -> 36 XUVs
778 XUVs total
582 collection wire objects
778 potential space points
Neighbour search...
18068 tests to find 4462 neighbours
Iterating with no regularization...
Begin: 1.5014e+09
0 1.46506e+09
1 1.46441e+09
Now with regularization...
Begin: 1.4516e+09
0 1.45157e+09
PcaShowerParticleBuildingAlgorithm::OpeningAngle - principal eigenvalue less than or equal to 0.
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
18-Sep-2025 07:52:40 UTC  Closed output file "np02vd_raw_run039433_0354_df-s05-d0_dw_0_20250918T055409_reco_stage1_20250918T075240_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        44.3152       79.9122       356.463       62.1506       56.305         48     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.3261e-05     0.000156     0.00034574    0.000130226   6.31857e-05      48     
produce:tpcrawdecoder:PDVDTPCReader               6.86586       7.87263       21.6083       7.19013       2.23791        48     
produce:triggerrawdecoder:PDVDTriggerReader4     0.391694      0.397804      0.444303      0.395677     0.00830368       48     
produce:pdvddaphne:DAPHNEReaderPDVD               3.07377       3.55433       3.98339       3.58153      0.197417        48     
produce:ophit:OpHitFinder                        0.036099      0.0490606     0.0645899     0.0491047    0.00556368       48     
produce:opflash:OpFlashFinderVerticalDrift       0.0042083     0.0123729     0.0267808     0.0125493    0.00431464       48     
produce:wclsdatavd:WireCellToolkit                19.8094       32.6802       66.7332        28.95        9.00367        48     
produce:gaushit:GausHitFinder                    0.274721       0.61937       1.51105      0.588431      0.228824        48     
produce:nhitsfilter:NumberOfHitsFilter          9.3671e-05    0.000231663   0.000638681   0.000198869   0.000111222      48     
produce:reco3d:SpacePointSolver                   1.60524       4.9665        19.0347       4.19146       2.93453        48     
produce:hitpdune:DisambigFromSpacePoints         0.0183359     0.0777507     0.406561      0.0548823     0.0686874       48     
produce:pandora:StandardPandora                   3.83758       26.8903       286.688       10.5358       49.6483        48     
produce:pandoraTrack:LArPandoraTrackCreation     0.0684638     0.315866       1.84584      0.228691      0.297555        48     
produce:pandoraGnocalo:GnocchiCalorimetry       0.00564614     0.0151773     0.0313698     0.0148596    0.00593721       48     
[art]:TriggerResults:TriggerResultInserter      1.8387e-05    4.97964e-05   0.000182264   4.15325e-05   2.9971e-05       48     
end_path:out1:RootOutput                         4.507e-06    1.47576e-05    5.944e-05    1.4752e-05    8.40322e-06      48     
end_path:out1:RootOutput(write)                   1.75043       2.42226       3.26131       2.43506      0.393186        48     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4905.1 MB
  Peak resident set size usage (VmHWM): 2736.84 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039433_0354_df-s05-d0_dw_0_20250918T055409_reco_stage1_20250918T075240_keepup.root
\tHists: np02vd_raw_run039433_0354_df-s05-d0_dw_0_20250918T055409_reco_stage1_20250918T075240_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039433_0354_df-s05-d0_dw_0_20250918T055409_reco_stage1_20250918T075240_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039433_0354_df-s05-d0_dw_0_20250918T055409_reco_stage1_20250918T075240_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758181961.0,
    "core.end_time": 1758181961.0,
    "core.events": [
      101616,
      101622,
      101628,
      101634,
      101640,
      101646,
      101652,
      101658,
      101664,
      101670,
      101676,
      101682,
      101688,
      101694,
      101700,
      101706,
      101712,
      101718,
      101724,
      101730,
      101736,
      101742,
      101748,
      101754,
      101760,
      101766,
      101772,
      101778,
      101784,
      101790,
      101796,
      101802,
      101808,
      101814,
      101820,
      101826,
      101832,
      101838,
      101844,
      101850,
      101856,
      101862,
      101868,
      101874,
      101880,
      101886,
      101892,
      101898
    ],
    "core.event_count": 48,
    "core.first_event_number": 101616,
    "core.last_event_number": 101898,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39433
    ],
    "core.runs_subruns": [
      3943300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039433_0354_df-s05-d0_dw_0_20250918T055409.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039433_0354_df-s05-d0_dw_0_20250918T055409_reco_stage1_20250918T075240_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758181961.0,
    "core.end_time": 1758181961.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39433
    ],
    "core.runs_subruns": [
      3943300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039433_0354_df-s05-d0_dw_0_20250918T055409.hdf5"
    }
  ]
}
justIN time: 2025-09-18 20:59:26 UTC       justIN version: 01.05.00