justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242909.21@dunegpschedd01.fnal.gov

Jobsub ID242909.21@dunegpschedd01.fnal.gov
Workflow ID9584
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-04 12:48:06
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce03
Last heartbeat2025-11-04 13:46:47
From worker nodeHostnamedune001-89055.0-lcg2727.gridpp.rl.ac.uk
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-04 12:48:24
Input filesvd-protodune:np02vd_raw_run040348_1232_df-s04-d1_dw_0_20251104T121647.hdf5
JobscriptExit code0
Real time56m (3419s)
CPU time51m (3062s = 89%)
Max RSS bytes4583157760 (4370 MiB)
Outputting started2025-11-04 13:45:24
Output filesvd-protodune-det-reco:np02vd_raw_run040348_1232_df-s04-d1_dw_0_20251104T121647_reco_stage1_20251104T134513_keepup.root
Finished2025-11-04 13:46:47
Saved logsjustin-logs:242909.21-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.82 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.36 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.14 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.91 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.9 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.73 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.39 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.06 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.25 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.22 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.2 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.16 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:44:10.390] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[13:44:10.390] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[13:44:10.390] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[13:44:10.390] I [ timer  ] Timer: Total node execution : 54.20999946258962 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 52811 traces tagged "gauss"
FrameSaver: q=1.36531e+07 n=1267324 tag=gauss
wclsFrameSaver: saving 66801 traces tagged "wiener"
FrameSaver: q=1.4724e+07 n=1216784 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 709 XUs and 770 XVs -> 28 XUVs
C:0 T:1 15787 XUs and 18751 XVs -> 1273 XUVs
C:0 T:2 471 XUs and 511 XVs -> 12 XUVs
C:0 T:3 403 XUs and 435 XVs -> 25 XUVs
C:0 T:4 161 XUs and 189 XVs -> 4 XUVs
C:0 T:5 1209 XUs and 1600 XVs -> 22 XUVs
C:0 T:6 166 XUs and 276 XVs -> 6 XUVs
C:0 T:7 226 XUs and 317 XVs -> 4 XUVs
C:0 T:8 428 XUs and 432 XVs -> 17 XUVs
C:0 T:9 379 XUs and 620 XVs -> 33 XUVs
C:0 T:10 3123 XUs and 3166 XVs -> 280 XUVs
C:0 T:11 14869 XUs and 18899 XVs -> 3982 XUVs
C:0 T:12 6093 XUs and 10953 XVs -> 2019 XUVs
C:0 T:13 18036 XUs and 21461 XVs -> 3901 XUVs
C:0 T:14 15408 XUs and 23669 XVs -> 5922 XUVs
C:0 T:15 20665 XUs and 51754 XVs -> 6246 XUVs
23774 XUVs total
2748 collection wire objects
23774 potential space points
Neighbour search...
3214484 tests to find 1386804 neighbours
Iterating with no regularization...
Begin: 1.11377e+10
0 1.0882e+10
1 1.08509e+10
2 1.08498e+10
Now with regularization...
Begin: 1.07962e+10
0 1.07954e+10
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
04-Nov-2025 13:45:13 UTC  Closed output file "np02vd_raw_run040348_1232_df-s04-d1_dw_0_20251104T121647_reco_stage1_20251104T134513_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        80.1377       139.931       499.448       118.318       81.9631        24     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.4668e-05    0.000201995    0.0021815    0.000105009   0.000414996      24     
produce:tpcrawdecoder:PDVDTPCReader               14.1637       15.0988       22.4587       14.5542       1.75151        24     
produce:triggerrawdecoder:PDVDTriggerReader4     0.685999      0.703092      0.747331       0.70055      0.0134201       24     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000313183   0.000449023   0.000893711   0.000437597   0.000108118      24     
produce:ophit:OpHitFinder                       7.5184e-05    0.000179154   0.000772888   0.000156682   0.000127465      24     
produce:opflash:OpFlashFinderVerticalDrift      5.7147e-05    0.000116067   0.000482499   0.000101358   8.17888e-05      24     
produce:wclsdatavd:WireCellToolkit                40.9447       61.1365       114.842       56.6964       15.0333        24     
produce:gaushit:GausHitFinder                    0.691394       1.25509       2.5159        1.1903        0.44162        24     
produce:nhitsfilter:NumberOfHitsFilter          0.00015252    0.000270824   0.000670714   0.000242772   0.000103631      24     
produce:reco3d:SpacePointSolver                    7.073        12.0982       25.9045       11.4635       4.87439        24     
produce:hitpdune:DisambigFromSpacePoints         0.0896614     0.218576      0.544532      0.172301      0.120323        24     
produce:pandora:StandardPandora                   11.4967        43.71        332.389       27.4069       63.9773        24     
produce:pandoraTrack:LArPandoraTrackCreation     0.386641       1.16897       2.9246       0.894832      0.754153        24     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0172325     0.0331564     0.0513086     0.0335898    0.00965575       24     
[art]:TriggerResults:TriggerResultInserter      2.9264e-05    5.3536e-05    0.000122084   4.91295e-05   1.81217e-05      24     
end_path:out1:RootOutput                         4.306e-06    1.17736e-05   4.0461e-05    1.0221e-05    6.84243e-06      24     
end_path:out1:RootOutput(write)                   3.20892       4.40376       5.95135       4.53665      0.652402        24     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6830.87 MB
  Peak resident set size usage (VmHWM): 4583.16 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040348_1232_df-s04-d1_dw_0_20251104T121647_reco_stage1_20251104T134513_keepup.root
\tHists: np02vd_raw_run040348_1232_df-s04-d1_dw_0_20251104T121647_reco_stage1_20251104T134513_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040348_1232_df-s04-d1_dw_0_20251104T121647_reco_stage1_20251104T134513_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040348_1232_df-s04-d1_dw_0_20251104T121647_reco_stage1_20251104T134513_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762263914.0,
    "core.end_time": 1762263914.0,
    "core.events": [
      113925,
      113929,
      113933,
      113937,
      113941,
      113945,
      113949,
      113953,
      113957,
      113961,
      113965,
      113969,
      113973,
      113977,
      113981,
      113985,
      113989,
      113993,
      113997,
      114001,
      114005,
      114009,
      114013,
      114017
    ],
    "core.event_count": 24,
    "core.first_event_number": 113925,
    "core.last_event_number": 114017,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40348
    ],
    "core.runs_subruns": [
      4034800001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040348_1232_df-s04-d1_dw_0_20251104T121647.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040348_1232_df-s04-d1_dw_0_20251104T121647_reco_stage1_20251104T134513_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762263914.0,
    "core.end_time": 1762263914.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40348
    ],
    "core.runs_subruns": [
      4034800001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040348_1232_df-s04-d1_dw_0_20251104T121647.hdf5"
    }
  ]
}
justIN time: 2025-11-05 09:07:12 UTC       justIN version: 01.05.01