justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 37938.159@dunegpschedd02.fnal.gov

Jobsub ID37938.159@dunegpschedd02.fnal.gov
Workflow ID2482
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-11 00:42:01
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2025-09-11 03:02:00
From worker nodeHostnameb9p10p6632.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-09-11 01:49:57
Input filesvd-protodune:np02vd_raw_run039255_1125_df-s04-d1_dw_0_20250830T223406.hdf5
Outputting started2025-09-11 02:55:58
Output filesvd-protodune-det-reco:np02vd_raw_run039255_1125_df-s04-d1_dw_0_20250830T223406_reco_stage1_20250911T025549_keepup.root
Finished2025-09-11 03:02:00
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

execution complete 
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.36 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.6 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.22 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.1 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.29 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.13 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.67 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.06 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.61 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.59 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.59 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.58 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:53:48.912] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[04:53:48.912] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[04:53:48.912] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[04:53:48.912] I [ timer  ] Timer: Total node execution : 55.82999929599464 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 67978 traces tagged "gauss"
FrameSaver: q=1.20958e+07 n=1509510 tag=gauss
wclsFrameSaver: saving 85315 traces tagged "wiener"
FrameSaver: q=1.29819e+07 n=1443473 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1870 XUs and 2743 XVs -> 130 XUVs
C:0 T:1 675 XUs and 886 XVs -> 44 XUVs
C:0 T:2 315 XUs and 200 XVs -> 15 XUVs
C:0 T:3 321 XUs and 331 XVs -> 17 XUVs
C:0 T:4 1867 XUs and 1766 XVs -> 80 XUVs
C:0 T:5 1080 XUs and 969 XVs -> 47 XUVs
C:0 T:6 387 XUs and 400 XVs -> 34 XUVs
C:0 T:7 1046 XUs and 873 XVs -> 54 XUVs
C:0 T:8 7387 XUs and 13281 XVs -> 2454 XUVs
C:0 T:9 12556 XUs and 19815 XVs -> 1801 XUVs
C:0 T:10 7499 XUs and 10905 XVs -> 2011 XUVs
C:0 T:11 10423 XUs and 12527 XVs -> 1516 XUVs
C:0 T:12 24224 XUs and 31117 XVs -> 3420 XUVs
C:0 T:13 8836 XUs and 9803 XVs -> 1218 XUVs
C:0 T:14 7486 XUs and 14126 XVs -> 1438 XUVs
C:0 T:15 6803 XUs and 5853 XVs -> 547 XUVs
14826 XUVs total
3402 collection wire objects
14826 potential space points
Neighbour search...
1358966 tests to find 654006 neighbours
Iterating with no regularization...
Begin: 9.65709e+08
0 8.25332e+08
1 8.15377e+08
2 8.14021e+08
3 8.13588e+08
Now with regularization...
Begin: 7.917e+08
0 7.91351e+08
PcaShowerParticleBuildingAlgorithm::OpeningAngle - principal eigenvalue less than or equal to 0.
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
11-Sep-2025 04:55:49 CEST  Closed output file "np02vd_raw_run039255_1125_df-s04-d1_dw_0_20250830T223406_reco_stage1_20250911T025549_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        106.772       217.254       656.203       184.144       115.198        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.503e-05    0.000106779   0.000197961   9.2141e-05    3.55986e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               6.0864        7.08327       7.69667       7.17255      0.380698        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00100604    0.00349003     0.0230741    0.00148913    0.00533953       18     
produce:pdvddaphne:DAPHNEReaderPDVD               3.38529       4.10749       5.27157       4.14974       0.44786        18     
produce:ophit:OpHitFinder                        0.0355675     0.0474116     0.0567907     0.0489193    0.00669138       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00699201     0.0130836     0.0248313     0.0137433     0.0044253       18     
produce:wclsdatavd:WireCellToolkit                55.4061       68.9384       85.5372       67.6132       8.57861        18     
produce:gaushit:GausHitFinder                    0.795292       1.44648       2.58195       1.39334      0.410954        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000225241   0.000369116   0.000656433   0.000337607   0.000120294      18     
produce:reco3d:SpacePointSolver                   9.84559       20.4429       31.7618       19.0363       5.3619         18     
produce:hitpdune:DisambigFromSpacePoints         0.112584      0.320029       0.60795      0.330053      0.127873        18     
produce:pandora:StandardPandora                   23.7997       107.58        524.598       79.2285       106.767        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.755779       1.7128        3.19804       1.67541      0.638199        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0231657     0.0391027     0.0696147     0.0377752     0.0139184       18     
[art]:TriggerResults:TriggerResultInserter       1.542e-05    3.00073e-05   5.9431e-05     2.805e-05    1.10578e-05      18     
end_path:out1:RootOutput                         4.27e-06     9.58056e-06    2.319e-05     8.58e-06     4.5654e-06       18     
end_path:out1:RootOutput(write)                   4.41972       5.46918       7.37263       5.2073       0.797531        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5357.22 MB
  Peak resident set size usage (VmHWM): 3395.52 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039255_1125_df-s04-d1_dw_0_20250830T223406_reco_stage1_20250911T025549_keepup.root
\tHists: np02vd_raw_run039255_1125_df-s04-d1_dw_0_20250830T223406_reco_stage1_20250911T025549_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039255_1125_df-s04-d1_dw_0_20250830T223406_reco_stage1_20250911T025549_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039255_1125_df-s04-d1_dw_0_20250830T223406_reco_stage1_20250911T025549_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757559349.0,
    "core.end_time": 1757559349.0,
    "core.events": [
      405824,
      405844,
      405864,
      405884,
      405904,
      405924,
      405944,
      405964,
      405984,
      406004,
      406024,
      406044,
      406064,
      406084,
      406104,
      406124,
      406144,
      406164
    ],
    "core.event_count": 18,
    "core.first_event_number": 405824,
    "core.last_event_number": 406164,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39255
    ],
    "core.runs_subruns": [
      3925500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039255_1125_df-s04-d1_dw_0_20250830T223406.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039255_1125_df-s04-d1_dw_0_20250830T223406_reco_stage1_20250911T025549_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757559349.0,
    "core.end_time": 1757559349.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39255
    ],
    "core.runs_subruns": [
      3925500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039255_1125_df-s04-d1_dw_0_20250830T223406.hdf5"
    }
  ]
}
justIN time: 2025-09-19 03:18:03 UTC       justIN version: 01.05.00