Jobsub ID 44686.20@dunegpschedd01.fnal.gov

Jobsub ID	44686.20@dunegpschedd01.fnal.gov
Workflow ID	2481
Stage ID	1
User name	ykermaid@fnal.gov
HTCondor Group	group_dune.prod_mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-11 00:39:58
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce509
Last heartbeat	2025-09-11 03:09:25
From worker node	Hostname	b9p02p6404.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	aborted
Started	2025-09-11 02:08:07
Input files	vd-protodune:np02vd_raw_run039350_1721_df-s04-d0_dw_0_20250911T012804.hdf5
Outputting started	2025-09-11 02:59:19
Output files	vd-protodune-det-reco:np02vd_raw_run039350_1721_df-s04-d0_dw_0_20250911T012804_reco_stage1_20250911T025907_keepup.root
Finished	2025-09-11 03:09:25
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

igProc : 7.46 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.45 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.23 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.57 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.16 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.1 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.85 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.09 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.05 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.78 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.66 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[04:57:19.013] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[04:57:19.013] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[04:57:19.013] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[04:57:19.013] I [ timer  ] Timer: Total node execution : 58.20999963954091 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 52570 traces tagged "gauss"
FrameSaver: q=8.4719e+06 n=1095745 tag=gauss
wclsFrameSaver: saving 62793 traces tagged "wiener"
FrameSaver: q=8.86651e+06 n=1062986 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 926 XUs and 1174 XVs -> 57 XUVs
C:0 T:1 1962 XUs and 1759 XVs -> 110 XUVs
C:0 T:2 774 XUs and 678 XVs -> 30 XUVs
C:0 T:3 1096 XUs and 1249 XVs -> 35 XUVs
C:0 T:4 289 XUs and 353 XVs -> 11 XUVs
C:0 T:5 781 XUs and 1052 XVs -> 26 XUVs
C:0 T:6 373 XUs and 457 XVs -> 14 XUVs
C:0 T:7 704 XUs and 758 XVs -> 34 XUVs
C:0 T:8 1739 XUs and 1451 XVs -> 60 XUVs
C:0 T:9 1667 XUs and 1606 XVs -> 87 XUVs
C:0 T:10 4076 XUs and 3206 XVs -> 166 XUVs
C:0 T:11 1682 XUs and 2478 XVs -> 120 XUVs
C:0 T:12 2865 XUs and 3467 XVs -> 233 XUVs
C:0 T:13 1327 XUs and 1803 XVs -> 103 XUVs
C:0 T:14 918 XUs and 1117 XVs -> 52 XUVs
C:0 T:15 1499 XUs and 1627 XVs -> 107 XUVs
1245 XUVs total
1080 collection wire objects
1245 potential space points
Neighbour search...
7767 tests to find 3636 neighbours
Iterating with no regularization...
Begin: 3.62408e+08
0 3.56964e+08
1 3.56764e+08
Now with regularization...
Begin: 3.4904e+08
0 3.48955e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
11-Sep-2025 04:59:07 CEST  Closed output file "np02vd_raw_run039350_1721_df-s04-d0_dw_0_20250911T012804_reco_stage1_20250911T025907_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        127.425       167.76        251.767       161.348       27.7463        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.2261e-05    0.000119177   0.000553246   8.9581e-05    0.000108272      18     
produce:tpcrawdecoder:PDVDTPCReader               6.54286       7.1184        7.93902       7.07655      0.341461        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00717306     0.0120897     0.0236272    0.00897066    0.00552333       18     
produce:pdvddaphne:DAPHNEReaderPDVD               2.18941       2.51265       2.78442       2.50347      0.178466        18     
produce:ophit:OpHitFinder                        0.0371033     0.0452966     0.0560863     0.0457398    0.00582711       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00623797     0.0115444     0.0202846     0.0109087    0.00354492       18     
produce:wclsdatavd:WireCellToolkit                59.6911       70.0615       83.1445       67.7813       7.22056        18     
produce:gaushit:GausHitFinder                     1.02136       1.46985       2.12687       1.42587      0.295325        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000222452   0.000348279   0.000488895   0.000337908   8.1578e-05       18     
produce:reco3d:SpacePointSolver                   9.9087        16.2039       24.2429       15.4019       3.44923        18     
produce:hitpdune:DisambigFromSpacePoints         0.160638      0.247295      0.379996      0.229442      0.065377        18     
produce:pandora:StandardPandora                   34.1342       62.9596       123.113       57.7159       23.4851        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.444298       1.14173       1.86854       1.11062      0.335056        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0177009     0.0367445     0.0500633     0.0369112    0.00921101       18     
[art]:TriggerResults:TriggerResultInserter       1.786e-05    3.06592e-05   7.0851e-05    2.8365e-05    1.12784e-05      18     
end_path:out1:RootOutput                         4.101e-06    9.58733e-06    3.525e-05     8.285e-06    6.37379e-06      18     
end_path:out1:RootOutput(write)                   4.80043       5.89142       7.24301       5.94875      0.579528        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4861.93 MB
  Peak resident set size usage (VmHWM): 2910.87 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039350_1721_df-s04-d0_dw_0_20250911T012804_reco_stage1_20250911T025907_keepup.root
\tHists: np02vd_raw_run039350_1721_df-s04-d0_dw_0_20250911T012804_reco_stage1_20250911T025907_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039350_1721_df-s04-d0_dw_0_20250911T012804_reco_stage1_20250911T025907_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039350_1721_df-s04-d0_dw_0_20250911T012804_reco_stage1_20250911T025907_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757559548.0,
    "core.end_time": 1757559548.0,
    "core.events": [
      125772,
      125776,
      125780,
      125784,
      125788,
      125792,
      125796,
      125800,
      125804,
      125808,
      125812,
      125816,
      125820,
      125824,
      125828,
      125832,
      125836,
      125840
    ],
    "core.event_count": 18,
    "core.first_event_number": 125772,
    "core.last_event_number": 125840,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39350
    ],
    "core.runs_subruns": [
      3935000001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039350_1721_df-s04-d0_dw_0_20250911T012804.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039350_1721_df-s04-d0_dw_0_20250911T012804_reco_stage1_20250911T025907_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757559548.0,
    "core.end_time": 1757559548.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39350
    ],
    "core.runs_subruns": [
      3935000001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039350_1721_df-s04-d0_dw_0_20250911T012804.hdf5"
    }
  ]
}