Jobsub ID 45987.61@dunegpschedd01.fnal.gov

Jobsub ID	45987.61@dunegpschedd01.fnal.gov
Workflow ID	2575
Stage ID	1
User name	ykermaid@fnal.gov
HTCondor Group	group_dune.prod_mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-13 15:32:15
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2025-09-13 18:13:42
From worker node	Hostname	comp15-00
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-09-13 16:03:35
Input files	vd-protodune:np02vd_raw_run039273_1728_df-s05-d0_dw_0_20250901T131053.hdf5
Outputting started	2025-09-13 17:29:03
Output files
Finished	2025-09-13 18:57:41
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

lsFrameSaver: no samples within desired window for channel 9931
wclsFrameSaver: no samples within desired window for channel 9933
wclsFrameSaver: no samples within desired window for channel 9967
wclsFrameSaver: no samples within desired window for channel 9967
wclsFrameSaver: no samples within desired window for channel 10016
wclsFrameSaver: no samples within desired window for channel 10016
wclsFrameSaver: no samples within desired window for channel 10016
wclsFrameSaver: no samples within desired window for channel 10202
wclsFrameSaver: no samples within desired window for channel 10253
wclsFrameSaver: no samples within desired window for channel 10255
wclsFrameSaver: no samples within desired window for channel 10255
wclsFrameSaver: no samples within desired window for channel 10620
wclsFrameSaver: no samples within desired window for channel 10779
wclsFrameSaver: no samples within desired window for channel 10880
wclsFrameSaver: no samples within desired window for channel 10882
wclsFrameSaver: no samples within desired window for channel 10889
wclsFrameSaver: no samples within desired window for channel 10927
wclsFrameSaver: no samples within desired window for channel 10936
wclsFrameSaver: no samples within desired window for channel 10937
wclsFrameSaver: no samples within desired window for channel 10937
wclsFrameSaver: no samples within desired window for channel 10938
wclsFrameSaver: no samples within desired window for channel 10938
wclsFrameSaver: no samples within desired window for channel 10939
wclsFrameSaver: no samples within desired window for channel 10982
wclsFrameSaver: no samples within desired window for channel 10997
wclsFrameSaver: no samples within desired window for channel 11074
wclsFrameSaver: no samples within desired window for channel 11182
wclsFrameSaver: no samples within desired window for channel 11182
wclsFrameSaver: no samples within desired window for channel 11182
wclsFrameSaver: no samples within desired window for channel 12047
wclsFrameSaver: no samples within desired window for channel 12062
wclsFrameSaver: no samples within desired window for channel 12062
wclsFrameSaver: no samples within desired window for channel 12065
wclsFrameSaver: no samples within desired window for channel 12065
wclsFrameSaver: no samples within desired window for channel 12133
wclsFrameSaver: no samples within desired window for channel 12260
wclsFrameSaver: no samples within desired window for channel 12271
FrameSaver: q=7.06907e+06 n=909905 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 349 XUs and 307 XVs -> 19 XUVs
C:0 T:1 3823 XUs and 3300 XVs -> 221 XUVs
C:0 T:2 102 XUs and 87 XVs -> 3 XUVs
C:0 T:3 2856 XUs and 2882 XVs -> 132 XUVs
C:0 T:4 3052 XUs and 2543 XVs -> 124 XUVs
C:0 T:5 1073 XUs and 1465 XVs -> 61 XUVs
C:0 T:6 1468 XUs and 1258 XVs -> 88 XUVs
C:0 T:7 1036 XUs and 2125 XVs -> 57 XUVs
C:0 T:8 742 XUs and 847 XVs -> 48 XUVs
C:0 T:9 2351 XUs and 3893 XVs -> 118 XUVs
C:0 T:10 287 XUs and 304 XVs -> 22 XUVs
C:0 T:11 289 XUs and 447 XVs -> 57 XUVs
C:0 T:12 968 XUs and 1411 XVs -> 80 XUVs
C:0 T:13 1000 XUs and 975 XVs -> 63 XUVs
C:0 T:14 199 XUs and 185 XVs -> 17 XUVs
C:0 T:15 4980 XUs and 4335 XVs -> 265 XUVs
1375 XUVs total
1114 collection wire objects
1375 potential space points
Neighbour search...
14497 tests to find 6976 neighbours
Iterating with no regularization...
Begin: 4.11169e+08
0 3.97974e+08
1 3.9704e+08
2 3.96994e+08
Now with regularization...
Begin: 3.91019e+08
0 3.91e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
13-Sep-2025 18:28:54 BST  Closed output file "np02vd_raw_run039273_1728_df-s05-d0_dw_0_20250901T131053_reco_stage1_20250913T172854_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        206.011       282.216       384.147       277.656       45.0205        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.6071e-05    7.93466e-05   0.000165512   6.8866e-05    2.37552e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               94.5251       117.289       149.811       115.15        16.554         18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0490016     0.0706872     0.320322      0.0532531     0.0610269       18     
produce:pdvddaphne:DAPHNEReaderPDVD               8.69235       14.3414       20.1554       14.063        3.01304        18     
produce:ophit:OpHitFinder                        0.0463534     0.0597444     0.0702649     0.0607624    0.00640354       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00740774     0.0115163     0.0159802     0.0115458    0.00272208       18     
produce:wclsdatavd:WireCellToolkit                58.9692       65.9668       75.4516       65.4658       4.01337        18     
produce:gaushit:GausHitFinder                    0.695111       1.22565       1.70471       1.23612      0.303208        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000179462   0.000278179   0.000461216   0.000256403   8.02277e-05      18     
produce:reco3d:SpacePointSolver                   7.26543       16.5806       25.5242       16.9894       5.23806        18     
produce:hitpdune:DisambigFromSpacePoints         0.0953056     0.245511      0.374341      0.248183      0.0850201       18     
produce:pandora:StandardPandora                   17.7463       59.9627       120.397       52.3117       28.9446        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.514098       1.14406       1.68232       1.30049      0.364915        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0160052     0.0285648     0.0395689     0.0297878    0.00682008       18     
[art]:TriggerResults:TriggerResultInserter       1.999e-05    3.29754e-05   6.4301e-05    3.08555e-05   9.43332e-06      18     
end_path:out1:RootOutput                         8.38e-06     1.35761e-05    3.163e-05    1.2605e-05    4.78207e-06      18     
end_path:out1:RootOutput(write)                   4.84763       5.22767       5.53803       5.23608      0.194277        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4996.89 MB
  Peak resident set size usage (VmHWM): 3095.45 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039273_1728_df-s05-d0_dw_0_20250901T131053_reco_stage1_20250913T172854_keepup.root
\tHists: np02vd_raw_run039273_1728_df-s05-d0_dw_0_20250901T131053_reco_stage1_20250913T172854_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039273_1728_df-s05-d0_dw_0_20250901T131053_reco_stage1_20250913T172854_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039273_1728_df-s05-d0_dw_0_20250901T131053_reco_stage1_20250913T172854_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757784534.0,
    "core.end_time": 1757784534.0,
    "core.events": [
      622094,
      622114,
      622134,
      622154,
      622174,
      622194,
      622214,
      622234,
      622254,
      622274,
      622294,
      622314,
      622334,
      622354,
      622374,
      622394,
      622414,
      622434
    ],
    "core.event_count": 18,
    "core.first_event_number": 622094,
    "core.last_event_number": 622434,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39273
    ],
    "core.runs_subruns": [
      3927300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039273_1728_df-s05-d0_dw_0_20250901T131053.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039273_1728_df-s05-d0_dw_0_20250901T131053_reco_stage1_20250913T172854_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757784534.0,
    "core.end_time": 1757784534.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39273
    ],
    "core.runs_subruns": [
      3927300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039273_1728_df-s05-d0_dw_0_20250901T131053.hdf5"
    }
  ]
}