Jobsub ID 39938.83@dunegpschedd02.fnal.gov

Jobsub ID	39938.83@dunegpschedd02.fnal.gov
Workflow ID	2606
Stage ID	1
User name	ykermaid@fnal.gov
HTCondor Group	group_dune.prod_mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 09:23:32
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2025-09-15 13:09:10
From worker node	Hostname	comp22-08
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-09-15 10:29:00
Input files	vd-protodune:np02vd_raw_run039275_0241_df-s02-d2_dw_0_20250901T225037.hdf5
Outputting started	2025-09-15 12:14:30
Output files
Finished	2025-09-15 14:10:59
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

in desired window for channel 10365
wclsFrameSaver: no samples within desired window for channel 10366
wclsFrameSaver: no samples within desired window for channel 10366
wclsFrameSaver: no samples within desired window for channel 10367
wclsFrameSaver: no samples within desired window for channel 10367
wclsFrameSaver: no samples within desired window for channel 10368
wclsFrameSaver: no samples within desired window for channel 10409
wclsFrameSaver: no samples within desired window for channel 10536
wclsFrameSaver: no samples within desired window for channel 10537
wclsFrameSaver: no samples within desired window for channel 10540
wclsFrameSaver: no samples within desired window for channel 10541
wclsFrameSaver: no samples within desired window for channel 10542
wclsFrameSaver: no samples within desired window for channel 10542
wclsFrameSaver: no samples within desired window for channel 10543
wclsFrameSaver: no samples within desired window for channel 10544
wclsFrameSaver: no samples within desired window for channel 10552
wclsFrameSaver: no samples within desired window for channel 10742
wclsFrameSaver: no samples within desired window for channel 10963
wclsFrameSaver: no samples within desired window for channel 11195
wclsFrameSaver: no samples within desired window for channel 11195
wclsFrameSaver: no samples within desired window for channel 11196
wclsFrameSaver: no samples within desired window for channel 11197
wclsFrameSaver: no samples within desired window for channel 11197
wclsFrameSaver: no samples within desired window for channel 11197
wclsFrameSaver: no samples within desired window for channel 11620
wclsFrameSaver: no samples within desired window for channel 11621
wclsFrameSaver: no samples within desired window for channel 11621
wclsFrameSaver: no samples within desired window for channel 11622
wclsFrameSaver: no samples within desired window for channel 11633
wclsFrameSaver: no samples within desired window for channel 11633
wclsFrameSaver: no samples within desired window for channel 11634
wclsFrameSaver: no samples within desired window for channel 11634
wclsFrameSaver: no samples within desired window for channel 11634
wclsFrameSaver: no samples within desired window for channel 11635
wclsFrameSaver: no samples within desired window for channel 11636
wclsFrameSaver: no samples within desired window for channel 11637
wclsFrameSaver: no samples within desired window for channel 11638
FrameSaver: q=1.09724e+07 n=1201961 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 421 XUs and 499 XVs -> 10 XUVs
C:0 T:1 848 XUs and 707 XVs -> 28 XUVs
C:0 T:2 2122 XUs and 2217 XVs -> 103 XUVs
C:0 T:3 1068 XUs and 1552 XVs -> 35 XUVs
C:0 T:4 714 XUs and 657 XVs -> 35 XUVs
C:0 T:5 286 XUs and 397 XVs -> 14 XUVs
C:0 T:6 688 XUs and 1056 XVs -> 37 XUVs
C:0 T:7 153 XUs and 194 XVs -> 7 XUVs
C:0 T:8 19579 XUs and 33101 XVs -> 8393 XUVs
C:0 T:9 6143 XUs and 12198 XVs -> 1501 XUVs
C:0 T:10 5543 XUs and 6369 XVs -> 1143 XUVs
C:0 T:11 7563 XUs and 14500 XVs -> 1950 XUVs
C:0 T:12 12965 XUs and 25176 XVs -> 2594 XUVs
C:0 T:13 3647 XUs and 4544 XVs -> 438 XUVs
C:0 T:14 1863 XUs and 2638 XVs -> 373 XUVs
C:0 T:15 2644 XUs and 3086 XVs -> 157 XUVs
16818 XUVs total
2443 collection wire objects
16818 potential space points
Neighbour search...
2678454 tests to find 1097832 neighbours
Iterating with no regularization...
Begin: 6.90684e+08
0 6.15363e+08
1 6.10833e+08
2 6.10067e+08
3 6.09791e+08
Now with regularization...
Begin: 5.97518e+08
0 5.97269e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
15-Sep-2025 13:14:17 BST  Closed output file "np02vd_raw_run039275_0241_df-s02-d2_dw_0_20250901T225037_reco_stage1_20250915T121417_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        271.067       348.205       473.749       337.862       57.2521        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      9.9285e-05    0.000117863   0.000231963   0.000105359   3.22897e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               99.8672       140.114       193.738       141.374       24.8563        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0494826     0.0536697     0.0692654     0.0498688    0.00695354       18     
produce:pdvddaphne:DAPHNEReaderPDVD               15.3867        20.77        26.2718       20.9574       3.43914        18     
produce:ophit:OpHitFinder                        0.0426727     0.0524405     0.0631793     0.0518921    0.00585667       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00739576     0.0136884     0.0208531     0.0144257    0.00358542       18     
produce:wclsdatavd:WireCellToolkit                71.1049       84.2584       104.097       82.4043       7.99632        18     
produce:gaushit:GausHitFinder                     1.15209       1.58905       2.26122       1.52928      0.310983        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000388724   0.000506338   0.000711095   0.000474838   8.01309e-05      18     
produce:reco3d:SpacePointSolver                   9.15068       17.2752       30.1599       16.0546       5.18349        18     
produce:hitpdune:DisambigFromSpacePoints         0.133026      0.324772       0.57769      0.308362       0.12893        18     
produce:pandora:StandardPandora                   30.6118       76.4101       167.684       60.2097       39.4077        18     
produce:pandoraTrack:LArPandoraTrackCreation      0.87273       2.44214       5.59656       1.90069       1.44504        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0225335     0.0350713     0.0546583     0.0313957     0.0093039       18     
[art]:TriggerResults:TriggerResultInserter      2.1378e-05    3.45014e-05   8.5369e-05    3.00075e-05   1.41688e-05      18     
end_path:out1:RootOutput                         8.01e-06     1.06314e-05   2.9158e-05     9.509e-06    4.60535e-06      18     
end_path:out1:RootOutput(write)                   4.46321       4.81124       5.36503       4.72273      0.252086        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5076.21 MB
  Peak resident set size usage (VmHWM): 3186.58 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0241_df-s02-d2_dw_0_20250901T225037_reco_stage1_20250915T121417_keepup.root
\tHists: np02vd_raw_run039275_0241_df-s02-d2_dw_0_20250901T225037_reco_stage1_20250915T121421_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039275_0241_df-s02-d2_dw_0_20250901T225037_reco_stage1_20250915T121417_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039275_0241_df-s02-d2_dw_0_20250901T225037_reco_stage1_20250915T121417_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757938461.0,
    "core.end_time": 1757938461.0,
    "core.events": [
      86764,
      86784,
      86804,
      86824,
      86844,
      86864,
      86884,
      86904,
      86924,
      86944,
      86964,
      86984,
      87004,
      87024,
      87044,
      87064,
      87084,
      87104
    ],
    "core.event_count": 18,
    "core.first_event_number": 86764,
    "core.last_event_number": 87104,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39275
    ],
    "core.runs_subruns": [
      3927500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039275_0241_df-s02-d2_dw_0_20250901T225037.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039275_0241_df-s02-d2_dw_0_20250901T225037_reco_stage1_20250915T121421_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757938461.0,
    "core.end_time": 1757938461.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39275
    ],
    "core.runs_subruns": [
      3927500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039275_0241_df-s02-d2_dw_0_20250901T225037.hdf5"
    }
  ]
}