Jobsub ID 37938.155@dunegpschedd02.fnal.gov

Jobsub ID	37938.155@dunegpschedd02.fnal.gov
Workflow ID	2482
Stage ID	1
User name	ykermaid@fnal.gov
HTCondor Group	group_dune.prod_mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-11 00:42:01
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2025-09-11 03:08:29
From worker node	Hostname	b9p07p4996.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	aborted
Started	2025-09-11 01:49:36
Input files	vd-protodune:np02vd_raw_run039255_1033_df-s01-d3_dw_0_20250830T213107.hdf5
Outputting started	2025-09-11 03:01:53
Output files	vd-protodune-det-reco:np02vd_raw_run039255_1033_df-s01-d3_dw_0_20250830T213107_reco_stage1_20250911T030136_keepup.root
Finished	2025-09-11 03:08:29
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

r:> graph execution complete 
[04:59:49.975] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.94 sec
[04:59:49.975] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.19 sec
[04:59:49.975] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.93 sec
[04:59:49.975] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.1 sec
[04:59:49.975] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.99 sec
[04:59:49.975] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.92 sec
[04:59:49.975] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.44 sec
[04:59:49.975] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.67 sec
[04:59:49.975] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.83 sec
[04:59:49.975] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.67 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.65 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.62 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:59:49.976] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[04:59:49.976] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[04:59:49.976] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[04:59:49.976] I [ timer  ] Timer: Total node execution : 65.00999885424972 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 57666 traces tagged "gauss"
FrameSaver: q=1.35237e+07 n=1464715 tag=gauss
wclsFrameSaver: saving 72972 traces tagged "wiener"
FrameSaver: q=1.4473e+07 n=1408408 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1581 XUs and 2171 XVs -> 68 XUVs
C:0 T:1 3377 XUs and 4006 XVs -> 173 XUVs
C:0 T:2 1228 XUs and 1265 XVs -> 42 XUVs
C:0 T:3 769 XUs and 647 XVs -> 34 XUVs
C:0 T:4 106 XUs and 97 XVs -> 5 XUVs
C:0 T:5 989 XUs and 1009 XVs -> 60 XUVs
C:0 T:6 173 XUs and 220 XVs -> 14 XUVs
C:0 T:7 101 XUs and 135 XVs -> 2 XUVs
C:0 T:8 26221 XUs and 52567 XVs -> 11479 XUVs
C:0 T:9 18725 XUs and 41151 XVs -> 4388 XUVs
C:0 T:10 19144 XUs and 31779 XVs -> 6104 XUVs
C:0 T:11 2846 XUs and 2296 XVs -> 94 XUVs
C:0 T:12 11484 XUs and 21500 XVs -> 1185 XUVs
C:0 T:13 2304 XUs and 6201 XVs -> 616 XUVs
C:0 T:14 1294 XUs and 1047 XVs -> 69 XUVs
C:0 T:15 4199 XUs and 3494 XVs -> 241 XUVs
24574 XUVs total
3054 collection wire objects
24574 potential space points
Neighbour search...
4229890 tests to find 1661174 neighbours
Iterating with no regularization...
Begin: 4.24161e+09
0 3.77778e+09
1 3.75463e+09
2 3.75348e+09
Now with regularization...
Begin: 3.72743e+09
0 3.72687e+09
TestBeamParticleCreationAlgorithm::SetupTestBeamVertex - Test beam particle has no initial vertex
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
11-Sep-2025 05:01:36 CEST  Closed output file "np02vd_raw_run039255_1033_df-s01-d3_dw_0_20250830T213107_reco_stage1_20250911T030136_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        125.11        237.666       317.483       261.273       62.9931        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.768e-05    0.000115014   0.000266752   0.000117241   4.40896e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               5.79166       6.96407       12.8882       6.67271       1.46971        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.000843899   0.00444763     0.0209966    0.00129308     0.0068047       18     
produce:pdvddaphne:DAPHNEReaderPDVD               3.71213       4.30291       4.93354       4.30386      0.312475        18     
produce:ophit:OpHitFinder                        0.0375733     0.0496243     0.060306      0.0507104    0.00630684       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00896656     0.0149103     0.0231492     0.0154295    0.00392833       18     
produce:wclsdatavd:WireCellToolkit                63.8525       77.5337       100.756       76.8517       9.70246        18     
produce:gaushit:GausHitFinder                    0.806371       1.75001       2.45149       1.82053      0.452499        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000282173   0.000396132   0.000551056   0.000385374   7.57554e-05      18     
produce:reco3d:SpacePointSolver                   11.5231       22.5497       31.5117       22.4422       5.96442        18     
produce:hitpdune:DisambigFromSpacePoints         0.159113       0.3856       0.682591       0.35805      0.158897        18     
produce:pandora:StandardPandora                   31.0659       116.383       173.616       143.509       49.3561        18     
produce:pandoraTrack:LArPandoraTrackCreation      0.85922       1.77152       2.95775       1.66444       0.62673        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0200876     0.0412455     0.0658205     0.0385469     0.0140884       18     
[art]:TriggerResults:TriggerResultInserter      1.8671e-05    3.21308e-05   0.000102201   2.70155e-05   1.83466e-05      18     
end_path:out1:RootOutput                         7.44e-06     1.06083e-05    3.913e-05     8.79e-06     7.00667e-06      18     
end_path:out1:RootOutput(write)                   4.87664       5.86833       7.18497       5.67533      0.718703        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5907.85 MB
  Peak resident set size usage (VmHWM): 3970.98 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039255_1033_df-s01-d3_dw_0_20250830T213107_reco_stage1_20250911T030136_keepup.root
\tHists: np02vd_raw_run039255_1033_df-s01-d3_dw_0_20250830T213107_reco_stage1_20250911T030136_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039255_1033_df-s01-d3_dw_0_20250830T213107_reco_stage1_20250911T030136_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039255_1033_df-s01-d3_dw_0_20250830T213107_reco_stage1_20250911T030136_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757559697.0,
    "core.end_time": 1757559697.0,
    "core.events": [
      371874,
      371894,
      371914,
      371934,
      371954,
      371974,
      371994,
      372014,
      372034,
      372054,
      372074,
      372094,
      372114,
      372134,
      372154,
      372174,
      372194,
      372214
    ],
    "core.event_count": 18,
    "core.first_event_number": 371874,
    "core.last_event_number": 372214,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39255
    ],
    "core.runs_subruns": [
      3925500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039255_1033_df-s01-d3_dw_0_20250830T213107.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039255_1033_df-s01-d3_dw_0_20250830T213107_reco_stage1_20250911T030136_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757559697.0,
    "core.end_time": 1757559697.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39255
    ],
    "core.runs_subruns": [
      3925500001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039255_1033_df-s01-d3_dw_0_20250830T213107.hdf5"
    }
  ]
}