Jobsub ID 47706.2@dunegpschedd01.fnal.gov

Jobsub ID	47706.2@dunegpschedd01.fnal.gov
Workflow ID	2816
Stage ID	1
User name	ykermaid@fnal.gov
HTCondor Group	group_dune
Requested	Processors	4
	GPU	No
	RSS bytes	6815744000 (6500 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-17 18:11:32
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw7
Last heartbeat	2025-09-17 22:18:14
From worker node	Hostname	mh-7662-5.t2.ucsd.edu
	cpuinfo	AMD EPYC 7662 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	4
	RSS bytes	7864320000 (7500 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-17 18:12:06
Input files	monte-carlo-002816-000006
Jobscript	Exit code	0
	Real time	3h (14005s)
	CPU time	3h (12794s = 91%)
	Max RSS bytes	6196678656 (5909 MiB)
Outputting started	2025-09-17 22:05:32
Output files	usertests:prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2.root usertests:prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2.log usertests:prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2.err usertests:prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2.root.json usertests:prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2_detsim_reco.root usertests:prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2_detsim_reco.log usertests:prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2_detsim_reco.err usertests:prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2_detsim_reco.root.json
Finished	2025-09-17 22:18:14
Saved logs	justin-logs:47706.2-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oduct: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2025 15:04:57 PDT run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
17-Sep-2025 15:05:01 PDT  Closed output file "prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2_detsim_reco.root"
17-Sep-2025 15:05:01 PDT  Closed input file "prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2_detsim.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              74.6859       112.906       217.488       102.36        37.4902        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00111809    0.00267621    0.00393709     0.0027405    0.000760487      10     
reco:rns:RandomNumberSaver                            5.1811e-05    0.000122927   0.000527242   6.45165e-05   0.000138168      10     
reco:wclsdatavd:WireCellToolkit                         55.7665       63.4541       74.1887       63.3007       5.07723        10     
reco:gaushit:GausHitFinder                             0.750166       1.53671       2.76732       1.46141      0.570524        10     
reco:reco3d:SpacePointSolver                            5.27795       11.8592        30.86        9.56032       7.08651        10     
reco:hitpdune:DisambigFromSpacePoints                  0.052175       0.15513      0.412974       0.13195      0.0959327       10     
reco:pandora:StandardPandora                            8.95158       33.0009       103.46        27.5549       25.4331        10     
reco:pandoraTrack:LArPandoraTrackCreation              0.244226      0.715069       1.75021      0.547883      0.427705        10     
reco:pandoraShower:LArPandoraModularShowerCreation     0.453913      0.740597       1.2477       0.658588      0.246603        10     
reco:pandoraStdcalo:Calorimetry                        0.136116      0.347994      0.810009      0.287924      0.185575        10     
reco:pandoraGnocalo:GnocchiCalorimetry                 0.0278649     0.0472807     0.0950362     0.0423353     0.0176686       10     
[art]:TriggerResults:TriggerResultInserter             2.22e-05     3.06505e-05   7.2351e-05    2.58105e-05   1.42044e-05      10     
end_path:out1:RootOutput                               9.57e-06     1.23892e-05   2.6071e-05    1.03605e-05   4.80073e-06      10     
end_path:out1:RootOutput(write)                        0.604426      0.996119       1.80463      0.935117      0.323625        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5839.22 MB
  Peak resident set size usage (VmHWM): 3576.06 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1758145418.0608602, 'end_time': 1758146703.0057929}
Forming g4 metadata
  G4 file: prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2.root
    KEY: g4_stage2
    FCL:  protodunevd_g4_stage2_nopds.fcl
TClass::Init:0: RuntimeWarning: no dictionary for class sim::IDE is available
Successfully opened file prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2.root
  Ran successfully
{
  "name": "prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_02d00",
    "core.data_tier": "detector-simulated",
    "core.runs": "1",
    "core.runs_subruns": "2",
    "dune.config_file": "protodunevd_g4_stage2_nopds.fcl",
    "dune.campaign": "pdvd-cosmics-pions-test",
    "core.start_time": 1758132739.0,
    "core.end_time": 1758146703.0,
    "core.subruns": "1",
    "core.events": [
      1,
      2,
      3,
      4,
      5,
      6,
      7,
      8,
      9,
      10
    ],
    "core.event_count": 10,
    "core.first_event_number": 1,
    "core.last_event_number": 10
  }
}Forming reco metadata
  subrun: 2 -> reco file: prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2_detsim_reco.root
    KEY: reco
    FCL:  protodunevd_reco_nopds.fcl
TClass::Init:0: RuntimeWarning: no dictionary for class sim::IDE is available
Successfully opened file prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2_detsim_reco.root
  Ran successfully
{
  "name": "prod_cosmics_1GeVpions_protodunevd_nopds_1_47706_2_1758132741_gen_g4_stage1_g4_stage2_detsim_reco.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_02d00",
    "core.data_tier": "full-reconstructed",
    "core.runs": "[1]",
    "core.runs_subruns": "[2]",
    "dune.config_file": "protodunevd_reco_nopds.fcl",
    "dune.campaign": "pdvd-cosmics-pions-test",
    "core.start_time": 1758132739.0,
    "core.end_time": 1758146703.0,
    "core.subruns": "[1]",
    "core.events": [
      1,
      2,
      3,
      4,
      5,
      6,
      7,
      8,
      9,
      10
    ],
    "core.event_count": 10,
    "core.first_event_number": 1,
    "core.last_event_number": 10
  }
}