Jobsub ID 302149.1@dunegpschedd01.fnal.gov

Jobsub ID	302149.1@dunegpschedd01.fnal.gov
Workflow ID	12525
Stage ID	1
User name	ykermaid@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	14680064000 (14000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-29 16:10:07
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2026-01-30 07:54:36
From worker node	Hostname	b9p17p7907.cern.ch
	cpuinfo	Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	15728640000 (15000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-01-29 16:10:45
Input files	monte-carlo-012525-000001
Jobscript	Exit code	0
	Real time	15h (55002s)
	CPU time	15h (55955s = 101%)
	Max RSS bytes	64615424000 (61622 MiB)
Outputting started	2026-01-30 07:27:28
Output files	usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen.root usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen.log usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen.err usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1.root usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1.log usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1.err usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2.root usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2.log usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2.err usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim.root usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim.log usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim.err usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim_reco_stage0.root usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim_reco_stage0_20260130T064336_sim_hists.root usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim_reco_stage0.log usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim_reco_stage0.err usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim_reco_stage0_reco_stage1.root usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim_reco_stage0_reco_stage1_20260130T072713_sim_hists.root usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim_reco_stage0_reco_stage1.log usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim_reco_stage0_reco_stage1.err usertests:prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2.root.json
Finished	2026-01-30 07:54:36
Saved logs	justin-logs:302149.1-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

n: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  30-Jan-2026 08:24:17 CET run: 1 subRun: 0 event: 10
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
Attempted to load data: generator
Attempted to load data: largeant LArG4DetectorServicevolTPCActive
Failed to get MCTruth from generator
Failed to get MCTruth from rain
Failed to get MCTruth from rain2
30-Jan-2026 08:27:13 CET  Closed output file "prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim_reco_stage0_reco_stage1.root"
30-Jan-2026 08:27:13 CET  Closed input file "prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim_reco_stage0.root"

===========================================================================================================================================
TimeTracker printout (sec)                                   Min           Avg           Max         Median          RMS         nEvts   
===========================================================================================================================================
Full event                                                 174.157       255.97        450.804       239.318       73.7162        10     
-------------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                   0.00075662    0.00143095    0.00206277    0.00131131    0.000432871      10     
produce:opflash:OpFlashFinderVerticalDrift                0.0176034     0.0211593     0.0277195     0.0209213    0.00278857       10     
produce:pandora:StandardPandora                            14.8254       32.0161       118.148       22.0559       29.2734        10     
produce:pandoraWriter:StandardPandora                     0.070033      0.147533      0.196811      0.154912      0.0409198       10     
produce:pandoraTrack:LArPandoraTrackCreation              0.377013      0.639628       1.02498       0.62529      0.183571        10     
produce:pandoraShower:LArPandoraModularShowerCreation     0.487694      0.683929      0.813244      0.727141       0.11465        10     
produce:pandoraGnocalo:GnocchiCalorimetry                 0.0358378     0.049441      0.0605694     0.0504097    0.00750987       10     
produce:pandorapid:Chi2ParticleID                        0.00164614    0.00204382    0.00372166    0.00188204    0.000572323      10     
[art]:TriggerResults:TriggerResultInserter               1.7988e-05    2.55012e-05   6.6112e-05    2.08835e-05   1.3664e-05       10     
end_path:out1:RootOutput                                  3.679e-06    5.4624e-06    1.8933e-05    3.9975e-06    4.49622e-06      10     
end_path:supera:LArSoftSuperaDriver                        155.343       222.01        330.361       215.959       47.3158        10     
end_path:out1:RootOutput(write)                           0.165639      0.365931      0.443443      0.390688      0.0746105       10     
===========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2855.98 MB
  Peak resident set size usage (VmHWM): 1757.27 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2_detsim_reco_stage0_reco_stage1.root', 'start_time': 1769755417.9047053, 'end_time': 1769758034.278507}
ls: cannot access *_reco.root: No such file or directory
Forming g4 metadata
  G4 file: prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2.root
    KEY: g4_stage2
    FCL:  protodunevd_g4_stage2_mcreco.fcl
TClass::Init:0: RuntimeWarning: no dictionary for class sim::IDE is available
Successfully opened file prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2.root
  Ran successfully
{
  "name": "prod_cosmics_protodunevd_1_302149_1_1769703058_gen_g4_stage1_g4_stage2.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "detector-simulated",
    "core.runs": "1",
    "core.runs_subruns": "1",
    "dune.config_file": "protodunevd_g4_stage2_mcreco.fcl",
    "dune.campaign": "pdvd-cosmics-pds-test",
    "core.start_time": 1769703056.0,
    "core.end_time": 1769758036.0,
    "core.subruns": "1",
    "core.events": [
      1,
      2,
      3,
      4,
      5,
      6,
      7,
      8,
      9,
      10
    ],
    "core.event_count": 10,
    "core.first_event_number": 1,
    "core.last_event_number": 10
  }
}Forming reco metadata