justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 298006.0@dunegpschedd02.fnal.gov

Jobsub ID298006.0@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:34:31
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 04:28:49
From worker nodeHostnamewn2205251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting
Started2026-02-04 01:35:23
Input filesvd-protodune:np02vd_raw_run042422_0110_df-s03-d1_dw_0_20260203T161415.hdf5
JobscriptExit code0
Real time2h (10402s)
CPU time1h (4418s = 42%)
Max RSS bytes2710028288 (2584 MiB)
Outputting started2026-02-04 04:28:49
Output files
Finished 
List job events     Live HTCondor job logs

Jobscript log (last 10,000 characters)

ught exception in looking up space point ptr... 0x2b38e6a0, 0x27fbf5f0
Caught exception in looking up space point ptr... 0x27fbf5f0, 0x27fad490
Caught exception in looking up space point ptr... 0x2b38e060, 0x27fbefb0
Caught exception in looking up space point ptr... 0x27fbefb0, 0x27face50
Caught exception in looking up space point ptr... 0x112dfea0, 0x112e5ca0
Caught exception in looking up space point ptr... 0x12b15df0, 0x112dfea0
Caught exception in looking up space point ptr... 0x44422510, 0x444333b0
Caught exception in looking up space point ptr... 0x44678b10, 0x44422510
Caught exception in looking up space point ptr... 0xad5dbd0, 0x4ec51bc0
Caught exception in looking up space point ptr... 0x3d429a00, 0xad5dbd0
Caught exception in looking up space point ptr... 0x267e7af0, 0x4ee77c10
Caught exception in looking up space point ptr... 0x4ee77c10, 0x1599ead0
Caught exception in looking up space point ptr... 0x3ea637d0, 0xba554f0
Caught exception in looking up space point ptr... 0xba554f0, 0x4e6ceba0
Caught exception in looking up space point ptr... 0x280c45d0, 0x138d1b50
Caught exception in looking up space point ptr... 0x280d6f00, 0x280c45d0
Caught exception in looking up space point ptr... 0x3097b850, 0x3098d4b0
Caught exception in looking up space point ptr... 0x595c4bf0, 0x3097b850
Caught exception in looking up space point ptr... 0x4312fe30, 0x4311e1d0
Caught exception in looking up space point ptr... 0x43142260, 0x4312fe30
Caught exception in looking up space point ptr... 0x617a0900, 0x61792990
Caught exception in looking up space point ptr... 0x617aea00, 0x617a0900
Caught exception in looking up space point ptr... 0x6179e380, 0x61790410
Caught exception in looking up space point ptr... 0x617ac480, 0x6179e380
Caught exception in looking up space point ptr... 0x102e0f60, 0x102ce810
Caught exception in looking up space point ptr... 0x461c7540, 0x102e0f60
Caught exception in looking up space point ptr... 0x269ec980, 0x4b25e780
Caught exception in looking up space point ptr... 0x4b25e780, 0x269ec980
Caught exception in looking up space point ptr... 0x269ec980, 0x479a5a70
Caught exception in looking up space point ptr... 0x479a5a70, 0x269ec980
Caught exception in looking up space point ptr... 0x47cd0b30, 0x46f4d100
Caught exception in looking up space point ptr... 0x47cd35b0, 0x47cd0b30
Caught exception in looking up space point ptr... 0x58cbdd90, 0x34480ca0
Caught exception in looking up space point ptr... 0x34480ca0, 0x24a6eff0
Caught exception in looking up space point ptr... 0xe368df0, 0x356e1e60
Caught exception in looking up space point ptr... 0x2e6ca2a0, 0xe368df0
Caught exception in looking up space point ptr... 0x3e814a80, 0x3e8147e0
Caught exception in looking up space point ptr... 0x3e8147e0, 0x3e8142a0
Caught exception in looking up space point ptr... 0x5a115f50, 0x42fa2c40
Caught exception in looking up space point ptr... 0x42fa2c40, 0x5ccf94b0
Caught exception in looking up space point ptr... 0x4eb1a4c0, 0x4eb08540
Caught exception in looking up space point ptr... 0x5cea19a0, 0x4eb1a4c0
Caught exception in looking up space point ptr... 0x2b66c090, 0x4705f120
Caught exception in looking up space point ptr... 0x47068d00, 0x4705cd30
Caught exception in looking up space point ptr... 0x4705cd30, 0x2b66c090
Caught exception in looking up space point ptr... 0x5c846fa0, 0x213d6cf0
Caught exception in looking up space point ptr... 0x5c858d90, 0x5c846fa0
Caught exception in looking up space point ptr... 0x21919e70, 0x21907ef0
Caught exception in looking up space point ptr... 0x21907ef0, 0x3099ac50
Caught exception in looking up space point ptr... 0x47da6c60, 0x3ce9efe0
Caught exception in looking up space point ptr... 0x3ce9efe0, 0x47da6ad0
Caught exception in looking up space point ptr... 0x43282500, 0x2c096340
Caught exception in looking up space point ptr... 0x2c096340, 0x2c084a00
Caught exception in looking up space point ptr... 0x31d4ad10, 0x31d4ae10
Caught exception in looking up space point ptr... 0x2b4492e0, 0x31d4ad10
Caught exception in looking up space point ptr... 0x4a2ffee0, 0x4a2fff80
Caught exception in looking up space point ptr... 0x4a2fff80, 0x4a2ffee0
Caught exception in looking up space point ptr... 0x2a243200, 0x123bced0
Caught exception in looking up space point ptr... 0x123bced0, 0x2a243200
Caught exception in looking up space point ptr... 0x53080ff0, 0x52bf8820
Caught exception in looking up space point ptr... 0x52bf8820, 0x53080ff0
Caught exception in looking up space point ptr... 0x2a242d60, 0x2a242fb0
Caught exception in looking up space point ptr... 0x2a242fb0, 0x4705a340
Caught exception in looking up space point ptr... 0x2e82a160, 0x36096fb0
Caught exception in looking up space point ptr... 0x36096fb0, 0x2e82a160
++++>>>> total num hits: 282992, num free: 277179
04-Feb-2026 04:28:36 GMT  Closed output file "np02vd_raw_run042422_0110_df-s03-d1_dw_0_20260203T161415_reco_stage0_20260204T042836_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         277.173       381.003       431.785       390.827       40.2025        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        8.506e-05    0.000149371   0.000308561   0.000124825   5.57827e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                110.549       198.506       241.334       206.008       33.1951        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.807983       1.11816       1.5705        1.08599      0.212719        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0220656     0.0327807     0.0752544     0.0276589     0.0157707       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.108816      0.115414       0.14821      0.110123      0.0108695       27     
produce:beamevent:BeamEvent                      0.000123061   0.000207585    0.0010652    0.000175741   0.000171002      27     
produce:pdvddaphne:DAPHNEReaderPDVD                15.8346       24.8908       30.9193       25.5539       3.86176        27     
produce:ophit:OpHitFinder                         0.0315257     0.0451991     0.0585834     0.0458718    0.00552175       27     
produce:wclsdatavd:WireCellToolkit                 85.6797       98.5953       106.442       101.808       6.49196        27     
produce:gaushit:GausHitFinder                     0.979781       1.82721       2.52428       1.85446      0.429321        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00018049    0.000396238   0.000611131   0.000399142   8.84875e-05      27     
produce:reco3d:SpacePointSolver                    6.03125       11.508        17.4548       10.7354       3.33583        27     
produce:hitpdune:DisambigFromSpacePoints          0.104209      0.256623      0.451868       0.26983      0.108292        27     
produce:cluster3d:Cluster3D                        1.55513       37.622        59.4104       49.907        23.012         27     
[art]:TriggerResults:TriggerResultInserter       2.7642e-05    4.15538e-05   9.3166e-05    4.0095e-05    1.27161e-05      27     
end_path:out1:RootOutput                          4.178e-06    9.5253e-06    2.3664e-05    1.0039e-05    3.50371e-06      27     
end_path:out1:RootOutput(write)                    4.8692        6.44129       7.60689       6.97454      0.863888        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4670.05 MB
  Peak resident set size usage (VmHWM): 2710.03 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0110_df-s03-d1_dw_0_20260203T161415_reco_stage0_20260204T042836_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0110_df-s03-d1_dw_0_20260203T161415_reco_stage0_20260204T042836_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0110_df-s03-d1_dw_0_20260203T161415_reco_stage0_20260204T042836_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770168937.0,
    "core.end_time": 1770179318.0,
    "core.events": [
      17835,
      17841,
      17847,
      17853,
      17859,
      17865,
      17871,
      17877,
      17883,
      17889,
      17895,
      17901,
      17907,
      17913,
      17919,
      17925,
      17931,
      17937,
      17943,
      17949,
      17955,
      17961,
      17967,
      17973,
      17979,
      17985,
      17991
    ],
    "core.event_count": 27,
    "core.first_event_number": 17835,
    "core.last_event_number": 17991,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0110_df-s03-d1_dw_0_20260203T161415.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:31:38 UTC       justIN version: 01.06.00