justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.1@dunegpschedd01.fnal.gov

Jobsub ID303656.1@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2026-02-04 00:57:56
From worker nodeHostnameheplnc172.pp.rl.ac.uk
cpuinfoAMD EPYC 9534 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:10:53
Input filesvd-protodune:np02vd_raw_run042421_0112_df-s03-d1_dw_0_20260203T143135.hdf5
JobscriptExit code0
Real time44m (2678s)
CPU time38m (2320s = 86%)
Max RSS bytes2522345472 (2405 MiB)
Outputting started2026-02-04 00:55:32
Output files
Finished2026-02-04 00:57:56
Saved logsjustin-logs:303656.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

5010, 0x45288ef0
Caught exception in looking up space point ptr... 0x45288ef0, 0x2b754770
Caught exception in looking up space point ptr... 0x2b754770, 0x2b754a10
Caught exception in looking up space point ptr... 0x2b754a10, 0x2b7561b0
Caught exception in looking up space point ptr... 0x55691320, 0x55691640
Caught exception in looking up space point ptr... 0x55691640, 0x55695de0
Caught exception in looking up space point ptr... 0x55695de0, 0x55695ac0
Caught exception in looking up space point ptr... 0x55695ac0, 0x55695930
Caught exception in looking up space point ptr... 0x7523430, 0x7523290
Caught exception in looking up space point ptr... 0x7523290, 0x7523430
Caught exception in looking up space point ptr... 0x61336090, 0x5219c490
Caught exception in looking up space point ptr... 0x5219c490, 0x521a7170
Caught exception in looking up space point ptr... 0x360047b0, 0x4d2119e0
Caught exception in looking up space point ptr... 0x4d2119e0, 0x28fc5b50
Caught exception in looking up space point ptr... 0x3e820240, 0x3e821bb0
Caught exception in looking up space point ptr... 0x3e821bb0, 0x3e820140
Caught exception in looking up space point ptr... 0xc1737a0, 0xc1732f0
Caught exception in looking up space point ptr... 0xc1732f0, 0xc1737a0
Caught exception in looking up space point ptr... 0xc1737a0, 0xc173110
Caught exception in looking up space point ptr... 0xc173110, 0xc1737a0
Caught exception in looking up space point ptr... 0x2d91ea20, 0x48867bb0
Caught exception in looking up space point ptr... 0x48867bb0, 0x2d91ea20
Caught exception in looking up space point ptr... 0x2a30e6b0, 0x4834f880
Caught exception in looking up space point ptr... 0x4834f880, 0x4834f780
Caught exception in looking up space point ptr... 0x2c9692f0, 0x2c9691a0
Caught exception in looking up space point ptr... 0x2c9691a0, 0x4ec867c0
Caught exception in looking up space point ptr... 0x156479d0, 0x33bdc560
Caught exception in looking up space point ptr... 0x33bdc560, 0x33bdc740
Caught exception in looking up space point ptr... 0x33a8d880, 0x4c7039f0
Caught exception in looking up space point ptr... 0x4c7039f0, 0x33a8cee0
Caught exception in looking up space point ptr... 0x4af03ed0, 0x4af01960
Caught exception in looking up space point ptr... 0x4af01960, 0x4af03ed0
Caught exception in looking up space point ptr... 0x25f0ccb0, 0x25f06a60
Caught exception in looking up space point ptr... 0x25f0ca60, 0x25f0ccb0
Caught exception in looking up space point ptr... 0x3a3c47b0, 0x3a3d6120
Caught exception in looking up space point ptr... 0x3a3d6120, 0x3a3d3800
Caught exception in looking up space point ptr... 0x282db3a0, 0x4d193330
Caught exception in looking up space point ptr... 0x4d193330, 0x33e78c10
Caught exception in looking up space point ptr... 0x364ceff0, 0x364ce870
Caught exception in looking up space point ptr... 0x364ce870, 0x364ceff0
Caught exception in looking up space point ptr... 0x4f0c5620, 0x4f0b2e20
Caught exception in looking up space point ptr... 0x4f0b2e20, 0x4f0b2fc0
Caught exception in looking up space point ptr... 0x3a3cf0c0, 0x3a3ceec0
Caught exception in looking up space point ptr... 0x3a3cedc0, 0x3a3cf0c0
Caught exception in looking up space point ptr... 0x2c931460, 0x2c931500
Caught exception in looking up space point ptr... 0x2c931500, 0x2c9315d0
Caught exception in looking up space point ptr... 0x6130bb70, 0x6130a960
Caught exception in looking up space point ptr... 0x6130a960, 0x48cdbd80
Caught exception in looking up space point ptr... 0x5568a400, 0x5568a6c0
Caught exception in looking up space point ptr... 0x5568a6c0, 0x5568a400
Caught exception in looking up space point ptr... 0x1a980d70, 0x5bfdb9a0
Caught exception in looking up space point ptr... 0x5bfdb9a0, 0x5bfdbbb0
Caught exception in looking up space point ptr... 0xce2aa90, 0xe0cc560
Caught exception in looking up space point ptr... 0xe0cc560, 0xce2aa90
Caught exception in looking up space point ptr... 0x7536670, 0x7536800
Caught exception in looking up space point ptr... 0x7536800, 0x75383c0
Caught exception in looking up space point ptr... 0x2589cce0, 0x2589cae0
Caught exception in looking up space point ptr... 0x2589cae0, 0x2589cde0
Caught exception in looking up space point ptr... 0x30625f40, 0x2e8ccc50
Caught exception in looking up space point ptr... 0x2e8ccc50, 0x142d9730
Caught exception in looking up space point ptr... 0xa447630, 0x105a77e0
Caught exception in looking up space point ptr... 0x105a77e0, 0x33bda950
Caught exception in looking up space point ptr... 0x18f42f70, 0xc536530
Caught exception in looking up space point ptr... 0xc536a60, 0x18f42f70
Caught exception in looking up space point ptr... 0x2b205ec0, 0x2b2065e0
Caught exception in looking up space point ptr... 0x2b2065e0, 0x2b205ec0
++++>>>> total num hits: 52051, num free: 50357
04-Feb-2026 00:55:23 GMT  Closed output file "np02vd_raw_run042421_0112_df-s03-d1_dw_0_20260203T143135_reco_stage0_20260204T005523_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         78.754        95.4912       116.326       96.1456       8.00688        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.7121e-05     0.0001409    0.000432345   9.6111e-05    8.31058e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                10.7634       12.7474       25.482        12.2432       2.59886        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.526896      0.546508      0.581835      0.547264      0.0132613       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176781     0.0182021     0.0259298     0.0178403    0.00152526       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0874644     0.0887575     0.105994      0.0880248    0.00341211       27     
produce:beamevent:BeamEvent                      0.000118572   0.000197613   0.00114209    0.000163682   0.000186545      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.00108       3.22589       3.42419       3.20615      0.117795        27     
produce:ophit:OpHitFinder                         0.0303836     0.0407656     0.0465149     0.0416521    0.00409245       27     
produce:wclsdatavd:WireCellToolkit                 50.1412       64.7395       72.5915       66.2008       6.2534         27     
produce:gaushit:GausHitFinder                     0.517449      0.891567       1.36406      0.829725      0.242791        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000126361   0.000253975   0.00112898    0.000233653   0.000179203      27     
produce:reco3d:SpacePointSolver                    4.25144       7.57965       10.9699       7.39852       1.87931        27     
produce:hitpdune:DisambigFromSpacePoints          0.0437196     0.0935106     0.146917      0.0848476     0.0300291       27     
produce:cluster3d:Cluster3D                       0.665172       1.65531       3.78606       1.3138       0.726501        27     
[art]:TriggerResults:TriggerResultInserter        2.625e-05    3.97144e-05   0.000127071   3.5531e-05    1.85443e-05      27     
end_path:out1:RootOutput                          3.649e-06    8.53741e-06    3.331e-05     8.52e-06     5.36142e-06      27     
end_path:out1:RootOutput(write)                    2.91041       3.83312       4.75543       4.08616      0.583827        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4526.08 MB
  Peak resident set size usage (VmHWM): 2522.35 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0112_df-s03-d1_dw_0_20260203T143135_reco_stage0_20260204T005523_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0112_df-s03-d1_dw_0_20260203T143135_reco_stage0_20260204T005523_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0112_df-s03-d1_dw_0_20260203T143135_reco_stage0_20260204T005523_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163867.0,
    "core.end_time": 1770166524.0,
    "core.events": [
      18213,
      18219,
      18225,
      18231,
      18237,
      18243,
      18249,
      18255,
      18261,
      18267,
      18273,
      18279,
      18285,
      18291,
      18297,
      18303,
      18309,
      18315,
      18321,
      18327,
      18333,
      18339,
      18345,
      18351,
      18357,
      18363,
      18369
    ],
    "core.event_count": 27,
    "core.first_event_number": 18213,
    "core.last_event_number": 18369,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0112_df-s03-d1_dw_0_20260203T143135.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:26:57 UTC       justIN version: 01.06.00