justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.119@dunegpschedd01.fnal.gov

Jobsub ID303652.119@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 03:06:33
From worker nodeHostnamewn5917290.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:28:36
Input filesvd-protodune:np02vd_raw_run042433_0108_df-s04-d0_dw_0_20260203T192311.hdf5
JobscriptExit code0
Real time3h (12861s)
CPU time2h (7613s = 59%)
Max RSS bytes6069665792 (5788 MiB)
Outputting started2026-02-04 03:03:00
Output files
Finished2026-02-04 03:06:33
Saved logsjustin-logs:303652.119-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oint ptr... 0x63135dc0, 0x63135c30
Caught exception in looking up space point ptr... 0x63135c30, 0x5923a7f0
Caught exception in looking up space point ptr... 0x5923a7f0, 0x5923ab10
Caught exception in looking up space point ptr... 0x3151d600, 0x3151cf10
Caught exception in looking up space point ptr... 0x3151ccc0, 0x3151d600
Caught exception in looking up space point ptr... 0xa471830, 0xa4719c0
Caught exception in looking up space point ptr... 0xa4719c0, 0xafd77980
Caught exception in looking up space point ptr... 0xafd77980, 0x322f2320
Caught exception in looking up space point ptr... 0x8dd57700, 0x8dd62080
Caught exception in looking up space point ptr... 0x8dd62080, 0x8dd745e0
Caught exception in looking up space point ptr... 0x4918eb0, 0x4919040
Caught exception in looking up space point ptr... 0x4919040, 0x3a8d18d0
Caught exception in looking up space point ptr... 0x18b71100, 0x18b71290
Caught exception in looking up space point ptr... 0x18b71290, 0x18b67c70
Caught exception in looking up space point ptr... 0x15491df0, 0x15489cd0
Caught exception in looking up space point ptr... 0x15489cd0, 0x15491df0
Caught exception in looking up space point ptr... 0x1959cce0, 0x1959cae0
Caught exception in looking up space point ptr... 0x1959cae0, 0x1959c9e0
Caught exception in looking up space point ptr... 0x1959cbe0, 0x12d25070
Caught exception in looking up space point ptr... 0x12d25070, 0x12d25200
Caught exception in looking up space point ptr... 0x12d25200, 0x1959cce0
Caught exception in looking up space point ptr... 0x80122dd0, 0x39f8aa60
Caught exception in looking up space point ptr... 0x39f8aa60, 0x631374e0
Caught exception in looking up space point ptr... 0x13bce670, 0x13bce550
Caught exception in looking up space point ptr... 0x13bce550, 0x556ff5f0
Caught exception in looking up space point ptr... 0x49245e60, 0x49245bc0
Caught exception in looking up space point ptr... 0x49245bc0, 0x49245e60
Caught exception in looking up space point ptr... 0x3a65cf60, 0x3a65cb10
Caught exception in looking up space point ptr... 0x3a65cb10, 0x3a65cf60
Caught exception in looking up space point ptr... 0x28b729f0, 0x28b72af0
Caught exception in looking up space point ptr... 0x28b72af0, 0x28b77940
Caught exception in looking up space point ptr... 0x29c659e0, 0x2b3d15e0
Caught exception in looking up space point ptr... 0x2b3d15e0, 0x29c659e0
Caught exception in looking up space point ptr... 0x43e1f4a0, 0x43e1f610
Caught exception in looking up space point ptr... 0x43e1f610, 0x43e1f4a0
Caught exception in looking up space point ptr... 0x11bbf9f0, 0x11bbfcd0
Caught exception in looking up space point ptr... 0x11bbfcd0, 0x11bbf9f0
Caught exception in looking up space point ptr... 0x16bf7d90, 0x16bf4080
Caught exception in looking up space point ptr... 0x16bf4080, 0x16bf8460
Caught exception in looking up space point ptr... 0x425d1500, 0x38c62c80
Caught exception in looking up space point ptr... 0x38c62c80, 0x5569d2e0
Caught exception in looking up space point ptr... 0x5569d2e0, 0x1627e980
Caught exception in looking up space point ptr... 0x1627e980, 0x425d1500
Caught exception in looking up space point ptr... 0x2b3ff6b0, 0x2b3fd050
Caught exception in looking up space point ptr... 0x2b3fd050, 0x2b3f00e0
Caught exception in looking up space point ptr... 0x5925ca10, 0x6de36740
Caught exception in looking up space point ptr... 0x6de36740, 0x810c6280
Caught exception in looking up space point ptr... 0x5216800, 0x5216670
Caught exception in looking up space point ptr... 0x5216670, 0x5216800
Caught exception in looking up space point ptr... 0x30f7b620, 0x30f7bfa0
Caught exception in looking up space point ptr... 0x30f7bfa0, 0x30f7b620
Caught exception in looking up space point ptr... 0x48213840, 0x48219f80
Caught exception in looking up space point ptr... 0x48219f80, 0x48223530
Caught exception in looking up space point ptr... 0x115646b0, 0x39fcaa90
Caught exception in looking up space point ptr... 0x39fcaa90, 0x311457e0
Caught exception in looking up space point ptr... 0x36e34830, 0xb754b5c0
Caught exception in looking up space point ptr... 0xb754b5c0, 0x36e34830
Caught exception in looking up space point ptr... 0x28f8cea0, 0x36e0dec0
Caught exception in looking up space point ptr... 0x36e0dec0, 0x28f8cea0
Caught exception in looking up space point ptr... 0xbf328b10, 0x6db63030
Caught exception in looking up space point ptr... 0x6db63030, 0xbf328b10
Caught exception in looking up space point ptr... 0x3a7fd570, 0x3a7fcf90
Caught exception in looking up space point ptr... 0x3a7fcf90, 0x3a7fd570
Caught exception in looking up space point ptr... 0x37b25f80, 0x11e7c780
Caught exception in looking up space point ptr... 0x11e7c780, 0x3dc7b210
Caught exception in looking up space point ptr... 0x6bfad240, 0x6bf9d0f0
Caught exception in looking up space point ptr... 0x6bf9d0f0, 0x6bfad240
++++>>>> total num hits: 95507, num free: 92504
04-Feb-2026 03:02:47 GMT  Closed output file "np02vd_raw_run042433_0108_df-s04-d0_dw_0_20260203T192311_reco_stage0_20260204T030247_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         475.34        668.341       1149.41       557.495       234.09         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000100009   0.000127002   0.000280148   0.000113504   3.93707e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                184.404       253.156       343.519       255.245       42.5654        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.79151       1.25983       2.03585       1.15614      0.329416        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0220809     0.0302981     0.0577046     0.0293095    0.00803509       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0220252     0.022359      0.0230218     0.0223345    0.000217831      19     
produce:beamevent:BeamEvent                      0.000103725   0.000211842   0.00137053     0.0001542    0.000273858      19     
produce:pdvddaphne:DAPHNEReaderPDVD                14.5789       22.3729       30.2434       21.4572       4.07629        19     
produce:ophit:OpHitFinder                         0.0321571     0.0538527     0.0653615     0.0551439    0.00748633       19     
produce:wclsdatavd:WireCellToolkit                 191.11        210.694       225.933       211.828       9.51487        19     
produce:gaushit:GausHitFinder                      1.71369       3.2835        7.75476       2.59029       1.63002        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000358806   0.000708347   0.00122523    0.000600432    0.0002461       19     
produce:reco3d:SpacePointSolver                    15.5811       31.1745       61.8465       26.1656       14.2758        19     
produce:hitpdune:DisambigFromSpacePoints          0.212332      0.636989       1.85684      0.379008      0.524626        19     
produce:cluster3d:Cluster3D                        2.48339       133.732       599.605       4.99383       217.353        19     
[art]:TriggerResults:TriggerResultInserter       2.9746e-05    5.2262e-05    0.000118413   4.8922e-05    2.18779e-05      19     
end_path:out1:RootOutput                          4.248e-06    9.41774e-06   2.8413e-05     6.041e-06    7.01124e-06      19     
end_path:out1:RootOutput(write)                    7.50766       11.8406       22.7273       9.35697       5.09043        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8409.76 MB
  Peak resident set size usage (VmHWM): 6069.67 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0108_df-s04-d0_dw_0_20260203T192311_reco_stage0_20260204T030247_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0108_df-s04-d0_dw_0_20260203T192311_reco_stage0_20260204T030247_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0108_df-s04-d0_dw_0_20260203T192311_reco_stage0_20260204T030247_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161333.0,
    "core.end_time": 1770174170.0,
    "core.events": [
      12322,
      12328,
      12334,
      12340,
      12346,
      12352,
      12358,
      12364,
      12370,
      12376,
      12382,
      12388,
      12394,
      12400,
      12406,
      12412,
      12418,
      12424,
      12430
    ],
    "core.event_count": 19,
    "core.first_event_number": 12322,
    "core.last_event_number": 12430,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0108_df-s04-d0_dw_0_20260203T192311.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:39 UTC       justIN version: 01.06.00