justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297936.0@dunegpschedd02.fnal.gov

Jobsub ID297936.0@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:32:25
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 03:04:32
From worker nodeHostnamewn5918241.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:14:13
Input filesvd-protodune:np02vd_raw_run042422_0062_df-s02-d1_dw_0_20260203T154818.hdf5
JobscriptExit code0
Real time2h (10051s)
CPU time1h (4351s = 43%)
Max RSS bytes2558341120 (2439 MiB)
Outputting started2026-02-04 03:01:46
Output files
Finished2026-02-04 03:04:32
Saved logsjustin-logs:297936.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0xcc312a0
Caught exception in looking up space point ptr... 0x3cb9edb0, 0x3cb9ec20
Caught exception in looking up space point ptr... 0x3cb9ec20, 0x3cb9ea90
Caught exception in looking up space point ptr... 0x3cb9ea90, 0x3293ef10
Caught exception in looking up space point ptr... 0x3293ef10, 0x17e8b010
Caught exception in looking up space point ptr... 0x17e8b010, 0x17e8b330
Caught exception in looking up space point ptr... 0x17e8b330, 0x3cb9edb0
Caught exception in looking up space point ptr... 0x1518eaa0, 0x1518e8c0
Caught exception in looking up space point ptr... 0x1518e8c0, 0x1518e6f0
Caught exception in looking up space point ptr... 0x1518e6f0, 0x297215c0
Caught exception in looking up space point ptr... 0x297215c0, 0xb11f0d0
Caught exception in looking up space point ptr... 0x1518e2e0, 0x171333b0
Caught exception in looking up space point ptr... 0x171333b0, 0x1518eaa0
Caught exception in looking up space point ptr... 0x38985150, 0x38984f80
Caught exception in looking up space point ptr... 0x52da3c40, 0x38985150
Caught exception in looking up space point ptr... 0x43f34560, 0x43f34380
Caught exception in looking up space point ptr... 0x43f34380, 0x43f34560
Caught exception in looking up space point ptr... 0xb11e8d0, 0xb11e700
Caught exception in looking up space point ptr... 0xb11e700, 0xb11e220
Caught exception in looking up space point ptr... 0x44893f30, 0x44893e30
Caught exception in looking up space point ptr... 0x44893e30, 0xb1a7bd0
Caught exception in looking up space point ptr... 0x5b089680, 0x104d8740
Caught exception in looking up space point ptr... 0x104d8740, 0x180c23b0
Caught exception in looking up space point ptr... 0x180c23b0, 0x1beeebc0
Caught exception in looking up space point ptr... 0x152a4ea0, 0x4477e9d0
Caught exception in looking up space point ptr... 0x4477e9d0, 0x1523cee0
Caught exception in looking up space point ptr... 0x3cd05200, 0x2e411540
Caught exception in looking up space point ptr... 0x2e411540, 0x5e152f30
Caught exception in looking up space point ptr... 0x1b288530, 0x3cd05200
Caught exception in looking up space point ptr... 0x2ffeca00, 0x30053e20
Caught exception in looking up space point ptr... 0x30053e20, 0x50eefd20
Caught exception in looking up space point ptr... 0x50e2cd90, 0x1554be70
Caught exception in looking up space point ptr... 0x1554be70, 0x29b9ff50
Caught exception in looking up space point ptr... 0x29b9ff50, 0xafac300
Caught exception in looking up space point ptr... 0xafac300, 0xe5cee50
Caught exception in looking up space point ptr... 0xe5cee50, 0x2ae292f0
Caught exception in looking up space point ptr... 0x41fb2960, 0x12fe5ee0
Caught exception in looking up space point ptr... 0x12fe5ee0, 0x3dac8a20
Caught exception in looking up space point ptr... 0xcc32700, 0xcc330d0
Caught exception in looking up space point ptr... 0x2e79dde0, 0xcc32700
Caught exception in looking up space point ptr... 0x5113f2b0, 0x104e6210
Caught exception in looking up space point ptr... 0x104e6210, 0x5113f2b0
Caught exception in looking up space point ptr... 0x2f1696c0, 0x341acab0
Caught exception in looking up space point ptr... 0x5291fab0, 0x2f1696c0
Caught exception in looking up space point ptr... 0x3cba18e0, 0x3cba19e0
Caught exception in looking up space point ptr... 0x3cba19e0, 0x3cba18e0
Caught exception in looking up space point ptr... 0x152913a0, 0x34caa260
Caught exception in looking up space point ptr... 0x34caa260, 0x152913a0
Caught exception in looking up space point ptr... 0x50d233a0, 0x50d220b0
Caught exception in looking up space point ptr... 0x50d220b0, 0x50d233a0
Caught exception in looking up space point ptr... 0x36c482f0, 0x33d04040
Caught exception in looking up space point ptr... 0x33d04040, 0x329f6990
Caught exception in looking up space point ptr... 0x59101e00, 0x15300d00
Caught exception in looking up space point ptr... 0x15300d00, 0x15300a80
Caught exception in looking up space point ptr... 0xb1a7d10, 0x44893bc0
Caught exception in looking up space point ptr... 0x44893bc0, 0xb1a7d10
Caught exception in looking up space point ptr... 0x3cb96600, 0x11a998e0
Caught exception in looking up space point ptr... 0x11a998e0, 0x3cb96600
Caught exception in looking up space point ptr... 0x1ba30f00, 0x36bcc320
Caught exception in looking up space point ptr... 0x36bcc320, 0x1ba30f00
Caught exception in looking up space point ptr... 0x411bff60, 0x3c54d300
Caught exception in looking up space point ptr... 0x3c54d300, 0x1b900040
Caught exception in looking up space point ptr... 0x5291cf10, 0x12e22c20
Caught exception in looking up space point ptr... 0x12e22c20, 0x5291cf10
Caught exception in looking up space point ptr... 0x84d9800, 0x448624e0
Caught exception in looking up space point ptr... 0x448624e0, 0x84d9800
++++>>>> total num hits: 41838, num free: 40402
04-Feb-2026 03:01:28 GMT  Closed output file "np02vd_raw_run042422_0062_df-s02-d1_dw_0_20260203T154818_reco_stage0_20260204T030128_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         302.001       365.938       450.866       356.116       35.6468        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.2378e-05    0.000159534   0.000355419   0.00012774    6.83046e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                135.482       189.946       278.909        186.7        32.6562        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.715055       1.78624       13.9578      0.989625       2.61772        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0222216     0.0272575     0.0791371     0.0228795     0.0108328       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.108982      0.115337      0.139578       0.10981     0.00931652       27     
produce:beamevent:BeamEvent                      0.000134371   0.000242488   0.000947426   0.000207905   0.000144156      27     
produce:pdvddaphne:DAPHNEReaderPDVD                21.1613       27.3686       35.5991       27.5536       4.14226        27     
produce:ophit:OpHitFinder                         0.0442348     0.0702359     0.102327      0.0719297     0.013682        27     
produce:wclsdatavd:WireCellToolkit                 95.7901       119.566       133.279       121.788       9.68375        27     
produce:gaushit:GausHitFinder                     0.853084       1.70259       2.3778        1.67906      0.340259        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000361575   0.000557594   0.00114802    0.000550463   0.00014784       27     
produce:reco3d:SpacePointSolver                    7.10147       15.463        23.6617       14.2906       4.09983        27     
produce:hitpdune:DisambigFromSpacePoints          0.107439      0.214799      0.414823       0.2197       0.0656305       27     
produce:cluster3d:Cluster3D                        1.37388       3.18277       6.05596       3.10222       1.09822        27     
[art]:TriggerResults:TriggerResultInserter       3.1754e-05    5.09149e-05   0.000135545   4.6313e-05    2.03312e-05      27     
end_path:out1:RootOutput                          3.981e-06    2.09271e-05   0.000296195    8.982e-06    5.42999e-05      27     
end_path:out1:RootOutput(write)                    5.10924       6.45473       8.59322       6.38485      0.673889        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4538.68 MB
  Peak resident set size usage (VmHWM): 2558.34 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0062_df-s02-d1_dw_0_20260203T154818_reco_stage0_20260204T030128_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0062_df-s02-d1_dw_0_20260203T154818_reco_stage0_20260204T030128_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0062_df-s02-d1_dw_0_20260203T154818_reco_stage0_20260204T030128_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164064.0,
    "core.end_time": 1770174094.0,
    "core.events": [
      10051,
      10057,
      10063,
      10069,
      10075,
      10081,
      10087,
      10093,
      10099,
      10105,
      10111,
      10117,
      10123,
      10129,
      10135,
      10141,
      10147,
      10153,
      10159,
      10165,
      10171,
      10177,
      10183,
      10189,
      10195,
      10201,
      10207
    ],
    "core.event_count": 27,
    "core.first_event_number": 10051,
    "core.last_event_number": 10207,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0062_df-s02-d1_dw_0_20260203T154818.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:27:07 UTC       justIN version: 01.06.00