justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.66@dunegpschedd01.fnal.gov

Jobsub ID303656.66@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2026-02-04 01:25:52
From worker nodeHostnameheplnc172.pp.rl.ac.uk
cpuinfoAMD EPYC 9534 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:12:03
Input filesvd-protodune:np02vd_raw_run042421_0108_df-s02-d0_dw_0_20260203T142926.hdf5
JobscriptExit code0
Real time1h (4243s)
CPU time1h (3919s = 92%)
Max RSS bytes5277458432 (5032 MiB)
Outputting started2026-02-04 01:22:49
Output files
Finished2026-02-04 01:25:52
Saved logsjustin-logs:303656.66-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

5deeed0
Caught exception in looking up space point ptr... 0x25deeed0, 0x54f63c50
Caught exception in looking up space point ptr... 0x2d58d1d0, 0x25bb8270
Caught exception in looking up space point ptr... 0x25bb8270, 0x25bb8400
Caught exception in looking up space point ptr... 0x3bc660b0, 0x3bc65910
Caught exception in looking up space point ptr... 0x3bc65910, 0x36d06500
Caught exception in looking up space point ptr... 0x2c4f1b40, 0x41e0ec00
Caught exception in looking up space point ptr... 0x41e0ec00, 0x603f7a20
Caught exception in looking up space point ptr... 0x1437db70, 0xe5b0f00
Caught exception in looking up space point ptr... 0x1437d9e0, 0x1437db70
Caught exception in looking up space point ptr... 0x10afc780, 0x10b0ebb0
Caught exception in looking up space point ptr... 0x10b0ebb0, 0x10b0ed40
Caught exception in looking up space point ptr... 0xde78f10, 0x118bb160
Caught exception in looking up space point ptr... 0xde78920, 0x4ffee8b0
Caught exception in looking up space point ptr... 0x4ffee8b0, 0xde78f10
Caught exception in looking up space point ptr... 0x29c13800, 0x29c034a0
Caught exception in looking up space point ptr... 0x29c239d0, 0x29c13800
Caught exception in looking up space point ptr... 0x25c07f80, 0x254c7a00
Caught exception in looking up space point ptr... 0x254c7a00, 0x2f8f4fd0
Caught exception in looking up space point ptr... 0x4dd03700, 0x40b87a30
Caught exception in looking up space point ptr... 0x50975fa0, 0x4dd03700
Caught exception in looking up space point ptr... 0x51830380, 0xd6b1eb0
Caught exception in looking up space point ptr... 0x51842300, 0x51830380
Caught exception in looking up space point ptr... 0xe267ce0, 0xdc35130
Caught exception in looking up space point ptr... 0xe27a430, 0xe267ce0
Caught exception in looking up space point ptr... 0x4bc4fcd0, 0x4bc3e070
Caught exception in looking up space point ptr... 0x2c821320, 0x4bc4fcd0
Caught exception in looking up space point ptr... 0x2ab2f6a0, 0xc39aeed0
Caught exception in looking up space point ptr... 0x2ab1cf50, 0x2ab2f6a0
Caught exception in looking up space point ptr... 0x53d3c2d0, 0x53d2ecc0
Caught exception in looking up space point ptr... 0x53d2ecc0, 0x53d3c140
Caught exception in looking up space point ptr... 0x5edbe6d0, 0x5bcc2d10
Caught exception in looking up space point ptr... 0x5bcc2d10, 0x2d5cb390
Caught exception in looking up space point ptr... 0x59b5d950, 0x31a32d80
Caught exception in looking up space point ptr... 0x53859010, 0x59b5d950
Caught exception in looking up space point ptr... 0xd79b550, 0xd79b3c0
Caught exception in looking up space point ptr... 0xd79b3c0, 0x13af5000
Caught exception in looking up space point ptr... 0x6249f8f0, 0x12f9c950
Caught exception in looking up space point ptr... 0x6249fa80, 0x6249f8f0
Caught exception in looking up space point ptr... 0x3c98d2e0, 0x550e7b90
Caught exception in looking up space point ptr... 0x3c99f8a0, 0x3c98d2e0
Caught exception in looking up space point ptr... 0x62978380, 0x62969150
Caught exception in looking up space point ptr... 0x62969150, 0x5d9d1940
Caught exception in looking up space point ptr... 0x2cf03e40, 0x4d903650
Caught exception in looking up space point ptr... 0x4d903650, 0x9618ab0
Caught exception in looking up space point ptr... 0x9618ab0, 0x280c2c70
Caught exception in looking up space point ptr... 0x5d8b5860, 0xfde97e0
Caught exception in looking up space point ptr... 0xfde97e0, 0x5d8b5860
Caught exception in looking up space point ptr... 0x116f1450, 0x116f16e0
Caught exception in looking up space point ptr... 0x116f16e0, 0x116f1450
Caught exception in looking up space point ptr... 0x2567e960, 0x55810080
Caught exception in looking up space point ptr... 0x55810080, 0x53782050
Caught exception in looking up space point ptr... 0x5daaf200, 0x25972750
Caught exception in looking up space point ptr... 0x25972750, 0x5daaf200
Caught exception in looking up space point ptr... 0x4db02340, 0x4db020a0
Caught exception in looking up space point ptr... 0x4db020a0, 0x4db01e00
Caught exception in looking up space point ptr... 0x1230e560, 0x4f324f00
Caught exception in looking up space point ptr... 0x4f324f00, 0x3dbf2320
Caught exception in looking up space point ptr... 0x50c5d840, 0x2f7fe3f0
Caught exception in looking up space point ptr... 0x2f7fe3f0, 0x50c5d840
Caught exception in looking up space point ptr... 0x36c67f20, 0x50b749d0
Caught exception in looking up space point ptr... 0x50b749d0, 0x4b6778d0
Caught exception in looking up space point ptr... 0x29f38400, 0x29f38840
Caught exception in looking up space point ptr... 0x29f38840, 0x29f38400
Caught exception in looking up space point ptr... 0x31e0d660, 0x25969bf0
Caught exception in looking up space point ptr... 0x25969bf0, 0x3064df00
++++>>>> total num hits: 294557, num free: 288607
04-Feb-2026 01:22:39 GMT  Closed output file "np02vd_raw_run042421_0108_df-s02-d0_dw_0_20260203T142926_reco_stage0_20260204T012239_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         90.6515       154.095       441.265       142.248       68.2555        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.6441e-05    0.000136823   0.000339624   0.000106381   5.88553e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                10.6496       10.9021       11.2147       10.8994       0.14445        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.524642      0.541344      0.563365      0.544878      0.0105164       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0175956     0.0179168     0.0184287     0.0178901    0.000175634      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.087244      0.0950477     0.107388      0.0888852    0.00851289       27     
produce:beamevent:BeamEvent                      0.000125741   0.000194903   0.00103455    0.000158652   0.000166397      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.74802       3.29071       3.73102       3.25496      0.183477        27     
produce:ophit:OpHitFinder                         0.0319601     0.0401594     0.0473489     0.0406859    0.00489233       27     
produce:wclsdatavd:WireCellToolkit                 60.4715       73.6021       84.7687       75.7674       7.21277        27     
produce:gaushit:GausHitFinder                     0.625396       1.62409       3.71908       1.39401       0.72946        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000181712   0.000308501   0.000564227   0.000288124   9.25293e-05      27     
produce:reco3d:SpacePointSolver                    4.39689       8.88408       33.3089       7.50819       5.65341        27     
produce:hitpdune:DisambigFromSpacePoints          0.0595853     0.312014       1.42909      0.220319      0.289164        27     
produce:cluster3d:Cluster3D                        1.21375       49.4138       298.945       38.5944       58.5743        27     
[art]:TriggerResults:TriggerResultInserter        2.632e-05    4.29153e-05   8.6501e-05     3.944e-05    1.21924e-05      27     
end_path:out1:RootOutput                          5.91e-06     1.86176e-05   0.000263463      9e-06      4.81055e-05      27     
end_path:out1:RootOutput(write)                    3.70828       5.32684       12.576        4.91653       1.87109        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7591.35 MB
  Peak resident set size usage (VmHWM): 5277.46 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0108_df-s02-d0_dw_0_20260203T142926_reco_stage0_20260204T012239_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0108_df-s02-d0_dw_0_20260203T142926_reco_stage0_20260204T012239_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0108_df-s02-d0_dw_0_20260203T142926_reco_stage0_20260204T012239_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163935.0,
    "core.end_time": 1770168160.0,
    "core.events": [
      17562,
      17568,
      17574,
      17580,
      17586,
      17592,
      17598,
      17604,
      17610,
      17616,
      17622,
      17628,
      17634,
      17640,
      17646,
      17652,
      17658,
      17664,
      17670,
      17676,
      17682,
      17688,
      17694,
      17700,
      17706,
      17712,
      17718
    ],
    "core.event_count": 27,
    "core.first_event_number": 17562,
    "core.last_event_number": 17718,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0108_df-s02-d0_dw_0_20260203T142926.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:33:13 UTC       justIN version: 01.06.00