justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.116@dunegpschedd01.fnal.gov

Jobsub ID303654.116@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce05
Last heartbeat2026-02-04 00:57:57
From worker nodeHostnamedune001-1846108.0-lcg2710.gridpp.rl.ac.uk
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:09:47
Input filesvd-protodune:np02vd_raw_run042433_0111_df-s03-d1_dw_0_20260203T192419.hdf5
JobscriptExit code0
Real time46m (2797s)
CPU time41m (2491s = 89%)
Max RSS bytes3359170560 (3203 MiB)
Outputting started2026-02-04 00:56:25
Output files
Finished2026-02-04 00:57:57
Saved logsjustin-logs:303654.116-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x6213e7f0, 0x6213faa0
Caught exception in looking up space point ptr... 0x6213faa0, 0x6213ebb0
Caught exception in looking up space point ptr... 0x6213ebb0, 0x6213faa0
Caught exception in looking up space point ptr... 0x3f865b60, 0x3f863d70
Caught exception in looking up space point ptr... 0x3f863d70, 0x3f864010
Caught exception in looking up space point ptr... 0x84b84220, 0x58cda050
Caught exception in looking up space point ptr... 0x58cda050, 0x58cd9ba0
Caught exception in looking up space point ptr... 0xa8b9cc70, 0x7ca2b070
Caught exception in looking up space point ptr... 0x7ca2b070, 0x6ee48910
Caught exception in looking up space point ptr... 0x6ee48910, 0x414e84b0
Caught exception in looking up space point ptr... 0x414e84b0, 0xa8b9cc70
Caught exception in looking up space point ptr... 0x6b8aeaa0, 0x6b8aedc0
Caught exception in looking up space point ptr... 0x6b8aedc0, 0x6b8a9d00
Caught exception in looking up space point ptr... 0x6b8a9d00, 0x6b8a5fe0
Caught exception in looking up space point ptr... 0x59903d50, 0x59903010
Caught exception in looking up space point ptr... 0x59903010, 0x59903d50
Caught exception in looking up space point ptr... 0x4797cc30, 0x4797ced0
Caught exception in looking up space point ptr... 0x4797ced0, 0x4797cc30
Caught exception in looking up space point ptr... 0x482d6600, 0x68a5ea60
Caught exception in looking up space point ptr... 0x68a5ea60, 0x47794e30
Caught exception in looking up space point ptr... 0x762e4f10, 0x762e4de0
Caught exception in looking up space point ptr... 0x762e4de0, 0x762e4f10
Caught exception in looking up space point ptr... 0x855de280, 0x855debc0
Caught exception in looking up space point ptr... 0x6a808cc0, 0x855de280
Caught exception in looking up space point ptr... 0x3f884340, 0x3f87ca70
Caught exception in looking up space point ptr... 0x3f87ca70, 0x3f8841f0
Caught exception in looking up space point ptr... 0x59911760, 0x59911330
Caught exception in looking up space point ptr... 0x59911330, 0x59911760
Caught exception in looking up space point ptr... 0x8fc17a70, 0x7ef52c50
Caught exception in looking up space point ptr... 0x7ef52c50, 0x8fc17a70
Caught exception in looking up space point ptr... 0x855e3ac0, 0x6a7f1060
Caught exception in looking up space point ptr... 0x6a7f1060, 0x855e9240
Caught exception in looking up space point ptr... 0x74c53a20, 0x397c7b60
Caught exception in looking up space point ptr... 0x397c7b60, 0x74c53a20
Caught exception in looking up space point ptr... 0xb8dde6e0, 0xb8dde870
Caught exception in looking up space point ptr... 0xb8dde870, 0xb8ddea00
Caught exception in looking up space point ptr... 0x6b8cfef0, 0x6b8d0d30
Caught exception in looking up space point ptr... 0x6b8d0d30, 0x6b8cfef0
Caught exception in looking up space point ptr... 0xa820eec0, 0xa820f240
Caught exception in looking up space point ptr... 0xa820f240, 0xa820f160
Caught exception in looking up space point ptr... 0xa820f160, 0xa820f080
Caught exception in looking up space point ptr... 0xa820f080, 0xa820f160
Caught exception in looking up space point ptr... 0xa820f160, 0xa820efa0
Caught exception in looking up space point ptr... 0x58cd3360, 0x58cd3820
Caught exception in looking up space point ptr... 0x58cd3820, 0x58cd3360
Caught exception in looking up space point ptr... 0x762dace0, 0x762dabb0
Caught exception in looking up space point ptr... 0x762dabb0, 0x762dace0
Caught exception in looking up space point ptr... 0x40045720, 0x74382fd0
Caught exception in looking up space point ptr... 0x74382fd0, 0x40045720
Caught exception in looking up space point ptr... 0x7b103310, 0x82711400
Caught exception in looking up space point ptr... 0x82711400, 0x70bbf110
Caught exception in looking up space point ptr... 0x6451e760, 0x6451e630
Caught exception in looking up space point ptr... 0x6451e630, 0x6451e250
Caught exception in looking up space point ptr... 0x6e77a340, 0x8d3d3200
Caught exception in looking up space point ptr... 0x8d3d3200, 0x6e77a340
Caught exception in looking up space point ptr... 0x8935b180, 0x7657a890
Caught exception in looking up space point ptr... 0x7657a890, 0x8935b180
Caught exception in looking up space point ptr... 0x81d18a80, 0x81d185d0
Caught exception in looking up space point ptr... 0x81d185d0, 0x81d18760
Caught exception in looking up space point ptr... 0x81d19570, 0x81d19700
Caught exception in looking up space point ptr... 0x81d19700, 0x81d19570
Caught exception in looking up space point ptr... 0x37ef3730, 0x37ef3540
Caught exception in looking up space point ptr... 0x37ef3540, 0x37ef3730
Caught exception in looking up space point ptr... 0x9f56f1b0, 0xa9f282d0
Caught exception in looking up space point ptr... 0xa9f282d0, 0x9f56f1b0
Caught exception in looking up space point ptr... 0x44082940, 0x440827b0
Caught exception in looking up space point ptr... 0x440827b0, 0x44082940
++++>>>> total num hits: 63997, num free: 61452
04-Feb-2026 00:56:16 UTC  Closed output file "np02vd_raw_run042433_0111_df-s03-d1_dw_0_20260203T192419_reco_stage0_20260204T005616_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         127.681       143.342       162.393       141.186       10.0117        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.0482e-05    0.000128288   0.000312542   9.2219e-05    6.53164e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                14.7039        16.67        19.9429       15.9104       1.68422        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.662531      0.712002      0.805683      0.714185      0.031491        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0228635     0.0246113     0.0400729     0.0230814     0.0043398       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0228024     0.0230324     0.0233525     0.0230185    0.000137381      19     
produce:beamevent:BeamEvent                      0.000109806   0.000189355   0.00102778    0.00013877    0.000198461      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.10314       3.33546       3.57387       3.32842      0.154954        19     
produce:ophit:OpHitFinder                         0.0325174     0.040964      0.0484422     0.0409666    0.00324535       19     
produce:wclsdatavd:WireCellToolkit                 85.0848       101.842       116.519       100.18        9.04674        19     
produce:gaushit:GausHitFinder                     0.598008       1.18614       2.03889       1.15319      0.308419        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000162666   0.000302228   0.000619646   0.000294935   0.000118392      19     
produce:reco3d:SpacePointSolver                    7.2413        12.6061       23.0043       11.7176       3.46689        19     
produce:hitpdune:DisambigFromSpacePoints          0.0746834     0.168482       0.29348      0.155856       0.05194        19     
produce:cluster3d:Cluster3D                       0.866739       2.07736       5.33865       2.01793      0.897355        19     
[art]:TriggerResults:TriggerResultInserter       2.4336e-05    3.27722e-05   8.4718e-05    2.7762e-05    1.3059e-05       19     
end_path:out1:RootOutput                          3.696e-06    7.85021e-06   2.3165e-05     5.939e-06    5.15125e-06      19     
end_path:out1:RootOutput(write)                    3.72442       4.61003       5.9461        4.20396      0.791269        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5709.75 MB
  Peak resident set size usage (VmHWM): 3359.17 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0111_df-s03-d1_dw_0_20260203T192419_reco_stage0_20260204T005616_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0111_df-s03-d1_dw_0_20260203T192419_reco_stage0_20260204T005616_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0111_df-s03-d1_dw_0_20260203T192419_reco_stage0_20260204T005616_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163798.0,
    "core.end_time": 1770166578.0,
    "core.events": [
      12663,
      12669,
      12675,
      12681,
      12687,
      12693,
      12699,
      12705,
      12711,
      12717,
      12723,
      12729,
      12735,
      12741,
      12747,
      12753,
      12759,
      12765,
      12771
    ],
    "core.event_count": 19,
    "core.first_event_number": 12663,
    "core.last_event_number": 12771,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0111_df-s03-d1_dw_0_20260203T192419.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:15:13 UTC       justIN version: 01.06.00