justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.65@dunegpschedd01.fnal.gov

Jobsub ID303656.65@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2026-02-04 01:15:45
From worker nodeHostnameheplnc172.pp.rl.ac.uk
cpuinfoAMD EPYC 9534 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:12:11
Input filesvd-protodune:np02vd_raw_run042421_0017_df-s04-d0_dw_0_20260203T134011.hdf5
JobscriptExit code0
Real time1h (3639s)
CPU time55m (3331s = 91%)
Max RSS bytes2684559360 (2560 MiB)
Outputting started2026-02-04 01:12:52
Output files
Finished2026-02-04 01:15:45
Saved logsjustin-logs:303656.65-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

... 0x5e1aae50, 0x5e1aacc0
Caught exception in looking up space point ptr... 0x5e1aacc0, 0x5e1aae50
Caught exception in looking up space point ptr... 0x10f89570, 0x10f89250
Caught exception in looking up space point ptr... 0x10f89250, 0x10f89570
Caught exception in looking up space point ptr... 0x10f89570, 0x10f80a10
Caught exception in looking up space point ptr... 0x10f80a10, 0x10f89570
Caught exception in looking up space point ptr... 0x10f89570, 0x10f893e0
Caught exception in looking up space point ptr... 0x10f893e0, 0x10f89570
Caught exception in looking up space point ptr... 0x268e2520, 0x24ec6d30
Caught exception in looking up space point ptr... 0x24ec6d30, 0x268e17d0
Caught exception in looking up space point ptr... 0x57c85f90, 0x57c85e90
Caught exception in looking up space point ptr... 0x57c85e90, 0x57c86480
Caught exception in looking up space point ptr... 0x445cae90, 0x445ccac0
Caught exception in looking up space point ptr... 0x445ccac0, 0x445cb440
Caught exception in looking up space point ptr... 0x54725df0, 0x1090b940
Caught exception in looking up space point ptr... 0x1090b940, 0x156b6430
Caught exception in looking up space point ptr... 0x36a53f90, 0x151ae3e0
Caught exception in looking up space point ptr... 0x151ae3e0, 0x2f6fa5b0
Caught exception in looking up space point ptr... 0x14df3c90, 0x14df39f0
Caught exception in looking up space point ptr... 0x14df39f0, 0x14df3780
Caught exception in looking up space point ptr... 0x510dd000, 0x510dc080
Caught exception in looking up space point ptr... 0x510dc080, 0x510dd000
Caught exception in looking up space point ptr... 0x152e6030, 0x152e63c0
Caught exception in looking up space point ptr... 0x152e63c0, 0x48a51f70
Caught exception in looking up space point ptr... 0x252285b0, 0x466e1f10
Caught exception in looking up space point ptr... 0x466e1f10, 0x466e18c0
Caught exception in looking up space point ptr... 0x9eb1ab0, 0x4a17bf20
Caught exception in looking up space point ptr... 0x4a17bf20, 0x9eb1ab0
Caught exception in looking up space point ptr... 0x5e4cdb30, 0x294a6500
Caught exception in looking up space point ptr... 0x294a6500, 0x5e4cdb30
Caught exception in looking up space point ptr... 0x5e1ab7b0, 0x5e1ab490
Caught exception in looking up space point ptr... 0x5e1ab490, 0x5e1ab7b0
Caught exception in looking up space point ptr... 0x5e1ab7b0, 0x5e1ab620
Caught exception in looking up space point ptr... 0x5e1ab620, 0x5e1ab7b0
Caught exception in looking up space point ptr... 0x58a3a370, 0x21885090
Caught exception in looking up space point ptr... 0x21885090, 0x58a3a0f0
Caught exception in looking up space point ptr... 0x246a4e40, 0x246a4fd0
Caught exception in looking up space point ptr... 0x5e53dee0, 0x246a4e40
Caught exception in looking up space point ptr... 0x27c0b4b0, 0x4f52d760
Caught exception in looking up space point ptr... 0x4f52d760, 0x27c0b4b0
Caught exception in looking up space point ptr... 0xa0aa950, 0x47a8a8e0
Caught exception in looking up space point ptr... 0x47a8a8e0, 0xa0aaa50
Caught exception in looking up space point ptr... 0x2445acf0, 0x3c3b1f70
Caught exception in looking up space point ptr... 0x3c3b1f70, 0x24450a40
Caught exception in looking up space point ptr... 0x55258e30, 0x31c00bf0
Caught exception in looking up space point ptr... 0x31c00bf0, 0x55258e30
Caught exception in looking up space point ptr... 0x256526e0, 0x4e48f5b0
Caught exception in looking up space point ptr... 0x4e48f5b0, 0x4e471710
Caught exception in looking up space point ptr... 0x21703330, 0x217037c0
Caught exception in looking up space point ptr... 0x217037c0, 0x21703330
Caught exception in looking up space point ptr... 0x3f5ef1c0, 0x3f5ef030
Caught exception in looking up space point ptr... 0x3f5ef030, 0x3f5ef1c0
Caught exception in looking up space point ptr... 0x20619900, 0x20619660
Caught exception in looking up space point ptr... 0x20619660, 0x20619900
Caught exception in looking up space point ptr... 0x20619900, 0x20619510
Caught exception in looking up space point ptr... 0x20619510, 0x20619900
Caught exception in looking up space point ptr... 0x46ad31d0, 0x14c0a7c0
Caught exception in looking up space point ptr... 0x14c0a7c0, 0x46ad31d0
Caught exception in looking up space point ptr... 0x487629b0, 0x145ceaf0
Caught exception in looking up space point ptr... 0x145d66c0, 0x487629b0
Caught exception in looking up space point ptr... 0x23abca40, 0x23abc590
Caught exception in looking up space point ptr... 0x23abc590, 0x23abc720
Caught exception in looking up space point ptr... 0x4a2d3720, 0x4a2d3a20
Caught exception in looking up space point ptr... 0x4a2d3a20, 0x4a2d3720
Caught exception in looking up space point ptr... 0x464bceb0, 0x32e03b00
Caught exception in looking up space point ptr... 0x32e03b00, 0x57c0b6a0
++++>>>> total num hits: 88506, num free: 85030
04-Feb-2026 01:12:43 GMT  Closed output file "np02vd_raw_run042421_0017_df-s04-d0_dw_0_20260203T134011_reco_stage0_20260204T011243_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         96.8401       131.893       160.335       139.165       18.4823        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.7821e-05    0.00013066    0.000234393   0.000118781   3.9914e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                10.4934       10.8246       11.1504       10.8221      0.197956        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.522037      0.541195      0.569565      0.543751      0.0104903       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.017555      0.018369      0.0311795     0.0178831    0.00251713       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0869539     0.0907623     0.106472      0.0881256    0.00648798       27     
produce:beamevent:BeamEvent                      0.000120802   0.00022536     0.0015513    0.000175122   0.000261379      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.88439       3.27692       3.57795       3.26487      0.149028        27     
produce:ophit:OpHitFinder                         0.0342327     0.0409739     0.0460615     0.041697     0.00308347       27     
produce:wclsdatavd:WireCellToolkit                 56.117        72.857        80.5187       75.3743       6.29557        27     
produce:gaushit:GausHitFinder                     0.769096       1.39275       2.01635       1.44143      0.305734        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000170171   0.000299297   0.00083026    0.000298523   0.000121597      27     
produce:reco3d:SpacePointSolver                    4.32341       7.4607        11.6952       6.7475        2.19877        27     
produce:hitpdune:DisambigFromSpacePoints          0.0718287     0.200859      0.327123      0.199458      0.061862        27     
produce:cluster3d:Cluster3D                        1.81832       30.5276       50.0808       36.5374       17.3543        27     
[art]:TriggerResults:TriggerResultInserter        3.205e-05    4.78738e-05   0.000149202    4.232e-05    2.13634e-05      27     
end_path:out1:RootOutput                          8.47e-06     1.36309e-05    3.755e-05     1.239e-05    5.55067e-06      27     
end_path:out1:RootOutput(write)                    3.22997       4.62475       5.89154       4.38027      0.760138        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4688.02 MB
  Peak resident set size usage (VmHWM): 2684.56 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0017_df-s04-d0_dw_0_20260203T134011_reco_stage0_20260204T011243_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0017_df-s04-d0_dw_0_20260203T134011_reco_stage0_20260204T011243_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0017_df-s04-d0_dw_0_20260203T134011_reco_stage0_20260204T011243_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163943.0,
    "core.end_time": 1770167565.0,
    "core.events": [
      2788,
      2794,
      2800,
      2806,
      2812,
      2818,
      2824,
      2830,
      2836,
      2842,
      2848,
      2854,
      2860,
      2866,
      2872,
      2878,
      2884,
      2890,
      2896,
      2902,
      2908,
      2914,
      2920,
      2926,
      2932,
      2938,
      2944
    ],
    "core.event_count": 27,
    "core.first_event_number": 2788,
    "core.last_event_number": 2944,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0017_df-s04-d0_dw_0_20260203T134011.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:02 UTC       justIN version: 01.06.00