justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.32@dunegpschedd01.fnal.gov

Jobsub ID303656.32@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 00:39:21
From worker nodeHostnameheplnc147.pp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:12:02
Input filesvd-protodune:np02vd_raw_run042421_0166_df-s03-d1_dw_0_20260203T150049.hdf5
JobscriptExit code0
Real time25m (1533s)
CPU time22m (1340s = 87%)
Max RSS bytes2663153664 (2539 MiB)
Outputting started2026-02-04 00:37:36
Output files
Finished2026-02-04 00:39:21
Saved logsjustin-logs:303656.32-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ht exception in looking up space point ptr... 0x4910bdb0, 0x2925ba20
Caught exception in looking up space point ptr... 0x492eb000, 0x4ddafbc0
Caught exception in looking up space point ptr... 0x4ddafbc0, 0x492eb270
Caught exception in looking up space point ptr... 0x492eb270, 0xa0215c0
Caught exception in looking up space point ptr... 0xa0215c0, 0x88f0f90
Caught exception in looking up space point ptr... 0x2d3ff060, 0x492eb000
Caught exception in looking up space point ptr... 0x113033a0, 0x11303530
Caught exception in looking up space point ptr... 0x11303530, 0x11309710
Caught exception in looking up space point ptr... 0x4f1a9480, 0x4f1a8200
Caught exception in looking up space point ptr... 0x4f1a8200, 0x4f1a9480
Caught exception in looking up space point ptr... 0x29257440, 0x562bf5e0
Caught exception in looking up space point ptr... 0x29249560, 0x29257440
Caught exception in looking up space point ptr... 0x4f94cd50, 0x333c8e90
Caught exception in looking up space point ptr... 0x333c8e90, 0x333c8fe0
Caught exception in looking up space point ptr... 0xf2e3da0, 0x63739fe0
Caught exception in looking up space point ptr... 0x63739fe0, 0x2b5924f0
Caught exception in looking up space point ptr... 0x2b5924f0, 0xb8278b0
Caught exception in looking up space point ptr... 0xb8278b0, 0xa04c930
Caught exception in looking up space point ptr... 0xb92ff00, 0xf2e3da0
Caught exception in looking up space point ptr... 0x49132790, 0x488ea560
Caught exception in looking up space point ptr... 0x49133480, 0x49132790
Caught exception in looking up space point ptr... 0xa257330, 0xa2570f0
Caught exception in looking up space point ptr... 0xa2570f0, 0x4f1a3fe0
Caught exception in looking up space point ptr... 0x4df72530, 0x4df52e60
Caught exception in looking up space point ptr... 0x4df52e60, 0x4df72530
Caught exception in looking up space point ptr... 0x9bf6ca0, 0x9bf6d80
Caught exception in looking up space point ptr... 0x9bf6d80, 0x9bf7c40
Caught exception in looking up space point ptr... 0x3dc275c0, 0x4f12ea70
Caught exception in looking up space point ptr... 0x4f12ea70, 0x3f89c2b0
Caught exception in looking up space point ptr... 0x8fded90, 0x488f0770
Caught exception in looking up space point ptr... 0x488f0770, 0x634ec950
Caught exception in looking up space point ptr... 0x634ec950, 0x633987e0
Caught exception in looking up space point ptr... 0xad3e920, 0xad3e7f0
Caught exception in looking up space point ptr... 0xad3e7f0, 0xad3c890
Caught exception in looking up space point ptr... 0x5628e710, 0x5628e970
Caught exception in looking up space point ptr... 0x5628e970, 0x5628e710
Caught exception in looking up space point ptr... 0x11fdece0, 0x11fdeab0
Caught exception in looking up space point ptr... 0x11fdeab0, 0x11fdece0
Caught exception in looking up space point ptr... 0x112bc460, 0xa1a1d40
Caught exception in looking up space point ptr... 0xa1a1d40, 0x375e1fc0
Caught exception in looking up space point ptr... 0x225bec70, 0x225a83f0
Caught exception in looking up space point ptr... 0x225a83f0, 0x225bec70
Caught exception in looking up space point ptr... 0x2d25d190, 0x2d25bb90
Caught exception in looking up space point ptr... 0x2d25bb90, 0x2d25b7a0
Caught exception in looking up space point ptr... 0x11fc4e90, 0x11fc4be0
Caught exception in looking up space point ptr... 0x11fc4be0, 0x11fc4e90
Caught exception in looking up space point ptr... 0x13a7c590, 0x13a87600
Caught exception in looking up space point ptr... 0x4f1b0830, 0x13a7c590
Caught exception in looking up space point ptr... 0x557d2140, 0x8fbf7d0
Caught exception in looking up space point ptr... 0x8fbf7d0, 0x8fd48c0
Caught exception in looking up space point ptr... 0x4f1a8ab0, 0x4f1a8680
Caught exception in looking up space point ptr... 0x4f1a8680, 0x4f1a8ab0
Caught exception in looking up space point ptr... 0x48b34b60, 0x57ff3c70
Caught exception in looking up space point ptr... 0x57ff3c70, 0x48b34b60
Caught exception in looking up space point ptr... 0x633a8680, 0x633a3eb0
Caught exception in looking up space point ptr... 0x633a3eb0, 0x633a8680
Caught exception in looking up space point ptr... 0x4df6fbb0, 0x4df7aa30
Caught exception in looking up space point ptr... 0x4df7aa30, 0x4df6fbb0
Caught exception in looking up space point ptr... 0x4d889c70, 0x4d889810
Caught exception in looking up space point ptr... 0x4d889810, 0x4d889950
Caught exception in looking up space point ptr... 0x3c467f50, 0x3c46a8f0
Caught exception in looking up space point ptr... 0x3c46a8f0, 0x142f3d00
Caught exception in looking up space point ptr... 0x11fdb1d0, 0x11fdb0e0
Caught exception in looking up space point ptr... 0x11fdb0e0, 0x11fdb1d0
Caught exception in looking up space point ptr... 0x59d8cb00, 0x272c0630
Caught exception in looking up space point ptr... 0x272c0630, 0x59d8cb00
Caught exception in looking up space point ptr... 0x28231420, 0x51d51860
Caught exception in looking up space point ptr... 0x51d51860, 0x2841a3f0
++++>>>> total num hits: 72886, num free: 70318
04-Feb-2026 00:37:27 GMT  Closed output file "np02vd_raw_run042421_0166_df-s03-d1_dw_0_20260203T150049_reco_stage0_20260204T003727_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         109.927       120.523       135.974       118.605       8.98525        12     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.4268e-05    0.000173344   0.000335704   0.000174063   6.83102e-05      12     
produce:tpcrawdecoder:PDVDTPCReader                13.0353       13.4478       13.9299       13.4054      0.266132        12     
produce:triggerrawdecoder:PDVDTriggerReader4      0.529398      0.573771      0.629426      0.575002      0.0284443       12     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.017829      0.0229975     0.035349      0.0182607    0.00713918       12     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0877091     0.0885741     0.0901304     0.088488     0.00063936       12     
produce:beamevent:BeamEvent                      0.000144513   0.000295153   0.00144097    0.000186532   0.000347724      12     
produce:pdvddaphne:DAPHNEReaderPDVD                3.40202       3.58092       3.75227       3.60313      0.107554        12     
produce:ophit:OpHitFinder                         0.0323018     0.0457862     0.0575292     0.0458178    0.00550622       12     
produce:wclsdatavd:WireCellToolkit                 63.9413       76.8027       86.9035       77.8832       8.91032        12     
produce:gaushit:GausHitFinder                     0.856936       1.27609       1.72015       1.30567      0.238121        12     
produce:nhitsfilter:NumberOfHitsFilter           0.000311117   0.000448056   0.000854473   0.000398307   0.000134524      12     
produce:reco3d:SpacePointSolver                    12.9667       16.6353       20.1283       17.3387       2.12781        12     
produce:hitpdune:DisambigFromSpacePoints          0.119823      0.170372      0.228661      0.168786      0.0360071       12     
produce:cluster3d:Cluster3D                        2.01114       3.10055       5.4743        2.72612       1.07696        12     
[art]:TriggerResults:TriggerResultInserter       2.7752e-05    4.29218e-05   0.000146797   3.2171e-05    3.19138e-05      12     
end_path:out1:RootOutput                          7.373e-06    1.19999e-05   3.3974e-05    1.0239e-05    6.76745e-06      12     
end_path:out1:RootOutput(write)                    3.65562       4.74961       5.58567       4.92349      0.750911        12     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4618.84 MB
  Peak resident set size usage (VmHWM): 2663.15 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0166_df-s03-d1_dw_0_20260203T150049_reco_stage0_20260204T003727_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0166_df-s03-d1_dw_0_20260203T150049_reco_stage0_20260204T003727_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0166_df-s03-d1_dw_0_20260203T150049_reco_stage0_20260204T003727_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163936.0,
    "core.end_time": 1770165448.0,
    "core.events": [
      26973,
      26979,
      26985,
      26991,
      26997,
      27003,
      27009,
      27015,
      27021,
      27027,
      27033,
      27039
    ],
    "core.event_count": 12,
    "core.first_event_number": 26973,
    "core.last_event_number": 27039,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0166_df-s03-d1_dw_0_20260203T150049.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:26:49 UTC       justIN version: 01.06.00