justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.40@dunegpschedd02.fnal.gov

Jobsub ID297923.40@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat2026-02-04 01:31:46
From worker nodeHostnameheplnc155.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:58:54
Input filesvd-protodune:np02vd_raw_run042433_0013_df-s02-d1_dw_0_20260203T184702.hdf5
JobscriptExit code0
Real time1h (5377s)
CPU time1h (4935s = 91%)
Max RSS bytes3576729600 (3411 MiB)
Outputting started2026-02-04 01:28:33
Output files
Finished2026-02-04 01:31:46
Saved logsjustin-logs:297923.40-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

xception in looking up space point ptr... 0x341bb9c0, 0x341bb660
Caught exception in looking up space point ptr... 0xb09a6c0, 0x538d09e0
Caught exception in looking up space point ptr... 0x538d09e0, 0x3a5918c0
Caught exception in looking up space point ptr... 0x7d851cf0, 0x60b94790
Caught exception in looking up space point ptr... 0x7d852010, 0x7d851cf0
Caught exception in looking up space point ptr... 0x3442fd10, 0x478e9370
Caught exception in looking up space point ptr... 0x478e9370, 0x34434270
Caught exception in looking up space point ptr... 0x876b6e10, 0x876b6bc0
Caught exception in looking up space point ptr... 0x876b6bc0, 0x876be1b0
Caught exception in looking up space point ptr... 0x6b8d8350, 0x6b8d8100
Caught exception in looking up space point ptr... 0x6b8d8100, 0x6b8dcb00
Caught exception in looking up space point ptr... 0x326a2e20, 0x2d7c2000
Caught exception in looking up space point ptr... 0x2d7c2000, 0x326a4c30
Caught exception in looking up space point ptr... 0x3444ead0, 0x478a4590
Caught exception in looking up space point ptr... 0x478a4590, 0x478e2a40
Caught exception in looking up space point ptr... 0xefe1e30, 0x1a857610
Caught exception in looking up space point ptr... 0x1a857610, 0x53832df0
Caught exception in looking up space point ptr... 0x708103e0, 0x34367d40
Caught exception in looking up space point ptr... 0x34367d40, 0x708103e0
Caught exception in looking up space point ptr... 0x584cc160, 0x584bc2b0
Caught exception in looking up space point ptr... 0x584cc2f0, 0x584cc160
Caught exception in looking up space point ptr... 0x3073c420, 0x7520e5f0
Caught exception in looking up space point ptr... 0x3073c5b0, 0x3073c420
Caught exception in looking up space point ptr... 0x5845ffb0, 0x1afa6a40
Caught exception in looking up space point ptr... 0x5846d110, 0x5845ffb0
Caught exception in looking up space point ptr... 0xec09ac0, 0xec1dbd0
Caught exception in looking up space point ptr... 0xec1dbd0, 0xec1ded0
Caught exception in looking up space point ptr... 0x17d7ed90, 0x17d7ec60
Caught exception in looking up space point ptr... 0x17d7ec60, 0x17d7ed90
Caught exception in looking up space point ptr... 0x60d9fbd0, 0x56634310
Caught exception in looking up space point ptr... 0x56634310, 0x537b4740
Caught exception in looking up space point ptr... 0x12a22110, 0x307b9640
Caught exception in looking up space point ptr... 0x307b9640, 0x307bbe70
Caught exception in looking up space point ptr... 0x412ffa20, 0x1b0cea80
Caught exception in looking up space point ptr... 0x1b0cea80, 0x1b0ce8f0
Caught exception in looking up space point ptr... 0xbcc7aa0, 0xbcd9ed0
Caught exception in looking up space point ptr... 0xbcb5800, 0xbcc7aa0
Caught exception in looking up space point ptr... 0xbce63b0, 0xbcd3df0
Caught exception in looking up space point ptr... 0xbcf8650, 0xbce63b0
Caught exception in looking up space point ptr... 0x1771a440, 0x68ff6770
Caught exception in looking up space point ptr... 0xf0bda70, 0x1771a440
Caught exception in looking up space point ptr... 0x14cf3710, 0x14cf38a0
Caught exception in looking up space point ptr... 0x3a004bd0, 0x14cf3710
Caught exception in looking up space point ptr... 0xa6826f0, 0x368ec880
Caught exception in looking up space point ptr... 0xa682840, 0xa6826f0
Caught exception in looking up space point ptr... 0x423a5380, 0x18ac7760
Caught exception in looking up space point ptr... 0x423a5510, 0x423a5380
Caught exception in looking up space point ptr... 0x19ec7760, 0x18ad5e10
Caught exception in looking up space point ptr... 0x19ec78f0, 0x19ec7760
Caught exception in looking up space point ptr... 0xa786360, 0x5afa1ef0
Caught exception in looking up space point ptr... 0x5afa1ef0, 0x5afa3d00
Caught exception in looking up space point ptr... 0x3de02fc0, 0x3ce99340
Caught exception in looking up space point ptr... 0x3ce99340, 0x3de03620
Caught exception in looking up space point ptr... 0xf0d4f60, 0x17748810
Caught exception in looking up space point ptr... 0x17748810, 0x6e071700
Caught exception in looking up space point ptr... 0x4d6e2760, 0x80fadf90
Caught exception in looking up space point ptr... 0x80fadf90, 0x68ef5ba0
Caught exception in looking up space point ptr... 0x74d2c730, 0x1ace0200
Caught exception in looking up space point ptr... 0x1ace0200, 0xf330800
Caught exception in looking up space point ptr... 0x16d471f0, 0x42cfcc00
Caught exception in looking up space point ptr... 0x42cfcc00, 0x16d471f0
Caught exception in looking up space point ptr... 0x538d80b0, 0x538d8240
Caught exception in looking up space point ptr... 0x538d8240, 0x538d80b0
Caught exception in looking up space point ptr... 0x72add720, 0x72addcc0
Caught exception in looking up space point ptr... 0x72addcc0, 0x72add720
Caught exception in looking up space point ptr... 0x13f25760, 0x37cbcf20
Caught exception in looking up space point ptr... 0x37cbcf20, 0x13f25760
++++>>>> total num hits: 533294, num free: 524597
04-Feb-2026 01:28:22 GMT  Closed output file "np02vd_raw_run042433_0013_df-s02-d1_dw_0_20260203T184702_reco_stage0_20260204T012822_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         209.356       277.823       305.582       286.751       26.2447        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000104005   0.000202548   0.000317234   0.000194524   7.20247e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                15.4143       16.7879       20.4552       16.6437       0.98601        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.57162      0.611632      0.733546      0.604124      0.0372893       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0178311     0.0238402     0.066076      0.0183838     0.0120068       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0179029     0.0182779     0.0188122     0.0183277    0.000270315      19     
produce:beamevent:BeamEvent                      0.000135864   0.000248498   0.00120339    0.000195466   0.000229382      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.48871       3.89377       4.24624       3.92139      0.257858        19     
produce:ophit:OpHitFinder                         0.0405302     0.0458971     0.0527489     0.0452649    0.00347383       19     
produce:wclsdatavd:WireCellToolkit                 133.09        151.273       156.774       152.523       5.33254        19     
produce:gaushit:GausHitFinder                      1.82622       3.20078        4.373        3.30115      0.627036        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00043803    0.000633873   0.00133744    0.000581197   0.000181817      19     
produce:reco3d:SpacePointSolver                    19.749        28.7223       43.8853       28.1701       6.25778        19     
produce:hitpdune:DisambigFromSpacePoints          0.224098       0.47122      0.715124      0.448219      0.114837        19     
produce:cluster3d:Cluster3D                        3.18413       64.3042       85.1965       74.0353       26.2597        19     
[art]:TriggerResults:TriggerResultInserter       2.9716e-05    5.53504e-05   0.00011194    5.0364e-05    1.81929e-05      19     
end_path:out1:RootOutput                          8.005e-06    1.31277e-05   3.4134e-05    1.1482e-05    6.96477e-06      19     
end_path:out1:RootOutput(write)                    6.62369       8.40919       10.3061       8.06442      0.965914        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5569.42 MB
  Peak resident set size usage (VmHWM): 3576.73 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0013_df-s02-d1_dw_0_20260203T184702_reco_stage0_20260204T012822_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0013_df-s02-d1_dw_0_20260203T184702_reco_stage0_20260204T012822_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0013_df-s02-d1_dw_0_20260203T184702_reco_stage0_20260204T012822_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163150.0,
    "core.end_time": 1770168504.0,
    "core.events": [
      1489,
      1495,
      1501,
      1507,
      1513,
      1519,
      1525,
      1531,
      1537,
      1543,
      1549,
      1555,
      1561,
      1567,
      1573,
      1579,
      1585,
      1591,
      1597
    ],
    "core.event_count": 19,
    "core.first_event_number": 1489,
    "core.last_event_number": 1597,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0013_df-s02-d1_dw_0_20260203T184702.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:42 UTC       justIN version: 01.06.00