justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.137@dunegpschedd02.fnal.gov

Jobsub ID297923.137@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2026-02-04 01:13:14
From worker nodeHostnameheplnc168.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:05:38
Input filesvd-protodune:np02vd_raw_run042433_0357_df-s04-d1_dw_0_20260203T205748.hdf5
JobscriptExit code0
Real time1h (3902s)
CPU time59m (3579s = 91%)
Max RSS bytes3337535488 (3182 MiB)
Outputting started2026-02-04 01:10:41
Output files
Finished2026-02-04 01:13:14
Saved logsjustin-logs:297923.137-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0x7b05ea00, 0x4e09f970
Caught exception in looking up space point ptr... 0x4e09f970, 0x4e09f1a0
Caught exception in looking up space point ptr... 0x4e09f650, 0x4e09fe20
Caught exception in looking up space point ptr... 0x4e09fe20, 0x10f16ee0
Caught exception in looking up space point ptr... 0x3f6ddad0, 0x167a6c40
Caught exception in looking up space point ptr... 0x167a6c40, 0x3f6dbc00
Caught exception in looking up space point ptr... 0xed554a0, 0x10ee8810
Caught exception in looking up space point ptr... 0xed55310, 0xed554a0
Caught exception in looking up space point ptr... 0xf5be540, 0x32367020
Caught exception in looking up space point ptr... 0x32367020, 0xf5be540
Caught exception in looking up space point ptr... 0x6f9d0a40, 0x6c08f1a0
Caught exception in looking up space point ptr... 0x6c08f1a0, 0x6f9d0a40
Caught exception in looking up space point ptr... 0x10f0d480, 0x7b039eb0
Caught exception in looking up space point ptr... 0x7b039eb0, 0x3548fbd0
Caught exception in looking up space point ptr... 0x4d7cf50, 0x31b80ae0
Caught exception in looking up space point ptr... 0x31b80ae0, 0x4d7cf50
Caught exception in looking up space point ptr... 0x7c867070, 0x7c863290
Caught exception in looking up space point ptr... 0x7c863290, 0x7c8691c0
Caught exception in looking up space point ptr... 0x7c8691c0, 0x7c867320
Caught exception in looking up space point ptr... 0x6f9b1980, 0x6f9ae960
Caught exception in looking up space point ptr... 0x6f9ae960, 0x6f9b1980
Caught exception in looking up space point ptr... 0x31a7e690, 0x162f2300
Caught exception in looking up space point ptr... 0x162f2300, 0x31a7e690
Caught exception in looking up space point ptr... 0x43b211a0, 0x580560f0
Caught exception in looking up space point ptr... 0x580560f0, 0x58055f10
Caught exception in looking up space point ptr... 0x4e0b5640, 0x4e0b5340
Caught exception in looking up space point ptr... 0x4e0b5340, 0x4e0b5640
Caught exception in looking up space point ptr... 0xec42ae0, 0xec43120
Caught exception in looking up space point ptr... 0xec43120, 0xec432b0
Caught exception in looking up space point ptr... 0xec432b0, 0xec42c70
Caught exception in looking up space point ptr... 0xec42c70, 0xec42f90
Caught exception in looking up space point ptr... 0x11847530, 0x6cd91a00
Caught exception in looking up space point ptr... 0x6cd91a00, 0x11847530
Caught exception in looking up space point ptr... 0x15d4e700, 0x15d49ba0
Caught exception in looking up space point ptr... 0x15d49ba0, 0x15d487f0
Caught exception in looking up space point ptr... 0x19fb0bb0, 0x6f746b70
Caught exception in looking up space point ptr... 0x6f746b70, 0x19fb0bb0
Caught exception in looking up space point ptr... 0x6f992de0, 0x6f992cb0
Caught exception in looking up space point ptr... 0x6f992cb0, 0x6f992de0
Caught exception in looking up space point ptr... 0xec46760, 0xec46630
Caught exception in looking up space point ptr... 0xec46630, 0xec46890
Caught exception in looking up space point ptr... 0x6f9afcf0, 0x6f9afa90
Caught exception in looking up space point ptr... 0x6f9afa90, 0x6f9afcf0
Caught exception in looking up space point ptr... 0xf5bd100, 0xf5bf600
Caught exception in looking up space point ptr... 0xf5bf600, 0x561a0980
Caught exception in looking up space point ptr... 0x183ee030, 0x183ee3a0
Caught exception in looking up space point ptr... 0x183ee3a0, 0x183ee210
Caught exception in looking up space point ptr... 0x509d1ac0, 0x1879fb30
Caught exception in looking up space point ptr... 0x1879fb30, 0x509d1ac0
Caught exception in looking up space point ptr... 0x67d71370, 0x67d9c150
Caught exception in looking up space point ptr... 0x67d9c150, 0x67d71370
Caught exception in looking up space point ptr... 0x1aec81b0, 0x1aedc3f0
Caught exception in looking up space point ptr... 0x1aedc3f0, 0x1aec0080
Caught exception in looking up space point ptr... 0x2ee51410, 0x2ee59be0
Caught exception in looking up space point ptr... 0x2ee59be0, 0x2ee51410
Caught exception in looking up space point ptr... 0x67d9aa20, 0x67d9ab20
Caught exception in looking up space point ptr... 0x67d9ab20, 0x67d9aa20
Caught exception in looking up space point ptr... 0x5fc5dc50, 0x5fc5db50
Caught exception in looking up space point ptr... 0x5fc5db50, 0x5fc5dd50
Caught exception in looking up space point ptr... 0x5fc38320, 0x1a48bf60
Caught exception in looking up space point ptr... 0x1a48bf60, 0x5fc38320
Caught exception in looking up space point ptr... 0x52912460, 0x82c66b60
Caught exception in looking up space point ptr... 0x82c66b60, 0x529106a0
Caught exception in looking up space point ptr... 0x6329df50, 0x6329e270
Caught exception in looking up space point ptr... 0x6329e270, 0x6329df50
Caught exception in looking up space point ptr... 0x19cea900, 0x19cea100
Caught exception in looking up space point ptr... 0x19cea100, 0x19cea7c0
++++>>>> total num hits: 123108, num free: 119338
04-Feb-2026 01:10:31 GMT  Closed output file "np02vd_raw_run042433_0357_df-s04-d1_dw_0_20260203T205748_reco_stage0_20260204T011031_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         177.677       200.719       220.97        200.342       10.774         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000100279   0.000155221   0.000331703   0.000157126   5.53105e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                14.2676       14.6423       15.5469       14.5707      0.281392        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.524084      0.548882      0.613075      0.548217      0.0203694       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0175737     0.017792      0.0184594     0.0177648    0.000198037      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.017567      0.017814      0.0187133     0.0177577    0.00024797       19     
produce:beamevent:BeamEvent                      0.000100519   0.000229785   0.00161655    0.000155513   0.000328335      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.41675       3.83954       4.32644       3.84784      0.287183        19     
produce:ophit:OpHitFinder                         0.0387873     0.0487624     0.0537487     0.0506572     0.0042182       19     
produce:wclsdatavd:WireCellToolkit                 130.978       142.835       154.319       143.31        5.8659         19     
produce:gaushit:GausHitFinder                      1.18166       1.92554       2.74447       1.94995      0.402823        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000354586   0.000477265   0.000636235   0.00047353    8.58668e-05      19     
produce:reco3d:SpacePointSolver                    13.5942       25.5322       39.0144       23.3981       7.84381        19     
produce:hitpdune:DisambigFromSpacePoints          0.178097      0.289661      0.381284      0.285219      0.0529212       19     
produce:cluster3d:Cluster3D                        1.95986       3.69501       5.79582       3.68192       1.08754        19     
[art]:TriggerResults:TriggerResultInserter       2.7021e-05    4.70727e-05   0.000142237   3.9986e-05    2.57651e-05      19     
end_path:out1:RootOutput                          7.224e-06    1.37278e-05   4.8801e-05    1.1401e-05    8.94053e-06      19     
end_path:out1:RootOutput(write)                    5.57665       7.27832       8.07597       7.50462      0.709214        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5379.33 MB
  Peak resident set size usage (VmHWM): 3337.54 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0357_df-s04-d1_dw_0_20260203T205748_reco_stage0_20260204T011031_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0357_df-s04-d1_dw_0_20260203T205748_reco_stage0_20260204T011031_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0357_df-s04-d1_dw_0_20260203T205748_reco_stage0_20260204T011031_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163552.0,
    "core.end_time": 1770167433.0,
    "core.events": [
      40709,
      40715,
      40721,
      40727,
      40733,
      40739,
      40745,
      40751,
      40757,
      40763,
      40769,
      40775,
      40781,
      40787,
      40793,
      40799,
      40805,
      40811,
      40817
    ],
    "core.event_count": 19,
    "core.first_event_number": 40709,
    "core.last_event_number": 40817,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0357_df-s04-d1_dw_0_20260203T205748.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:29:52 UTC       justIN version: 01.06.00