justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.19@dunegpschedd01.fnal.gov

Jobsub ID303656.19@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 01:28:32
From worker nodeHostnameheplnc147.pp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:11:42
Input filesvd-protodune:np02vd_raw_run042421_0009_df-s02-d1_dw_0_20260203T133547.hdf5
JobscriptExit code0
Real time1h (4440s)
CPU time1h (3943s = 88%)
Max RSS bytes10447929344 (9963 MiB)
Outputting started2026-02-04 01:25:44
Output files
Finished2026-02-04 01:28:32
Saved logsjustin-logs:303656.19-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t ptr... 0x2ed1d1f0, 0x2144ae80
Caught exception in looking up space point ptr... 0x9b46e2f0, 0x9b46e060
Caught exception in looking up space point ptr... 0x9b46e060, 0x9b46e580
Caught exception in looking up space point ptr... 0x23e7c4e0, 0x23e7ca00
Caught exception in looking up space point ptr... 0x23e7c770, 0x23e7c4e0
Caught exception in looking up space point ptr... 0x7bfe3030, 0x73c03170
Caught exception in looking up space point ptr... 0x73c03170, 0x738cf470
Caught exception in looking up space point ptr... 0x96d88180, 0x96d15730
Caught exception in looking up space point ptr... 0x96d8be00, 0x96d88180
Caught exception in looking up space point ptr... 0x331f55b0, 0x23dd39a0
Caught exception in looking up space point ptr... 0x23dd39a0, 0x15df8030
Caught exception in looking up space point ptr... 0x2652d6a0, 0x37af6f50
Caught exception in looking up space point ptr... 0xd9cbaa0, 0x2652d6a0
Caught exception in looking up space point ptr... 0x774d8390, 0x774444f0
Caught exception in looking up space point ptr... 0x773bb3a0, 0x774d8390
Caught exception in looking up space point ptr... 0x8bf8a420, 0x837670c0
Caught exception in looking up space point ptr... 0x8bfb2230, 0x8bfb1d50
Caught exception in looking up space point ptr... 0x8bfb1d50, 0x8bf8a420
Caught exception in looking up space point ptr... 0x4c218e70, 0x28c0cfc0
Caught exception in looking up space point ptr... 0x522bc7e0, 0x4c218e70
Caught exception in looking up space point ptr... 0x6c6d56b0, 0x6c6a94c0
Caught exception in looking up space point ptr... 0x6c6a94c0, 0x6c67c190
Caught exception in looking up space point ptr... 0x38c86f50, 0x23b04730
Caught exception in looking up space point ptr... 0x23b04730, 0x420fc010
Caught exception in looking up space point ptr... 0x32edbe50, 0xe620a80
Caught exception in looking up space point ptr... 0xe620a80, 0x168f4670
Caught exception in looking up space point ptr... 0x6ae1a580, 0x26af7080
Caught exception in looking up space point ptr... 0x88938ed0, 0x6ae1a580
Caught exception in looking up space point ptr... 0x50644ea0, 0x4c2165f0
Caught exception in looking up space point ptr... 0x4c2165f0, 0x11c5a810
Caught exception in looking up space point ptr... 0x7c84fc80, 0x7c20dcd0
Caught exception in looking up space point ptr... 0x7c20dcd0, 0x7c84fc80
Caught exception in looking up space point ptr... 0x28f3a0c0, 0xa9101b0
Caught exception in looking up space point ptr... 0xa9101b0, 0x4be4e990
Caught exception in looking up space point ptr... 0x5a6fde40, 0x694b5250
Caught exception in looking up space point ptr... 0x129323a0, 0x5a6fde40
Caught exception in looking up space point ptr... 0x591c6660, 0x14386a30
Caught exception in looking up space point ptr... 0x14386a30, 0x69efc510
Caught exception in looking up space point ptr... 0x21904fc0, 0x21905150
Caught exception in looking up space point ptr... 0x21905150, 0x21904e30
Caught exception in looking up space point ptr... 0x6d9eaf80, 0x3a30d310
Caught exception in looking up space point ptr... 0x3a30d310, 0x496d5b70
Caught exception in looking up space point ptr... 0x8c0af230, 0x8c0f9db0
Caught exception in looking up space point ptr... 0x8c0f9db0, 0x77c9b030
Caught exception in looking up space point ptr... 0x9e0a2560, 0x9e0a23d0
Caught exception in looking up space point ptr... 0x9e0a23d0, 0x9e0a2560
Caught exception in looking up space point ptr... 0x944ca4b0, 0x97bc69f0
Caught exception in looking up space point ptr... 0x97bc69f0, 0x944ca4b0
Caught exception in looking up space point ptr... 0x69a08060, 0x2a4e9140
Caught exception in looking up space point ptr... 0x2a4e9140, 0x69a08060
Caught exception in looking up space point ptr... 0x69a08060, 0x69899c60
Caught exception in looking up space point ptr... 0x69899c60, 0x69a08060
Caught exception in looking up space point ptr... 0x97f4a850, 0x9a134370
Caught exception in looking up space point ptr... 0x9a134370, 0x97f4a850
Caught exception in looking up space point ptr... 0x7e09ac90, 0x7df82840
Caught exception in looking up space point ptr... 0x7df82840, 0x7e09ac90
Caught exception in looking up space point ptr... 0x128ba470, 0x2146b5f0
Caught exception in looking up space point ptr... 0x2146b5f0, 0x128ba470
Caught exception in looking up space point ptr... 0xed9fa50, 0xed9f900
Caught exception in looking up space point ptr... 0xed9f900, 0xed9fa50
Caught exception in looking up space point ptr... 0xed9ff90, 0xed9fe40
Caught exception in looking up space point ptr... 0xed9fe40, 0xed9ff90
Caught exception in looking up space point ptr... 0x128ca420, 0x214834c0
Caught exception in looking up space point ptr... 0x214834c0, 0x128ca420
Caught exception in looking up space point ptr... 0x4d610d40, 0x4d6109d0
Caught exception in looking up space point ptr... 0x4d6109d0, 0x4d6107f0
++++>>>> total num hits: 4856645, num free: 4696021
04-Feb-2026 01:25:32 GMT  Closed output file "np02vd_raw_run042421_0009_df-s02-d1_dw_0_20260203T133547_reco_stage0_20260204T012532_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         97.5432       160.571       1131.13       115.954       194.53         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.2666e-05    0.000135562   0.000411887   0.000106381   6.7435e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                12.7496       15.0668       20.327        15.1697       1.41253        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.526507      0.548562      0.587589      0.548331      0.0142228       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.017716      0.0180189     0.0183773     0.0180178    0.000136648      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0872172     0.090534      0.106321      0.0887937    0.00544151       27     
produce:beamevent:BeamEvent                      9.6352e-05    0.000222682   0.00178076    0.000155364   0.000310784      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.54009       3.87144       4.09817       3.87323       0.12834        27     
produce:ophit:OpHitFinder                         0.0346867     0.0448836     0.0643852     0.0448919    0.00556498       27     
produce:wclsdatavd:WireCellToolkit                 59.6347       75.3837       93.3044       74.6875        8.486         27     
produce:gaushit:GausHitFinder                     0.714283       1.35732       5.37735       1.09875       1.00761        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000222671   0.000445795   0.00156179    0.000353487   0.000256175      27     
produce:reco3d:SpacePointSolver                    6.46553       18.7306       144.559       12.9013       25.4387        27     
produce:hitpdune:DisambigFromSpacePoints          0.0838051     0.504829       7.0736       0.117876       1.41332        27     
produce:cluster3d:Cluster3D                        1.20399       39.6772       841.395       2.03735       160.362        27     
[art]:TriggerResults:TriggerResultInserter       2.1881e-05    3.40615e-05   0.000109918   3.0698e-05    1.67069e-05      27     
end_path:out1:RootOutput                          4.237e-06    1.08393e-05   3.6238e-05     9.728e-06    5.82912e-06      27     
end_path:out1:RootOutput(write)                    3.43831       5.24282       20.343        4.76457       3.19007        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12702.3 MB
  Peak resident set size usage (VmHWM): 10447.9 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0009_df-s02-d1_dw_0_20260203T133547_reco_stage0_20260204T012532_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0009_df-s02-d1_dw_0_20260203T133547_reco_stage0_20260204T012532_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0009_df-s02-d1_dw_0_20260203T133547_reco_stage0_20260204T012532_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163915.0,
    "core.end_time": 1770168335.0,
    "core.events": [
      1471,
      1477,
      1483,
      1489,
      1495,
      1501,
      1507,
      1513,
      1519,
      1525,
      1531,
      1537,
      1543,
      1549,
      1555,
      1561,
      1567,
      1573,
      1579,
      1585,
      1591,
      1597,
      1603,
      1609,
      1615,
      1621,
      1627
    ],
    "core.event_count": 27,
    "core.first_event_number": 1471,
    "core.last_event_number": 1627,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0009_df-s02-d1_dw_0_20260203T133547.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:37 UTC       justIN version: 01.06.00