justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.60@dunegpschedd02.fnal.gov

Jobsub ID297923.60@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat2026-02-04 01:03:45
From worker nodeHostnameheplnc153.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:07
Input filesvd-protodune:np02vd_raw_run042433_0175_df-s04-d1_dw_0_20260203T194837.hdf5
JobscriptExit code0
Real time1h (3726s)
CPU time56m (3403s = 91%)
Max RSS bytes3339972608 (3185 MiB)
Outputting started2026-02-04 01:01:14
Output files
Finished2026-02-04 01:03:45
Saved logsjustin-logs:297923.60-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t ptr... 0x2c33b4e0, 0x433f2ac0
Caught exception in looking up space point ptr... 0x36bc4dd0, 0x36bc50a0
Caught exception in looking up space point ptr... 0x36bc50a0, 0x36bd1540
Caught exception in looking up space point ptr... 0x2b8cf770, 0x2b8cf900
Caught exception in looking up space point ptr... 0x2b8cf900, 0x2b8cf770
Caught exception in looking up space point ptr... 0x6cd0c1b0, 0x6cd08a10
Caught exception in looking up space point ptr... 0x6cd08a10, 0x5cd66a10
Caught exception in looking up space point ptr... 0x2b92ce30, 0x5c220230
Caught exception in looking up space point ptr... 0x5c220230, 0x2ce25e80
Caught exception in looking up space point ptr... 0x33489f20, 0x523b4650
Caught exception in looking up space point ptr... 0x523b4650, 0x13473c40
Caught exception in looking up space point ptr... 0x13473c40, 0x523b4980
Caught exception in looking up space point ptr... 0x523b4980, 0x33489f20
Caught exception in looking up space point ptr... 0x33c481e0, 0x33c480e0
Caught exception in looking up space point ptr... 0x33c480e0, 0x38cf6280
Caught exception in looking up space point ptr... 0x6f78fed0, 0x1b3b39c0
Caught exception in looking up space point ptr... 0x1b3b39c0, 0x6f78fed0
Caught exception in looking up space point ptr... 0xf94e3b0, 0xf95a8b0
Caught exception in looking up space point ptr... 0xf95a8b0, 0xf94e3b0
Caught exception in looking up space point ptr... 0x36c139d0, 0x36c12d90
Caught exception in looking up space point ptr... 0x52ceb3d0, 0x36c139d0
Caught exception in looking up space point ptr... 0x482d5c80, 0x482d71c0
Caught exception in looking up space point ptr... 0x482dbf90, 0x482d5c80
Caught exception in looking up space point ptr... 0x305152e0, 0x3051a970
Caught exception in looking up space point ptr... 0x3051a970, 0x305152e0
Caught exception in looking up space point ptr... 0x3294cb90, 0x32951e20
Caught exception in looking up space point ptr... 0x32951e20, 0x3294cb90
Caught exception in looking up space point ptr... 0x2973a040, 0x29739ee0
Caught exception in looking up space point ptr... 0x29739ee0, 0x2973a040
Caught exception in looking up space point ptr... 0x38ccce60, 0x33c49600
Caught exception in looking up space point ptr... 0x33c49600, 0x38ccef80
Caught exception in looking up space point ptr... 0x6dd1f1f0, 0x6dd17dc0
Caught exception in looking up space point ptr... 0x6dd17dc0, 0x6dd1cb30
Caught exception in looking up space point ptr... 0x1b8953f0, 0x1b89f660
Caught exception in looking up space point ptr... 0x1b89f660, 0x1b8953f0
Caught exception in looking up space point ptr... 0x2b658910, 0x2b658c30
Caught exception in looking up space point ptr... 0x2b658c30, 0x2b658910
Caught exception in looking up space point ptr... 0x2a2a6190, 0x2a2a6680
Caught exception in looking up space point ptr... 0x2a2a6680, 0x2a2a6190
Caught exception in looking up space point ptr... 0x2a2a6190, 0x2a2a5620
Caught exception in looking up space point ptr... 0x2a2a5620, 0x2a2a6190
Caught exception in looking up space point ptr... 0x371e6df0, 0x371e6c20
Caught exception in looking up space point ptr... 0x371e6c20, 0x371e6df0
Caught exception in looking up space point ptr... 0x371e6df0, 0x371e6a50
Caught exception in looking up space point ptr... 0x371e6a50, 0x371e6df0
Caught exception in looking up space point ptr... 0x28cb08e0, 0x4a13e3d0
Caught exception in looking up space point ptr... 0x4a13e3d0, 0x667161b0
Caught exception in looking up space point ptr... 0x2b3b0ef0, 0x1b0f8460
Caught exception in looking up space point ptr... 0x1b0f8460, 0x556aa7a0
Caught exception in looking up space point ptr... 0x6cfa28c0, 0x6cfa29a0
Caught exception in looking up space point ptr... 0x6cfa29a0, 0x6cfa28c0
Caught exception in looking up space point ptr... 0x525f4400, 0x525f4590
Caught exception in looking up space point ptr... 0x525f4590, 0x525f4400
Caught exception in looking up space point ptr... 0x68f4b2d0, 0x68f51880
Caught exception in looking up space point ptr... 0x68f51880, 0x68f4b2d0
Caught exception in looking up space point ptr... 0x68f4b2d0, 0x68f520d0
Caught exception in looking up space point ptr... 0x68f520d0, 0x68f4b2d0
Caught exception in looking up space point ptr... 0x47921590, 0x47926a50
Caught exception in looking up space point ptr... 0x47926a50, 0x47927870
Caught exception in looking up space point ptr... 0x15dcc380, 0x15dcc770
Caught exception in looking up space point ptr... 0x15dcc770, 0x15dcc620
Caught exception in looking up space point ptr... 0x335ffdb0, 0x5d8aeab0
Caught exception in looking up space point ptr... 0x5d8aeab0, 0x42461b90
Caught exception in looking up space point ptr... 0x8bc12f0, 0x8bc1440
Caught exception in looking up space point ptr... 0x8bc1440, 0x8bc1590
Caught exception in looking up space point ptr... 0x2be71be0, 0x2be79520
Caught exception in looking up space point ptr... 0x2be79520, 0x2be71be0
++++>>>> total num hits: 104218, num free: 99334
04-Feb-2026 01:01:05 GMT  Closed output file "np02vd_raw_run042433_0175_df-s04-d1_dw_0_20260203T194837_reco_stage0_20260204T010105_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         169.266       191.457       219.683       191.236       13.356         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000103955   0.000163714   0.000285775   0.000152426   5.15354e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                14.2194       14.485        15.484        14.4656      0.272387        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.524481      0.550881      0.578887      0.547614      0.0166239       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176634     0.0180217     0.0186573     0.0179983    0.000254044      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.017546      0.0179889     0.0185817     0.0179364    0.000257856      19     
produce:beamevent:BeamEvent                      0.000135233   0.000267484   0.00150735    0.000194836   0.000294833      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.32942       3.79228       4.38436       3.73183      0.300186        19     
produce:ophit:OpHitFinder                         0.0318957     0.0443697     0.0555801     0.0440615    0.00487793       19     
produce:wclsdatavd:WireCellToolkit                 118.096       138.589       157.225       141.643       10.633         19     
produce:gaushit:GausHitFinder                      1.00862       1.64203       2.26827       1.61581      0.363842        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000290084   0.000620589   0.00150753    0.000445825   0.000370521      19     
produce:reco3d:SpacePointSolver                    12.9353       21.7214       37.6767       20.3293       6.43042        19     
produce:hitpdune:DisambigFromSpacePoints          0.145565       0.25715      0.375605      0.299801      0.0763304       19     
produce:cluster3d:Cluster3D                        1.67025       3.57228       5.51457       3.6082        1.05346        19     
[art]:TriggerResults:TriggerResultInserter       3.0127e-05    5.25368e-05   0.00015979    4.5836e-05    2.71297e-05      19     
end_path:out1:RootOutput                          6.011e-06    1.57168e-05   4.6467e-05    1.2453e-05    9.16166e-06      19     
end_path:out1:RootOutput(write)                    4.71447       6.71402       8.7721        7.16594       1.12266        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5364.96 MB
  Peak resident set size usage (VmHWM): 3339.97 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0175_df-s04-d1_dw_0_20260203T194837_reco_stage0_20260204T010105_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0175_df-s04-d1_dw_0_20260203T194837_reco_stage0_20260204T010105_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0175_df-s04-d1_dw_0_20260203T194837_reco_stage0_20260204T010105_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163161.0,
    "core.end_time": 1770166867.0,
    "core.events": [
      19961,
      19967,
      19973,
      19979,
      19985,
      19991,
      19997,
      20003,
      20009,
      20015,
      20021,
      20027,
      20033,
      20039,
      20045,
      20051,
      20057,
      20063,
      20069
    ],
    "core.event_count": 19,
    "core.first_event_number": 19961,
    "core.last_event_number": 20069,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0175_df-s04-d1_dw_0_20260203T194837.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:13:42 UTC       justIN version: 01.06.00