justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.25@dunegpschedd02.fnal.gov

Jobsub ID297921.25@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 00:49:38
From worker nodeHostnameheplnc171.pp.rl.ac.uk
cpuinfoAMD EPYC 9534 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:45:20
Input filesvd-protodune:np02vd_raw_run042433_0130_df-s04-d0_dw_0_20260203T193133.hdf5
JobscriptExit code0
Real time1h (3702s)
CPU time57m (3433s = 92%)
Max RSS bytes3581407232 (3415 MiB)
Outputting started2026-02-04 00:47:04
Output files
Finished2026-02-04 00:49:38
Saved logsjustin-logs:297921.25-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oint ptr... 0x3db960d0, 0x3db96260
Caught exception in looking up space point ptr... 0x4f647cc0, 0x3db960d0
Caught exception in looking up space point ptr... 0x36f07c50, 0x36f41ae0
Caught exception in looking up space point ptr... 0x36f41ae0, 0x36ef00d0
Caught exception in looking up space point ptr... 0x67508070, 0x674e2270
Caught exception in looking up space point ptr... 0x674e2270, 0x674e2570
Caught exception in looking up space point ptr... 0x9f0c610, 0x9f0c380
Caught exception in looking up space point ptr... 0x9f0cec0, 0x9f0c610
Caught exception in looking up space point ptr... 0x3a612a50, 0x3a60d7f0
Caught exception in looking up space point ptr... 0x3a60dac0, 0x3a612a50
Caught exception in looking up space point ptr... 0x1987d490, 0x1986f840
Caught exception in looking up space point ptr... 0x1986f840, 0x67c8a8b0
Caught exception in looking up space point ptr... 0x6eaa4750, 0x6eaa4ac0
Caught exception in looking up space point ptr... 0x6eaa4ac0, 0x6eaa4750
Caught exception in looking up space point ptr... 0x313b56f0, 0x4338f8a0
Caught exception in looking up space point ptr... 0x4338f8a0, 0x14ddb5b0
Caught exception in looking up space point ptr... 0x18ba71a0, 0x3a4b49e0
Caught exception in looking up space point ptr... 0x18bb7cd0, 0x18ba71a0
Caught exception in looking up space point ptr... 0x16cc4d40, 0x6b9dc730
Caught exception in looking up space point ptr... 0x16cc4ed0, 0x16cc4d40
Caught exception in looking up space point ptr... 0x8457b570, 0x8d4a7c0
Caught exception in looking up space point ptr... 0x8d4a7c0, 0x8d4a950
Caught exception in looking up space point ptr... 0x16cd3160, 0x16c8ffc0
Caught exception in looking up space point ptr... 0x16cd32f0, 0x16cd3160
Caught exception in looking up space point ptr... 0xe8fff90, 0x845de950
Caught exception in looking up space point ptr... 0xe92e250, 0x6ec18880
Caught exception in looking up space point ptr... 0x6ec18880, 0xe8fff90
Caught exception in looking up space point ptr... 0xfaca300, 0x14c7a060
Caught exception in looking up space point ptr... 0x18d085f0, 0xfaca300
Caught exception in looking up space point ptr... 0x19b75c80, 0x65827f90
Caught exception in looking up space point ptr... 0x7a4c30f0, 0x19b75c80
Caught exception in looking up space point ptr... 0x67724b70, 0x71f0e8e0
Caught exception in looking up space point ptr... 0x71f0e8e0, 0x71f20ea0
Caught exception in looking up space point ptr... 0x4910cc90, 0x4910cb80
Caught exception in looking up space point ptr... 0x4910cb80, 0x4910c600
Caught exception in looking up space point ptr... 0x67b16cf0, 0x67b279b0
Caught exception in looking up space point ptr... 0x67b279b0, 0x67b38fd0
Caught exception in looking up space point ptr... 0x183c60f0, 0x183b3fe0
Caught exception in looking up space point ptr... 0x183b3fe0, 0x183c5f60
Caught exception in looking up space point ptr... 0x4829afe0, 0x3a0a2570
Caught exception in looking up space point ptr... 0x3a0a2570, 0x383c1580
Caught exception in looking up space point ptr... 0x18d97fb0, 0x18d856d0
Caught exception in looking up space point ptr... 0x18d856d0, 0x18d73430
Caught exception in looking up space point ptr... 0x11ad6900, 0x4b8d3b00
Caught exception in looking up space point ptr... 0x4b8d3b00, 0x39d7d4f0
Caught exception in looking up space point ptr... 0x700f97d0, 0x4fc6b3e0
Caught exception in looking up space point ptr... 0x4fc6b3e0, 0x700f97d0
Caught exception in looking up space point ptr... 0x84505ae0, 0x84505cc0
Caught exception in looking up space point ptr... 0x84505cc0, 0x84505ae0
Caught exception in looking up space point ptr... 0x485d2300, 0x70330780
Caught exception in looking up space point ptr... 0x70330780, 0x70331190
Caught exception in looking up space point ptr... 0x14be49c0, 0x46da70d0
Caught exception in looking up space point ptr... 0x46da70d0, 0x5c2e0160
Caught exception in looking up space point ptr... 0x845045e0, 0x84504f50
Caught exception in looking up space point ptr... 0x84504f50, 0x845045e0
Caught exception in looking up space point ptr... 0x845045e0, 0x84504e30
Caught exception in looking up space point ptr... 0x84504e30, 0x845045e0
Caught exception in looking up space point ptr... 0x4ff5b830, 0x4ff5b6a0
Caught exception in looking up space point ptr... 0x4ff5b6a0, 0x4ff5b830
Caught exception in looking up space point ptr... 0x354e3e20, 0x354e3fb0
Caught exception in looking up space point ptr... 0x354e3fb0, 0x354e3e20
Caught exception in looking up space point ptr... 0x42ff4a30, 0x3b932f10
Caught exception in looking up space point ptr... 0x3b932f10, 0x42ff4a30
Caught exception in looking up space point ptr... 0x5eaf1200, 0x6224ca10
Caught exception in looking up space point ptr... 0x6224ca10, 0x6224cc70
Caught exception in looking up space point ptr... 0x6eaa79e0, 0x6eaa7850
Caught exception in looking up space point ptr... 0x6eaa7850, 0x45a63220
++++>>>> total num hits: 499434, num free: 489915
04-Feb-2026 00:46:55 GMT  Closed output file "np02vd_raw_run042433_0130_df-s04-d0_dw_0_20260203T193133_reco_stage0_20260204T004655_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         144.577       190.772       236.855       190.792       30.2376        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.1031e-05    0.000145793   0.000308184   0.000110142   6.19509e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                13.5735       14.513        19.6972       14.3133       1.26953        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.529063      0.551291      0.620561      0.544901      0.0231072       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0177924     0.0211991     0.0439845     0.018172      0.0067933       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0178066     0.018036      0.0183219     0.0180026    0.00015139       19     
produce:beamevent:BeamEvent                      0.000119041   0.000244485   0.00145232    0.000175362   0.000286584      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.93915       3.26167        3.485        3.2616        0.14171        19     
produce:ophit:OpHitFinder                         0.0344143     0.0416339     0.0471206     0.0416432    0.00359573       19     
produce:wclsdatavd:WireCellToolkit                 103.823       117.683       131.628       118.361       7.30488        19     
produce:gaushit:GausHitFinder                     0.847086       1.91854        2.985        1.65428      0.648707        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000211522   0.000335489   0.00077905    0.000338425   0.000118287      19     
produce:reco3d:SpacePointSolver                    7.85905       12.5854       16.3952       12.6919       2.09284        19     
produce:hitpdune:DisambigFromSpacePoints          0.130634      0.266761      0.404562      0.256268      0.0869015       19     
produce:cluster3d:Cluster3D                        1.58806       33.5062       66.4489       46.8461       27.0623        19     
[art]:TriggerResults:TriggerResultInserter        3.114e-05    5.20902e-05   0.000156332   4.8731e-05    2.57669e-05      19     
end_path:out1:RootOutput                          7.82e-06     1.25354e-05    4.262e-05     1.127e-05    7.20086e-06      19     
end_path:out1:RootOutput(write)                    4.22359       6.35153       8.04144       6.08736       1.18925        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5537.51 MB
  Peak resident set size usage (VmHWM): 3581.41 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0130_df-s04-d0_dw_0_20260203T193133_reco_stage0_20260204T004655_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0130_df-s04-d0_dw_0_20260203T193133_reco_stage0_20260204T004655_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0130_df-s04-d0_dw_0_20260203T193133_reco_stage0_20260204T004655_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162332.0,
    "core.end_time": 1770166017.0,
    "core.events": [
      14830,
      14836,
      14842,
      14848,
      14854,
      14860,
      14866,
      14872,
      14878,
      14884,
      14890,
      14896,
      14902,
      14908,
      14914,
      14920,
      14926,
      14932,
      14938
    ],
    "core.event_count": 19,
    "core.first_event_number": 14830,
    "core.last_event_number": 14938,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0130_df-s04-d0_dw_0_20260203T193133.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:21 UTC       justIN version: 01.06.00