justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.66@dunegpschedd01.fnal.gov

Jobsub ID303647.66@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-03 23:33:36
From worker nodeHostnameb9p08p6650.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:08
Input filesvd-protodune:np02vd_raw_run042421_0127_df-s04-d1_dw_0_20260203T143943.hdf5
JobscriptExit code0
Real time44m (2642s)
CPU time41m (2515s = 95%)
Max RSS bytes2547482624 (2429 MiB)
Outputting started2026-02-03 23:31:11
Output files
Finished2026-02-03 23:33:36
Saved logsjustin-logs:303647.66-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t exception in looking up space point ptr... 0x15b5c190, 0x49e1bcc0
Caught exception in looking up space point ptr... 0x4af4f4b0, 0x49ca3a80
Caught exception in looking up space point ptr... 0x49ca3a80, 0x49ca3c10
Caught exception in looking up space point ptr... 0x49ca3c10, 0x49ca3f30
Caught exception in looking up space point ptr... 0x49ca3f30, 0x4af4f7b0
Caught exception in looking up space point ptr... 0x5556cdd0, 0x48c59d90
Caught exception in looking up space point ptr... 0x48c59d90, 0x3b9277f0
Caught exception in looking up space point ptr... 0x61451970, 0x61450c60
Caught exception in looking up space point ptr... 0x61450c60, 0x61451970
Caught exception in looking up space point ptr... 0x52704e80, 0x52704d30
Caught exception in looking up space point ptr... 0x248a3070, 0x248a2070
Caught exception in looking up space point ptr... 0x248a2070, 0x52704e80
Caught exception in looking up space point ptr... 0x4bf81050, 0x4663cb30
Caught exception in looking up space point ptr... 0x4663cb30, 0x4bf817a0
Caught exception in looking up space point ptr... 0x4bf817a0, 0x4bf80760
Caught exception in looking up space point ptr... 0x4bf80760, 0x4bf81160
Caught exception in looking up space point ptr... 0x15b4b2e0, 0x60fe3b40
Caught exception in looking up space point ptr... 0x60fe3b40, 0x15b4b2e0
Caught exception in looking up space point ptr... 0x20fe4df0, 0x20fe4c10
Caught exception in looking up space point ptr... 0x20fe4c10, 0x20fe4df0
Caught exception in looking up space point ptr... 0x109fc210, 0x2992a530
Caught exception in looking up space point ptr... 0xb0f5910, 0x109fc210
Caught exception in looking up space point ptr... 0x25d1fa40, 0x25d1f920
Caught exception in looking up space point ptr... 0x25d1f920, 0x25d1fa40
Caught exception in looking up space point ptr... 0x25d1fa40, 0x25d1ee00
Caught exception in looking up space point ptr... 0x25d1ee00, 0x25d1fa40
Caught exception in looking up space point ptr... 0x4f2d0030, 0x11602340
Caught exception in looking up space point ptr... 0x11602340, 0x2b869bf0
Caught exception in looking up space point ptr... 0x5e887360, 0x217b4ef0
Caught exception in looking up space point ptr... 0x217b4ef0, 0x248a1030
Caught exception in looking up space point ptr... 0x248a1030, 0x53490ff0
Caught exception in looking up space point ptr... 0x53490ff0, 0x481c09d0
Caught exception in looking up space point ptr... 0x51f5b5a0, 0x2f4cf0c0
Caught exception in looking up space point ptr... 0x2f4cf0c0, 0x51f5b3c0
Caught exception in looking up space point ptr... 0x2fcaa770, 0x2fcaa940
Caught exception in looking up space point ptr... 0x2fcaa940, 0x2fcaadc0
Caught exception in looking up space point ptr... 0x2fcaadc0, 0x4767dd50
Caught exception in looking up space point ptr... 0x33f166b0, 0x33f17b60
Caught exception in looking up space point ptr... 0x33f17b60, 0x33f166b0
Caught exception in looking up space point ptr... 0x33f166b0, 0x33f17ee0
Caught exception in looking up space point ptr... 0x33f17ee0, 0x33f166b0
Caught exception in looking up space point ptr... 0x63229800, 0x611c1850
Caught exception in looking up space point ptr... 0x611c1850, 0x2b3b9b80
Caught exception in looking up space point ptr... 0x2b3b9b80, 0x2c851a80
Caught exception in looking up space point ptr... 0x527057c0, 0x52705eb0
Caught exception in looking up space point ptr... 0x52705eb0, 0x527078d0
Caught exception in looking up space point ptr... 0x527078d0, 0x52707700
Caught exception in looking up space point ptr... 0x2fa181c0, 0x4886b010
Caught exception in looking up space point ptr... 0x4886b010, 0x2fa181c0
Caught exception in looking up space point ptr... 0x5d90f750, 0x46429e90
Caught exception in looking up space point ptr... 0x46429e90, 0x5d90f750
Caught exception in looking up space point ptr... 0x3d518d00, 0x135c94d0
Caught exception in looking up space point ptr... 0x135c94d0, 0x34550000
Caught exception in looking up space point ptr... 0x34550000, 0x493c1d70
Caught exception in looking up space point ptr... 0x493c1d70, 0x24280920
Caught exception in looking up space point ptr... 0x60d990b0, 0x33794300
Caught exception in looking up space point ptr... 0x33794300, 0x60d990b0
Caught exception in looking up space point ptr... 0x15ff5fa0, 0x3a811c60
Caught exception in looking up space point ptr... 0x3a811c60, 0x15ff5fa0
Caught exception in looking up space point ptr... 0x577119e0, 0x5d5cf9b0
Caught exception in looking up space point ptr... 0x5d5cf9b0, 0x577119e0
Caught exception in looking up space point ptr... 0x25da9780, 0x57242f30
Caught exception in looking up space point ptr... 0x57242f30, 0x2c9b63c0
Caught exception in looking up space point ptr... 0x49c4fb70, 0x49c4f5f0
Caught exception in looking up space point ptr... 0x49c4f5f0, 0x49c4f850
++++>>>> total num hits: 34633, num free: 33231
04-Feb-2026 00:31:03 CET  Closed output file "np02vd_raw_run042421_0127_df-s04-d1_dw_0_20260203T143943_reco_stage0_20260203T233103_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         88.0765       95.3447       109.784       94.9284       4.59131        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.215e-05    0.000110166   0.000385794   8.7131e-05    6.2812e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                4.79631       5.25203       7.50904       5.14798      0.483815        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0108872     0.0183738     0.0651146     0.0137318     0.0129527       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000457186   0.000547408   0.000736479   0.000529876   7.28996e-05      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00170773    0.00208524    0.00384721    0.00196697    0.000442155      27     
produce:beamevent:BeamEvent                       6.909e-05    0.000141101   0.000918731   0.000104501   0.000155796      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.52914       3.00823       3.2798        3.08411      0.215346        27     
produce:ophit:OpHitFinder                         0.0271483     0.039397      0.0514773     0.0404522    0.00505493       27     
produce:wclsdatavd:WireCellToolkit                 62.9272       69.967        75.6141       69.4809       2.81341        27     
produce:gaushit:GausHitFinder                     0.551603      0.947843       1.56327      0.938317      0.239522        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000159461   0.000266665   0.000426905   0.000263413   5.76648e-05      27     
produce:reco3d:SpacePointSolver                    5.0817        9.91784       14.5243       9.49906       2.77755        27     
produce:hitpdune:DisambigFromSpacePoints          0.0509174     0.113219      0.236587      0.113981      0.0391243       27     
produce:cluster3d:Cluster3D                       0.765182       1.91251       5.92962       1.71657      0.979104        27     
[art]:TriggerResults:TriggerResultInserter        1.934e-05    3.54572e-05   8.5301e-05    3.2151e-05    1.40471e-05      27     
end_path:out1:RootOutput                          3.38e-06     8.30281e-06   2.6161e-05     6.76e-06     5.08341e-06      27     
end_path:out1:RootOutput(write)                    3.08055       4.14296       5.05667       4.22249      0.554559        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4539.46 MB
  Peak resident set size usage (VmHWM): 2547.48 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0127_df-s04-d1_dw_0_20260203T143943_reco_stage0_20260203T233103_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0127_df-s04-d1_dw_0_20260203T143943_reco_stage0_20260203T233103_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0127_df-s04-d1_dw_0_20260203T143943_reco_stage0_20260203T233103_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158839.0,
    "core.end_time": 1770161464.0,
    "core.events": [
      20645,
      20651,
      20657,
      20663,
      20669,
      20675,
      20681,
      20687,
      20693,
      20699,
      20705,
      20711,
      20717,
      20723,
      20729,
      20735,
      20741,
      20747,
      20753,
      20759,
      20765,
      20771,
      20777,
      20783,
      20789,
      20795,
      20801
    ],
    "core.event_count": 27,
    "core.first_event_number": 20645,
    "core.last_event_number": 20801,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0127_df-s04-d1_dw_0_20260203T143943.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:20 UTC       justIN version: 01.06.00