justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.5@dunegpschedd01.fnal.gov

Jobsub ID303653.5@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-04 00:20:50
From worker nodeHostnameb9p08p6650.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:34:57
Input filesvd-protodune:np02vd_raw_run042421_0049_df-s02-d1_dw_0_20260203T135729.hdf5
JobscriptExit code0
Real time43m (2602s)
CPU time41m (2497s = 95%)
Max RSS bytes2613993472 (2492 MiB)
Outputting started2026-02-04 00:18:21
Output files
Finished2026-02-04 00:20:50
Saved logsjustin-logs:303653.5-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ace point ptr... 0x5e8ea0d0, 0x5e8ea260
Caught exception in looking up space point ptr... 0x5e8e9a90, 0x5e8ea0d0
Caught exception in looking up space point ptr... 0x856d8a0, 0x856d560
Caught exception in looking up space point ptr... 0x856d560, 0x856d8a0
Caught exception in looking up space point ptr... 0x3aa89b10, 0x346f9050
Caught exception in looking up space point ptr... 0x346f9050, 0x3aa89b10
Caught exception in looking up space point ptr... 0x138e9490, 0x138e9170
Caught exception in looking up space point ptr... 0x138e9170, 0x138e9490
Caught exception in looking up space point ptr... 0x61ffe1d0, 0x6200b1f0
Caught exception in looking up space point ptr... 0x62003390, 0x61ffe1d0
Caught exception in looking up space point ptr... 0x12ba9ed0, 0x24cafde0
Caught exception in looking up space point ptr... 0x410b4400, 0x12ba9ed0
Caught exception in looking up space point ptr... 0x36e49270, 0x37aa6930
Caught exception in looking up space point ptr... 0x37aa6930, 0x37aa6770
Caught exception in looking up space point ptr... 0x28a6f890, 0x27b1cc10
Caught exception in looking up space point ptr... 0x27b1cc10, 0x28a6f890
Caught exception in looking up space point ptr... 0x28a6f890, 0x28a76940
Caught exception in looking up space point ptr... 0x28a76940, 0x28a6f890
Caught exception in looking up space point ptr... 0x36237720, 0x12135250
Caught exception in looking up space point ptr... 0x12135250, 0x36237720
Caught exception in looking up space point ptr... 0x2cef7900, 0x2cef75c0
Caught exception in looking up space point ptr... 0x2cef75c0, 0x2cef7900
Caught exception in looking up space point ptr... 0x392bf150, 0x36e8f240
Caught exception in looking up space point ptr... 0x36e8f240, 0x10c0dc10
Caught exception in looking up space point ptr... 0x2cb203f0, 0x2cb20590
Caught exception in looking up space point ptr... 0x2cb20590, 0x2cb203f0
Caught exception in looking up space point ptr... 0x2cb203f0, 0x2cb20880
Caught exception in looking up space point ptr... 0x2cb20880, 0x2cb203f0
Caught exception in looking up space point ptr... 0x250078b0, 0x448a1660
Caught exception in looking up space point ptr... 0x28e313b0, 0x250078b0
Caught exception in looking up space point ptr... 0x489adff0, 0x3e01ddc0
Caught exception in looking up space point ptr... 0x3e01ddc0, 0x489adff0
Caught exception in looking up space point ptr... 0xe0aacd0, 0x2edb6300
Caught exception in looking up space point ptr... 0xe098110, 0xe0aacd0
Caught exception in looking up space point ptr... 0x389e8cd0, 0xfb46030
Caught exception in looking up space point ptr... 0xfb46030, 0x5fae5650
Caught exception in looking up space point ptr... 0x36184810, 0x36184530
Caught exception in looking up space point ptr... 0x36184530, 0x36184650
Caught exception in looking up space point ptr... 0x463fdd10, 0x463fdb80
Caught exception in looking up space point ptr... 0x463fdb80, 0x3667af10
Caught exception in looking up space point ptr... 0x3d292330, 0x28a7b5b0
Caught exception in looking up space point ptr... 0x28a7b5b0, 0x3d292330
Caught exception in looking up space point ptr... 0x53a4260, 0x5d470470
Caught exception in looking up space point ptr... 0x5d470470, 0x53a40c0
Caught exception in looking up space point ptr... 0x12c6c2e0, 0x118623a0
Caught exception in looking up space point ptr... 0x118623a0, 0x1186d830
Caught exception in looking up space point ptr... 0x11e9ef50, 0x780da30
Caught exception in looking up space point ptr... 0x780da30, 0x11e9ef50
Caught exception in looking up space point ptr... 0x214a2ec0, 0x214a30c0
Caught exception in looking up space point ptr... 0x214a30c0, 0x214a2ec0
Caught exception in looking up space point ptr... 0x3d28e960, 0x3d28fa90
Caught exception in looking up space point ptr... 0x3d28fa90, 0x3d28e960
Caught exception in looking up space point ptr... 0x22c29bb0, 0x27498060
Caught exception in looking up space point ptr... 0x27497dd0, 0x22c29bb0
Caught exception in looking up space point ptr... 0x25414f70, 0x46c02d70
Caught exception in looking up space point ptr... 0x46c02d70, 0x25414f70
Caught exception in looking up space point ptr... 0x3623d8d0, 0x4924dea0
Caught exception in looking up space point ptr... 0x4924dea0, 0x3623d8d0
Caught exception in looking up space point ptr... 0x13b65590, 0x13b6c290
Caught exception in looking up space point ptr... 0x13b6c290, 0x13b65590
Caught exception in looking up space point ptr... 0x4cef67d0, 0x4cef8e00
Caught exception in looking up space point ptr... 0x4cef8e00, 0x4cef67d0
Caught exception in looking up space point ptr... 0x12b76100, 0x12b75f20
Caught exception in looking up space point ptr... 0x12b75f20, 0x12b76100
Caught exception in looking up space point ptr... 0x858c810, 0x858b1d0
Caught exception in looking up space point ptr... 0x858b1d0, 0x858c810
++++>>>> total num hits: 84636, num free: 81252
04-Feb-2026 01:18:11 CET  Closed output file "np02vd_raw_run042421_0049_df-s02-d1_dw_0_20260203T135729_reco_stage0_20260204T001811_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         77.7181       93.7835       108.425       93.5072       8.25855        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.7061e-05    0.000101447   0.000176802   8.9871e-05    3.42441e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.45642       4.78476       5.34015       4.76212      0.210497        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0113656     0.0167091     0.0649519     0.0136136     0.0102653       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000467176   0.000670345   0.000930711   0.000673979   0.00012225       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0015609     0.0021264    0.00302107    0.00206492    0.00033489       27     
produce:beamevent:BeamEvent                      7.3711e-05    0.000141114   0.000920371   0.000105331   0.000156599      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.44974       2.99242       3.42501       3.01015      0.218018        27     
produce:ophit:OpHitFinder                         0.0223223     0.0393629     0.046059      0.0406256    0.00577529       27     
produce:wclsdatavd:WireCellToolkit                 55.7347       66.7065       78.246        66.8462       5.56542        27     
produce:gaushit:GausHitFinder                     0.551055       1.06123       1.59463      0.992814      0.284085        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000185082   0.000300159   0.000578157   0.000291194   7.98185e-05      27     
produce:reco3d:SpacePointSolver                    4.86373       11.9592       20.4332       11.7924       3.39024        27     
produce:hitpdune:DisambigFromSpacePoints          0.0478989     0.115415      0.264449      0.103186      0.0446094       27     
produce:cluster3d:Cluster3D                       0.766497       1.96544       5.43719       1.68163       0.93938        27     
[art]:TriggerResults:TriggerResultInserter        2.091e-05    3.45768e-05   9.9971e-05    3.2001e-05    1.4757e-05       27     
end_path:out1:RootOutput                          5.77e-06     1.08494e-05   3.6391e-05     8.951e-06    7.02058e-06      27     
end_path:out1:RootOutput(write)                    2.92829       4.11318       5.24151       4.33944      0.628858        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4563.99 MB
  Peak resident set size usage (VmHWM): 2613.99 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0049_df-s02-d1_dw_0_20260203T135729_reco_stage0_20260204T001811_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0049_df-s02-d1_dw_0_20260203T135729_reco_stage0_20260204T001811_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0049_df-s02-d1_dw_0_20260203T135729_reco_stage0_20260204T001811_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161711.0,
    "core.end_time": 1770164293.0,
    "core.events": [
      7981,
      7987,
      7993,
      7999,
      8005,
      8011,
      8017,
      8023,
      8029,
      8035,
      8041,
      8047,
      8053,
      8059,
      8065,
      8071,
      8077,
      8083,
      8089,
      8095,
      8101,
      8107,
      8113,
      8119,
      8125,
      8131,
      8137
    ],
    "core.event_count": 27,
    "core.first_event_number": 7981,
    "core.last_event_number": 8137,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0049_df-s02-d1_dw_0_20260203T135729.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:45:20 UTC       justIN version: 01.06.00