justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297919.3@dunegpschedd02.fnal.gov

Jobsub ID297919.3@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:25
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce503
Last heartbeat2026-02-04 00:19:01
From worker nodeHostnameb9p05p0564.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:28:37
Input filesvd-protodune:np02vd_raw_run042421_0066_df-s03-d1_dw_0_20260203T140644.hdf5
JobscriptExit code0
Real time47m (2853s)
CPU time45m (2751s = 96%)
Max RSS bytes2758234112 (2630 MiB)
Outputting started2026-02-04 00:16:12
Output files
Finished2026-02-04 00:19:01
Saved logsjustin-logs:297919.3-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ght exception in looking up space point ptr... 0x34eb27a0, 0x34eb2610
Caught exception in looking up space point ptr... 0x2c9ef920, 0x283eef00
Caught exception in looking up space point ptr... 0x283eef00, 0x283ef090
Caught exception in looking up space point ptr... 0x24241a40, 0x4ea0d070
Caught exception in looking up space point ptr... 0x4ea0fce0, 0x24241a40
Caught exception in looking up space point ptr... 0x490d0970, 0x490caa40
Caught exception in looking up space point ptr... 0x490caa40, 0x10afd170
Caught exception in looking up space point ptr... 0x490de560, 0x5a056f10
Caught exception in looking up space point ptr... 0x5a056f10, 0x10b0be40
Caught exception in looking up space point ptr... 0x4ea0faf0, 0x4b0c3510
Caught exception in looking up space point ptr... 0x4b0c3510, 0x4b0c3e30
Caught exception in looking up space point ptr... 0x4d5c7670, 0x39690690
Caught exception in looking up space point ptr... 0x39690690, 0x4baaea60
Caught exception in looking up space point ptr... 0x10eb1610, 0x10ea4c80
Caught exception in looking up space point ptr... 0x10ea4c80, 0x39832c00
Caught exception in looking up space point ptr... 0x1623cc00, 0x261b3130
Caught exception in looking up space point ptr... 0x261b3130, 0x1623ca70
Caught exception in looking up space point ptr... 0x160a7e90, 0x160a7c40
Caught exception in looking up space point ptr... 0x160a7c40, 0x868f9f0
Caught exception in looking up space point ptr... 0x45f8fb10, 0x2d7071d0
Caught exception in looking up space point ptr... 0x2d7071d0, 0x45f8fb10
Caught exception in looking up space point ptr... 0x45f8fb10, 0x45f90760
Caught exception in looking up space point ptr... 0x45f90760, 0x45f8fb10
Caught exception in looking up space point ptr... 0x52c50de0, 0x52c50f80
Caught exception in looking up space point ptr... 0x52c50f80, 0x52c50de0
Caught exception in looking up space point ptr... 0x52c50de0, 0x52c51290
Caught exception in looking up space point ptr... 0x52c51290, 0x52c50de0
Caught exception in looking up space point ptr... 0x55a2c6d0, 0x148fcf80
Caught exception in looking up space point ptr... 0x148fcf80, 0x55a2c6d0
Caught exception in looking up space point ptr... 0x558d9010, 0x1138aa00
Caught exception in looking up space point ptr... 0x1138aa00, 0x558d9010
Caught exception in looking up space point ptr... 0x558d9010, 0x161e5420
Caught exception in looking up space point ptr... 0x161e5420, 0x558d9010
Caught exception in looking up space point ptr... 0x12057200, 0x24f32f90
Caught exception in looking up space point ptr... 0x24f32f90, 0x12057200
Caught exception in looking up space point ptr... 0x14fa9310, 0x14fa95a0
Caught exception in looking up space point ptr... 0x14fa95a0, 0x14fa90c0
Caught exception in looking up space point ptr... 0x1063c120, 0x1063bff0
Caught exception in looking up space point ptr... 0x1063bff0, 0x1063c250
Caught exception in looking up space point ptr... 0x1063c250, 0x1063c380
Caught exception in looking up space point ptr... 0x1063c380, 0x39400830
Caught exception in looking up space point ptr... 0xf390d20, 0x36ba5800
Caught exception in looking up space point ptr... 0x36ba5800, 0xf390d20
Caught exception in looking up space point ptr... 0x15f98cc0, 0x15f98410
Caught exception in looking up space point ptr... 0x15f98410, 0x2d1aa900
Caught exception in looking up space point ptr... 0x3e9407e0, 0x2dad84a0
Caught exception in looking up space point ptr... 0x2dad84a0, 0x3e9407e0
Caught exception in looking up space point ptr... 0x3929d870, 0x3e314c60
Caught exception in looking up space point ptr... 0x3e314c60, 0x3582b3f0
Caught exception in looking up space point ptr... 0x501bcd80, 0x4e909120
Caught exception in looking up space point ptr... 0x4e909120, 0x501bcd80
Caught exception in looking up space point ptr... 0x4d71a080, 0x8241570
Caught exception in looking up space point ptr... 0x8241570, 0x46a2e680
Caught exception in looking up space point ptr... 0x160acf80, 0x29448790
Caught exception in looking up space point ptr... 0x29448790, 0x160acf80
Caught exception in looking up space point ptr... 0x2d1a9f00, 0x5344eb50
Caught exception in looking up space point ptr... 0x5344eb50, 0x2d1a9f00
Caught exception in looking up space point ptr... 0x3d343d90, 0x5b601eb0
Caught exception in looking up space point ptr... 0x5b601eb0, 0x3d343d90
Caught exception in looking up space point ptr... 0x160ad1a0, 0x29449d40
Caught exception in looking up space point ptr... 0x29449d40, 0x160ad1a0
Caught exception in looking up space point ptr... 0x160abce0, 0x2f698e50
Caught exception in looking up space point ptr... 0x2f698e50, 0x294498c0
Caught exception in looking up space point ptr... 0x160ac500, 0x160ac6d0
Caught exception in looking up space point ptr... 0x160ac6d0, 0x160ac500
++++>>>> total num hits: 327156, num free: 320440
04-Feb-2026 01:16:03 CET  Closed output file "np02vd_raw_run042421_0066_df-s03-d1_dw_0_20260203T140644_reco_stage0_20260204T001603_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         71.6393       103.469       126.704       108.882       15.4806        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.3591e-05    0.000102413   0.000182942   9.0721e-05    4.06741e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.98163       4.45025       5.1649        4.43895      0.288849        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00981839     0.0143549     0.0272356     0.0127441    0.00451667       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000393235   0.000536879   0.00103666    0.000484545   0.000137681      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00149698    0.00186227    0.00304847     0.0017803    0.000319827      27     
produce:beamevent:BeamEvent                       4.368e-05    0.000128171   0.000896761   9.1721e-05    0.000155986      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.0892        2.72731       3.60879       2.72508      0.354151        27     
produce:ophit:OpHitFinder                         0.0281088     0.0364231     0.0441324     0.0359895    0.00457692       27     
produce:wclsdatavd:WireCellToolkit                 53.1928       60.4954       68.4099       59.7859       4.07604        27     
produce:gaushit:GausHitFinder                     0.567724       1.12592       2.24615       1.12272      0.400247        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000166432   0.00027631    0.000476866   0.000249743   8.26791e-05      27     
produce:reco3d:SpacePointSolver                    3.50625       7.47705       13.7247       6.71055       2.56033        27     
produce:hitpdune:DisambigFromSpacePoints          0.0469053     0.149349      0.260618      0.144267      0.0634157       27     
produce:cluster3d:Cluster3D                       0.776613       23.0645       35.9973       28.6344        13.28         27     
[art]:TriggerResults:TriggerResultInserter       2.3201e-05    3.63024e-05   9.7361e-05      3.2e-05     1.53073e-05      27     
end_path:out1:RootOutput                          4.33e-06     8.19522e-06    2.731e-05      6.6e-06     5.08452e-06      27     
end_path:out1:RootOutput(write)                    2.97824       3.9024        5.2925        3.84009      0.605438        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4781.93 MB
  Peak resident set size usage (VmHWM): 2758.23 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0066_df-s03-d1_dw_0_20260203T140644_reco_stage0_20260204T001603_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0066_df-s03-d1_dw_0_20260203T140644_reco_stage0_20260204T001603_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0066_df-s03-d1_dw_0_20260203T140644_reco_stage0_20260204T001603_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161328.0,
    "core.end_time": 1770164165.0,
    "core.events": [
      10755,
      10761,
      10767,
      10773,
      10779,
      10785,
      10791,
      10797,
      10803,
      10809,
      10815,
      10821,
      10827,
      10833,
      10839,
      10845,
      10851,
      10857,
      10863,
      10869,
      10875,
      10881,
      10887,
      10893,
      10899,
      10905,
      10911
    ],
    "core.event_count": 27,
    "core.first_event_number": 10755,
    "core.last_event_number": 10911,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0066_df-s03-d1_dw_0_20260203T140644.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:45:15 UTC       justIN version: 01.06.00