justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.3@dunegpschedd02.fnal.gov

Jobsub ID297923.3@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce513
Last heartbeat2026-02-04 00:35:50
From worker nodeHostnameb9p35p0008.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:57:24
Input filesvd-protodune:np02vd_raw_run042433_0325_df-s04-d1_dw_0_20260203T204540.hdf5
JobscriptExit code0
Real time36m (2179s)
CPU time34m (2070s = 94%)
Max RSS bytes3412643840 (3254 MiB)
Outputting started2026-02-04 00:33:45
Output files
Finished2026-02-04 00:35:50
Saved logsjustin-logs:297923.3-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e point ptr... 0x62023010, 0x46c36b00
Caught exception in looking up space point ptr... 0x46c36b00, 0x5f74f0c0
Caught exception in looking up space point ptr... 0x62f974e0, 0x699d4240
Caught exception in looking up space point ptr... 0x699d4240, 0x62f974e0
Caught exception in looking up space point ptr... 0x17e73d30, 0x17e73b30
Caught exception in looking up space point ptr... 0x17e73b30, 0x17e73d30
Caught exception in looking up space point ptr... 0x415693d0, 0x32682dd0
Caught exception in looking up space point ptr... 0x32682dd0, 0x415693d0
Caught exception in looking up space point ptr... 0x409192a0, 0xd58ee50
Caught exception in looking up space point ptr... 0x92484f0, 0x409192a0
Caught exception in looking up space point ptr... 0x2f3288c0, 0x81e4a250
Caught exception in looking up space point ptr... 0x81e4a250, 0x2f3288c0
Caught exception in looking up space point ptr... 0x601e9970, 0x411a01a0
Caught exception in looking up space point ptr... 0x411a01a0, 0x601e9970
Caught exception in looking up space point ptr... 0x59e5aa60, 0x6308bab0
Caught exception in looking up space point ptr... 0x404a2bc0, 0x59e5aa60
Caught exception in looking up space point ptr... 0x4765d5e0, 0x4765d200
Caught exception in looking up space point ptr... 0x4765d200, 0x4765d5e0
Caught exception in looking up space point ptr... 0x6d889010, 0x6d888b80
Caught exception in looking up space point ptr... 0x15a69ce0, 0x6d889010
Caught exception in looking up space point ptr... 0x47778ff0, 0x47778d90
Caught exception in looking up space point ptr... 0x4777afd0, 0x4777bb20
Caught exception in looking up space point ptr... 0x4777bb20, 0x477ea4b0
Caught exception in looking up space point ptr... 0x477ea4b0, 0x47778ff0
Caught exception in looking up space point ptr... 0xbfe6b30, 0x872c14b0
Caught exception in looking up space point ptr... 0x872c14b0, 0xbfe6b30
Caught exception in looking up space point ptr... 0x413eb800, 0x77ee9fb0
Caught exception in looking up space point ptr... 0x77ee9fb0, 0x413eb800
Caught exception in looking up space point ptr... 0x739c3020, 0x739c12f0
Caught exception in looking up space point ptr... 0x739c12f0, 0x739c2340
Caught exception in looking up space point ptr... 0x598c7cb0, 0x598c7e00
Caught exception in looking up space point ptr... 0x598c7e00, 0x598c7cb0
Caught exception in looking up space point ptr... 0x8553d700, 0x8553d460
Caught exception in looking up space point ptr... 0x8553d460, 0x8553d700
Caught exception in looking up space point ptr... 0x141e9fb0, 0x411e2ed0
Caught exception in looking up space point ptr... 0x411e2ed0, 0x141e9fb0
Caught exception in looking up space point ptr... 0xd313100, 0xd312e10
Caught exception in looking up space point ptr... 0xd312e10, 0xd313100
Caught exception in looking up space point ptr... 0x19956e20, 0x7db7e60
Caught exception in looking up space point ptr... 0x7db7e60, 0x19956e20
Caught exception in looking up space point ptr... 0x6dd628b0, 0x6dd63000
Caught exception in looking up space point ptr... 0x6dd63000, 0x6dd628b0
Caught exception in looking up space point ptr... 0x3629b500, 0x654d3d20
Caught exception in looking up space point ptr... 0x654d3d20, 0x3629b500
Caught exception in looking up space point ptr... 0x601d0220, 0x601cbdb0
Caught exception in looking up space point ptr... 0x601cbdb0, 0x601d0220
Caught exception in looking up space point ptr... 0x3bd94f50, 0x1741eda0
Caught exception in looking up space point ptr... 0x1741eda0, 0x79924380
Caught exception in looking up space point ptr... 0x7d97280, 0x79924970
Caught exception in looking up space point ptr... 0x79924970, 0x79924840
Caught exception in looking up space point ptr... 0xdbb7050, 0x413c7040
Caught exception in looking up space point ptr... 0x413c7040, 0xdbb7050
Caught exception in looking up space point ptr... 0x3853abf0, 0x3ff9d5f0
Caught exception in looking up space point ptr... 0x3ff9d5f0, 0x3853abf0
Caught exception in looking up space point ptr... 0x606efbf0, 0x606efaf0
Caught exception in looking up space point ptr... 0x606efaf0, 0x606efcf0
Caught exception in looking up space point ptr... 0x65558d30, 0x65555ca0
Caught exception in looking up space point ptr... 0x65555ca0, 0x65558d30
Caught exception in looking up space point ptr... 0x36f58730, 0x83d30410
Caught exception in looking up space point ptr... 0x83d30410, 0x1640c800
Caught exception in looking up space point ptr... 0x1640c800, 0x2ed37f00
Caught exception in looking up space point ptr... 0x2ed37f00, 0x6e5dd580
Caught exception in looking up space point ptr... 0x6e5dd580, 0x36f58730
Caught exception in looking up space point ptr... 0x640a8360, 0x3fd8df50
Caught exception in looking up space point ptr... 0x3fd8df50, 0x640a8360
Caught exception in looking up space point ptr... 0x10c53180, 0x6fc1d870
Caught exception in looking up space point ptr... 0x6fc1d870, 0x10c53180
++++>>>> total num hits: 54299, num free: 51971
04-Feb-2026 01:33:39 CET  Closed output file "np02vd_raw_run042433_0325_df-s04-d1_dw_0_20260203T204540_reco_stage0_20260204T003339_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         102.843       111.969       124.734       110.575       5.33101        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.6451e-05    7.78643e-05   0.00016194    7.5321e-05    2.08616e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                7.73567       8.18955       8.97879       8.06622       0.33417        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0080716     0.0398253      0.17149      0.0228425     0.0386579       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000332182   0.000641179   0.00112966    0.000586893   0.000207184      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000291992   0.000480933   0.000773534   0.000440782   0.000128126      19     
produce:beamevent:BeamEvent                       6.223e-05    0.000115747   0.000715004   7.7541e-05    0.000142385      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.68041       2.15059       2.52148       2.19807      0.188336        19     
produce:ophit:OpHitFinder                         0.0238459     0.0337187     0.0419165     0.0351466    0.00497186       19     
produce:wclsdatavd:WireCellToolkit                 74.3816       80.8704       86.6319       79.937        3.86629        19     
produce:gaushit:GausHitFinder                     0.636616       1.06606       1.46482       1.0785       0.259619        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000152141   0.000229012   0.000457103   0.000214771   6.22334e-05      19     
produce:reco3d:SpacePointSolver                    8.23182       12.9431       21.4455       12.7341       2.71762        19     
produce:hitpdune:DisambigFromSpacePoints          0.0710121     0.133188      0.253714      0.126694      0.0442233       19     
produce:cluster3d:Cluster3D                        1.09524       1.83924       3.69614       1.77396      0.623492        19     
[art]:TriggerResults:TriggerResultInserter       2.6961e-05    4.90661e-05   0.000332182    3.156e-05    6.74241e-05      19     
end_path:out1:RootOutput                           4.7e-06     6.76474e-06    1.812e-05     5.66e-06     3.63127e-06      19     
end_path:out1:RootOutput(write)                    4.04001       4.66066       5.35617       4.66937      0.345343        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5462.02 MB
  Peak resident set size usage (VmHWM): 3412.64 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0325_df-s04-d1_dw_0_20260203T204540_reco_stage0_20260204T003339_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0325_df-s04-d1_dw_0_20260203T204540_reco_stage0_20260204T003339_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0325_df-s04-d1_dw_0_20260203T204540_reco_stage0_20260204T003339_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163052.0,
    "core.end_time": 1770165220.0,
    "core.events": [
      37061,
      37067,
      37073,
      37079,
      37085,
      37091,
      37097,
      37103,
      37109,
      37115,
      37121,
      37127,
      37133,
      37139,
      37145,
      37151,
      37157,
      37163,
      37169
    ],
    "core.event_count": 19,
    "core.first_event_number": 37061,
    "core.last_event_number": 37169,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0325_df-s04-d1_dw_0_20260203T204540.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:45:15 UTC       justIN version: 01.06.00