justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303646.187@dunegpschedd01.fnal.gov

Jobsub ID303646.187@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:44:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce511
Last heartbeat2026-02-03 23:29:12
From worker nodeHostnameb9p02p3983.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:46:50
Input filesvd-protodune:np02vd_raw_run042422_0084_df-s02-d1_dw_0_20260203T160010.hdf5
JobscriptExit code0
Real time40m (2400s)
CPU time38m (2288s = 95%)
Max RSS bytes2588192768 (2468 MiB)
Outputting started2026-02-03 23:26:52
Output files
Finished2026-02-03 23:29:12
Saved logsjustin-logs:303646.187-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

xb7a3de0
Caught exception in looking up space point ptr... 0xb7a3de0, 0x598dd470
Caught exception in looking up space point ptr... 0x29ae0aa0, 0xf70a960
Caught exception in looking up space point ptr... 0x26670220, 0x26670090
Caught exception in looking up space point ptr... 0x26ce4080, 0xf71e830
Caught exception in looking up space point ptr... 0xf71e830, 0xf71e980
Caught exception in looking up space point ptr... 0xf71e980, 0xf71ead0
Caught exception in looking up space point ptr... 0xf71ead0, 0x26670220
Caught exception in looking up space point ptr... 0x11202bf0, 0x616aa410
Caught exception in looking up space point ptr... 0x616aa410, 0x2bf1f9d0
Caught exception in looking up space point ptr... 0x4856a720, 0x11202bf0
Caught exception in looking up space point ptr... 0x54a99090, 0x31c5d730
Caught exception in looking up space point ptr... 0x31c5d730, 0x3069d390
Caught exception in looking up space point ptr... 0x3069d390, 0x16a74430
Caught exception in looking up space point ptr... 0x16a74690, 0x54a99090
Caught exception in looking up space point ptr... 0x581f9a40, 0xed897e0
Caught exception in looking up space point ptr... 0xed897e0, 0x11e4a250
Caught exception in looking up space point ptr... 0xd992d60, 0x148e6750
Caught exception in looking up space point ptr... 0x148e6750, 0xd992d60
Caught exception in looking up space point ptr... 0x590b71d0, 0xdb3d650
Caught exception in looking up space point ptr... 0xdb3d650, 0x2f4d2290
Caught exception in looking up space point ptr... 0x2362a070, 0x310e5fe0
Caught exception in looking up space point ptr... 0x310e5fe0, 0x2362a070
Caught exception in looking up space point ptr... 0x23624de0, 0x56b39e00
Caught exception in looking up space point ptr... 0x56b39e00, 0x24d570e0
Caught exception in looking up space point ptr... 0x15ec24b0, 0x2f76ae30
Caught exception in looking up space point ptr... 0x2f76ae30, 0x15ec24b0
Caught exception in looking up space point ptr... 0x58fb05c0, 0x13f7b5f0
Caught exception in looking up space point ptr... 0x13f7b5f0, 0x612fad10
Caught exception in looking up space point ptr... 0x2a7d5c20, 0x2a7d5b20
Caught exception in looking up space point ptr... 0x2a7d5b20, 0x33984510
Caught exception in looking up space point ptr... 0x13f52b50, 0x3d4c6ad0
Caught exception in looking up space point ptr... 0x3d4c6ad0, 0x3d4c7360
Caught exception in looking up space point ptr... 0x3d4c7360, 0x16bac330
Caught exception in looking up space point ptr... 0x4ee47f10, 0x31b9b200
Caught exception in looking up space point ptr... 0x31b9b200, 0x4ee47f10
Caught exception in looking up space point ptr... 0x4ee47f10, 0x31b55ba0
Caught exception in looking up space point ptr... 0x31b55ba0, 0x4ee47f10
Caught exception in looking up space point ptr... 0x485a44d0, 0x4952c6e0
Caught exception in looking up space point ptr... 0xe99fae0, 0x485a44d0
Caught exception in looking up space point ptr... 0x4d71a780, 0x286fc730
Caught exception in looking up space point ptr... 0x3f83e860, 0x4d71a780
Caught exception in looking up space point ptr... 0x1069ae60, 0x1069a9d0
Caught exception in looking up space point ptr... 0x1069a9d0, 0x1069ae60
Caught exception in looking up space point ptr... 0x339129f0, 0x339128c0
Caught exception in looking up space point ptr... 0x339128c0, 0x339129f0
Caught exception in looking up space point ptr... 0x2bb69820, 0x11a71500
Caught exception in looking up space point ptr... 0x11a71500, 0x2bb69820
Caught exception in looking up space point ptr... 0x2544f290, 0x58ff88b0
Caught exception in looking up space point ptr... 0x2544f130, 0x2544f290
Caught exception in looking up space point ptr... 0x2f1249d0, 0x4c3f2500
Caught exception in looking up space point ptr... 0x4c3f2500, 0x2f1249d0
Caught exception in looking up space point ptr... 0x56629df0, 0x48e6e5d0
Caught exception in looking up space point ptr... 0x48e6e5d0, 0x48e6e370
Caught exception in looking up space point ptr... 0x4eb241a0, 0x31b81480
Caught exception in looking up space point ptr... 0x31b81480, 0x4ed8ce70
Caught exception in looking up space point ptr... 0x5789c9c0, 0x5789c870
Caught exception in looking up space point ptr... 0x5789c870, 0x4284b540
Caught exception in looking up space point ptr... 0x47199580, 0x48c5cec0
Caught exception in looking up space point ptr... 0x48c5cec0, 0x47199580
Caught exception in looking up space point ptr... 0x3cdff5c0, 0x4e1742a0
Caught exception in looking up space point ptr... 0x4e1742a0, 0x3cdff5c0
Caught exception in looking up space point ptr... 0x2cbfbf30, 0x3d64e700
Caught exception in looking up space point ptr... 0x3d64e700, 0x2cbfbf30
Caught exception in looking up space point ptr... 0x2f1472e0, 0x3f59f5c0
Caught exception in looking up space point ptr... 0x3f59f5c0, 0x3d5e0b40
++++>>>> total num hits: 32821, num free: 31549
04-Feb-2026 00:26:44 CET  Closed output file "np02vd_raw_run042422_0084_df-s02-d1_dw_0_20260203T160010_reco_stage0_20260203T232644_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         73.4619       86.3998       98.4286       86.3924       6.82044        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.601e-05    0.000132919   0.000345061   0.00010482    7.56068e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.35279       5.07994       9.63144       4.7327        1.27154        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00933085     0.0160708     0.0702447     0.0124946     0.0122872       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0003669    0.000890125    0.0102091    0.000524551   0.00182963       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00132535     0.0019526    0.00309656    0.00188794    0.000364756      27     
produce:beamevent:BeamEvent                       5.057e-05    0.000130295   0.000976172   9.6751e-05    0.000167751      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.31626       2.88883       3.45618       2.91917      0.282023        27     
produce:ophit:OpHitFinder                         0.0261586     0.0355991     0.0414561     0.0358058    0.00402184       27     
produce:wclsdatavd:WireCellToolkit                 52.9618       61.0598       69.5073       60.3076       4.78678        27     
produce:gaushit:GausHitFinder                     0.439762      0.909879       1.55252      0.885448      0.266494        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000125331   0.000254679   0.000392681   0.000246651   6.14302e-05      27     
produce:reco3d:SpacePointSolver                    4.49429       10.2032       19.2862       9.92735       3.66354        27     
produce:hitpdune:DisambigFromSpacePoints          0.0407932     0.104703       0.17716      0.103629      0.036824        27     
produce:cluster3d:Cluster3D                       0.700993       2.02638       3.91535       1.7872       0.898504        27     
[art]:TriggerResults:TriggerResultInserter        2.019e-05    2.9522e-05     9.774e-05     2.477e-05    1.49801e-05      27     
end_path:out1:RootOutput                          3.49e-06     7.16296e-06    2.913e-05     6.26e-06     4.69672e-06      27     
end_path:out1:RootOutput(write)                    3.08374       4.04521       5.18458       4.13994      0.506245        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4588.58 MB
  Peak resident set size usage (VmHWM): 2588.19 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0084_df-s02-d1_dw_0_20260203T160010_reco_stage0_20260203T232644_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0084_df-s02-d1_dw_0_20260203T160010_reco_stage0_20260203T232644_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0084_df-s02-d1_dw_0_20260203T160010_reco_stage0_20260203T232644_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158822.0,
    "core.end_time": 1770161205.0,
    "core.events": [
      13621,
      13627,
      13633,
      13639,
      13645,
      13651,
      13657,
      13663,
      13669,
      13675,
      13681,
      13687,
      13693,
      13699,
      13705,
      13711,
      13717,
      13723,
      13729,
      13735,
      13741,
      13747,
      13753,
      13759,
      13765,
      13771,
      13777
    ],
    "core.event_count": 27,
    "core.first_event_number": 13621,
    "core.last_event_number": 13777,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0084_df-s02-d1_dw_0_20260203T160010.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:08:23 UTC       justIN version: 01.06.00