justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303651.108@dunegpschedd01.fnal.gov

Jobsub ID303651.108@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-03 23:37:35
From worker nodeHostnameb9p35p0043.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:21:32
Input filesvd-protodune:np02vd_raw_run042421_0166_df-s03-d0_dw_0_20260203T150049.hdf5
JobscriptExit code0
Real time14m (876s)
CPU time13m (832s = 94%)
Max RSS bytes2597163008 (2476 MiB)
Outputting started2026-02-03 23:36:09
Output files
Finished2026-02-03 23:37:35
Saved logsjustin-logs:303651.108-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ace point ptr... 0xd9d10d0, 0xd9d62a0
Caught exception in looking up space point ptr... 0x66653d50, 0xeb2d870
Caught exception in looking up space point ptr... 0xeb2d870, 0x66653d50
Caught exception in looking up space point ptr... 0x75dd240, 0x75dd700
Caught exception in looking up space point ptr... 0x75dd700, 0x75dd240
Caught exception in looking up space point ptr... 0x75dd240, 0x75dd370
Caught exception in looking up space point ptr... 0x75dd370, 0x75dd240
Caught exception in looking up space point ptr... 0x20100df0, 0x3f5e1fd0
Caught exception in looking up space point ptr... 0x3f5e1fd0, 0x20100df0
Caught exception in looking up space point ptr... 0x57fae260, 0x57faea50
Caught exception in looking up space point ptr... 0x57fae4f0, 0x57fae260
Caught exception in looking up space point ptr... 0x97fabe0, 0x97f0e90
Caught exception in looking up space point ptr... 0x97f0e90, 0x97fabe0
Caught exception in looking up space point ptr... 0x5813a1d0, 0x40b09d10
Caught exception in looking up space point ptr... 0x40b09d10, 0x5813a1d0
Caught exception in looking up space point ptr... 0x4e8311f0, 0x4e830f90
Caught exception in looking up space point ptr... 0x4e830f90, 0x4e8311f0
Caught exception in looking up space point ptr... 0x75d7440, 0x75d6830
Caught exception in looking up space point ptr... 0x75d6830, 0x75d7440
Caught exception in looking up space point ptr... 0x2656bec0, 0x2656bba0
Caught exception in looking up space point ptr... 0x2656bba0, 0x2656bec0
Caught exception in looking up space point ptr... 0x75faf40, 0x75fb070
Caught exception in looking up space point ptr... 0x75fb070, 0x75faf40
Caught exception in looking up space point ptr... 0x4e535980, 0x4e535660
Caught exception in looking up space point ptr... 0x4e535660, 0x4e535980
Caught exception in looking up space point ptr... 0x981b960, 0x98188c0
Caught exception in looking up space point ptr... 0x98188c0, 0x981b960
Caught exception in looking up space point ptr... 0x59ddc9a0, 0x59ddca70
Caught exception in looking up space point ptr... 0x59ddca70, 0x59ddc9a0
Caught exception in looking up space point ptr... 0x97fd3e0, 0x24b4fc10
Caught exception in looking up space point ptr... 0x24b4fc10, 0x97fd3e0
Caught exception in looking up space point ptr... 0xce05890, 0xce05570
Caught exception in looking up space point ptr... 0xce05570, 0xce053e0
Caught exception in looking up space point ptr... 0x4e557f80, 0x4e557df0
Caught exception in looking up space point ptr... 0x4e557df0, 0x4e557c60
Caught exception in looking up space point ptr... 0x24b17620, 0x24b156d0
Caught exception in looking up space point ptr... 0x24b156d0, 0x24b17620
Caught exception in looking up space point ptr... 0x75d6e90, 0x75d7140
Caught exception in looking up space point ptr... 0x75d7140, 0x75d6e90
Caught exception in looking up space point ptr... 0x97efe40, 0x97efcb0
Caught exception in looking up space point ptr... 0x97efcb0, 0x97efe40
Caught exception in looking up space point ptr... 0x9836770, 0x98365e0
Caught exception in looking up space point ptr... 0x98365e0, 0x9836770
Caught exception in looking up space point ptr... 0x75d7600, 0x75dbf20
Caught exception in looking up space point ptr... 0x75dbf20, 0x75d7600
Caught exception in looking up space point ptr... 0x75e2f20, 0x75e2df0
Caught exception in looking up space point ptr... 0x75e2df0, 0x75e2f20
Caught exception in looking up space point ptr... 0x3df27410, 0x3df2bb40
Caught exception in looking up space point ptr... 0x3df2bb40, 0x3df27410
Caught exception in looking up space point ptr... 0x7a1a410, 0x54daf180
Caught exception in looking up space point ptr... 0x54daf180, 0x5e08e0d0
Caught exception in looking up space point ptr... 0x2da3e5d0, 0x2da42520
Caught exception in looking up space point ptr... 0x2da42520, 0x2da3e5d0
Caught exception in looking up space point ptr... 0x3b75b510, 0x3b75ba60
Caught exception in looking up space point ptr... 0x3b75ba60, 0x3b75b880
Caught exception in looking up space point ptr... 0x139b4b80, 0x139b5090
Caught exception in looking up space point ptr... 0x139b5090, 0x139b4b80
Caught exception in looking up space point ptr... 0x139b4b80, 0x139b5450
Caught exception in looking up space point ptr... 0x139b5450, 0x139b4b80
Caught exception in looking up space point ptr... 0x24b50800, 0x33cbd850
Caught exception in looking up space point ptr... 0x33cbd850, 0x24b50800
Caught exception in looking up space point ptr... 0x128bb180, 0x128bae90
Caught exception in looking up space point ptr... 0x128bae90, 0x128b8ef0
Caught exception in looking up space point ptr... 0xfcf63f0, 0xfcf5dc0
Caught exception in looking up space point ptr... 0xfcf5dc0, 0xfcf6540
Caught exception in looking up space point ptr... 0x1398e570, 0x566c25a0
Caught exception in looking up space point ptr... 0x566c25a0, 0x1398e570
Caught exception in looking up space point ptr... 0x25e572b0, 0x25e57160
Caught exception in looking up space point ptr... 0x25e57160, 0x25e57400
++++>>>> total num hits: 118202, num free: 114027
04-Feb-2026 00:36:03 CET  Closed output file "np02vd_raw_run042421_0166_df-s03-d0_dw_0_20260203T150049_reco_stage0_20260203T233603_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         64.625        74.7386       82.7478       76.0259       6.0442         11     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.225e-05    8.68236e-05   0.00022945     7.426e-05    4.56012e-05      11     
produce:tpcrawdecoder:PDVDTPCReader                3.93224       4.46929       5.08035       4.44365      0.278057        11     
produce:triggerrawdecoder:PDVDTriggerReader4      0.008502      0.0105815     0.015453      0.0100532    0.00188773       11     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00039572    0.000592767   0.00091773    0.00058981    0.00013553       11     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00124215    0.00172641    0.00328045    0.00146031    0.000567405      11     
produce:beamevent:BeamEvent                       7.425e-05    0.000167435   0.00088638     9.439e-05    0.000227782      11     
produce:pdvddaphne:DAPHNEReaderPDVD                1.78176       2.04281       2.26572       2.03434      0.121564        11     
produce:ophit:OpHitFinder                         0.0233689     0.0339621     0.0431627     0.035635     0.00534674       11     
produce:wclsdatavd:WireCellToolkit                 44.5509       50.819        55.0983       52.4716       3.50436        11     
produce:gaushit:GausHitFinder                     0.466239      0.791888       1.08315      0.792741       0.17383        11     
produce:nhitsfilter:NumberOfHitsFilter            0.0001492    0.000208977   0.00035003    0.00018469    6.36716e-05      11     
produce:reco3d:SpacePointSolver                    7.3955        10.7367       14.2553       10.8015       2.47429        11     
produce:hitpdune:DisambigFromSpacePoints          0.0652814     0.103544      0.166901      0.0945895     0.0312834       11     
produce:cluster3d:Cluster3D                       0.947614       1.6413        2.41563       1.53612      0.544506        11     
[art]:TriggerResults:TriggerResultInserter        2.514e-05    3.61009e-05    7.249e-05     2.912e-05    1.50042e-05      11     
end_path:out1:RootOutput                          4.49e-06     6.15818e-06    1.741e-05      5.1e-06     3.58922e-06      11     
end_path:out1:RootOutput(write)                    3.11163       4.06134       4.67235       4.13001      0.445885        11     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4581.77 MB
  Peak resident set size usage (VmHWM): 2597.16 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0166_df-s03-d0_dw_0_20260203T150049_reco_stage0_20260203T233603_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0166_df-s03-d0_dw_0_20260203T150049_reco_stage0_20260203T233603_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0166_df-s03-d0_dw_0_20260203T150049_reco_stage0_20260203T233603_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160899.0,
    "core.end_time": 1770161764.0,
    "core.events": [
      26978,
      26984,
      26990,
      26996,
      27002,
      27008,
      27014,
      27020,
      27026,
      27032,
      27038
    ],
    "core.event_count": 11,
    "core.first_event_number": 26978,
    "core.last_event_number": 27038,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0166_df-s03-d0_dw_0_20260203T150049.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:37 UTC       justIN version: 01.06.00