justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.157@dunegpschedd01.fnal.gov

Jobsub ID303647.157@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-03 23:34:12
From worker nodeHostnameb9p08p6650.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:46:56
Input filesvd-protodune:np02vd_raw_run042421_0140_df-s03-d0_dw_0_20260203T144643.hdf5
JobscriptExit code0
Real time44m (2688s)
CPU time42m (2563s = 95%)
Max RSS bytes2581536768 (2461 MiB)
Outputting started2026-02-03 23:31:45
Output files
Finished2026-02-03 23:34:12
Saved logsjustin-logs:303647.157-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ught exception in looking up space point ptr... 0x361b0d20, 0x2d84cb30
Caught exception in looking up space point ptr... 0x5ccd8d70, 0x5f878ad0
Caught exception in looking up space point ptr... 0x5f878ad0, 0x4f5c0300
Caught exception in looking up space point ptr... 0xa153cb0, 0x2f0072a0
Caught exception in looking up space point ptr... 0x2f0072a0, 0xa153cb0
Caught exception in looking up space point ptr... 0x61743840, 0x61743990
Caught exception in looking up space point ptr... 0x61743990, 0x52fc7080
Caught exception in looking up space point ptr... 0x52fc7080, 0x52fc6ef0
Caught exception in looking up space point ptr... 0x617436f0, 0x61743840
Caught exception in looking up space point ptr... 0x62a873b0, 0x62a8a360
Caught exception in looking up space point ptr... 0x62a8a360, 0x62a8f750
Caught exception in looking up space point ptr... 0x614fb080, 0x4ef51030
Caught exception in looking up space point ptr... 0x39b3fce0, 0x614fb080
Caught exception in looking up space point ptr... 0x39cda530, 0x4667d050
Caught exception in looking up space point ptr... 0x1575bfe0, 0x39cda530
Caught exception in looking up space point ptr... 0x60ea3fe0, 0x60ea3cc0
Caught exception in looking up space point ptr... 0x60ea3cc0, 0x60ea3fe0
Caught exception in looking up space point ptr... 0x4b3e9010, 0x52f46480
Caught exception in looking up space point ptr... 0x4b3e8e80, 0x4b3e9010
Caught exception in looking up space point ptr... 0x47aa7f00, 0x47aa7d70
Caught exception in looking up space point ptr... 0x47aa7d70, 0x47aa7f00
Caught exception in looking up space point ptr... 0x3a6581b0, 0x3a658340
Caught exception in looking up space point ptr... 0x3a658340, 0x37497ba0
Caught exception in looking up space point ptr... 0x37497ba0, 0x37497880
Caught exception in looking up space point ptr... 0x37497880, 0x374976f0
Caught exception in looking up space point ptr... 0x2c2d3330, 0x52adc790
Caught exception in looking up space point ptr... 0x3cc36770, 0x2c2d3330
Caught exception in looking up space point ptr... 0x14fc1be0, 0x28846d30
Caught exception in looking up space point ptr... 0x28846d30, 0x14fc1be0
Caught exception in looking up space point ptr... 0x56324080, 0x35081000
Caught exception in looking up space point ptr... 0x35081000, 0x5ac1a3a0
Caught exception in looking up space point ptr... 0x30a13760, 0x13827e20
Caught exception in looking up space point ptr... 0x13827e20, 0x30a13760
Caught exception in looking up space point ptr... 0x30a13760, 0x8561ba0
Caught exception in looking up space point ptr... 0x8561ba0, 0x30a13760
Caught exception in looking up space point ptr... 0x30a13760, 0x2af43cc0
Caught exception in looking up space point ptr... 0x2af43cc0, 0x30a13760
Caught exception in looking up space point ptr... 0x52f65f20, 0x52f65cc0
Caught exception in looking up space point ptr... 0x52f65cc0, 0x52f65f20
Caught exception in looking up space point ptr... 0x463789d0, 0x46378cf0
Caught exception in looking up space point ptr... 0x46378cf0, 0x463789d0
Caught exception in looking up space point ptr... 0x2d25c550, 0x41f2b7b0
Caught exception in looking up space point ptr... 0x41f2b7b0, 0x37475820
Caught exception in looking up space point ptr... 0x4548cce0, 0x2c0cd400
Caught exception in looking up space point ptr... 0x2c0cd400, 0x4548cce0
Caught exception in looking up space point ptr... 0x5293c470, 0x40e59330
Caught exception in looking up space point ptr... 0x40e59330, 0x5293c470
Caught exception in looking up space point ptr... 0x2ef236c0, 0x2ef233a0
Caught exception in looking up space point ptr... 0x2ef233a0, 0x2ef236c0
Caught exception in looking up space point ptr... 0x43688b60, 0x43688880
Caught exception in looking up space point ptr... 0x43688880, 0x43688b60
Caught exception in looking up space point ptr... 0x30fd3ab0, 0x30fd3970
Caught exception in looking up space point ptr... 0x30fd3970, 0x4edd1e60
Caught exception in looking up space point ptr... 0x2ef10880, 0x2ef10510
Caught exception in looking up space point ptr... 0x2ef10510, 0x2ef10880
Caught exception in looking up space point ptr... 0x52f65930, 0x558131b0
Caught exception in looking up space point ptr... 0x558131b0, 0x52f65930
Caught exception in looking up space point ptr... 0x14a3f320, 0x2db28860
Caught exception in looking up space point ptr... 0x2db28860, 0x14a3f320
Caught exception in looking up space point ptr... 0x46377ee0, 0x46377d50
Caught exception in looking up space point ptr... 0x46377d50, 0x46377ee0
Caught exception in looking up space point ptr... 0x46377ee0, 0x46378070
Caught exception in looking up space point ptr... 0x46378070, 0x46377ee0
Caught exception in looking up space point ptr... 0x46377ee0, 0x46378390
Caught exception in looking up space point ptr... 0x46378390, 0x46377ee0
++++>>>> total num hits: 54098, num free: 52357
04-Feb-2026 00:31:37 CET  Closed output file "np02vd_raw_run042421_0140_df-s03-d0_dw_0_20260203T144643_reco_stage0_20260203T233137_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         84.2924       96.9108       107.945       96.0648       6.74064        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.5261e-05    0.000102527   0.000233362   8.9061e-05    3.84314e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.46889       5.05482       5.47523       5.08498      0.250061        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00900274     0.0224143     0.071169      0.016023      0.0146568       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000411615   0.000608194   0.00116018    0.000548986   0.000188602      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00153514    0.00189428    0.00325644    0.00180256    0.000331523      27     
produce:beamevent:BeamEvent                      6.3641e-05    0.000128339    0.0010987    8.6971e-05    0.000191509      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.49757       3.01392       3.63222       3.00023      0.278811        27     
produce:ophit:OpHitFinder                         0.0260086     0.037195      0.0455456     0.0374873     0.0049323       27     
produce:wclsdatavd:WireCellToolkit                 59.0168       69.9278       76.6445       70.1752       3.86004        27     
produce:gaushit:GausHitFinder                     0.402941       1.13588       2.06417       1.17437      0.350707        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000159612   0.00028514    0.000665698   0.000280333   9.74925e-05      27     
produce:reco3d:SpacePointSolver                    4.85724       11.1938       20.1336       11.1423       3.69244        27     
produce:hitpdune:DisambigFromSpacePoints          0.0366287     0.121457      0.228779      0.117355      0.0469838       27     
produce:cluster3d:Cluster3D                       0.498321       2.13358       3.94879       2.17611      0.780339        27     
[art]:TriggerResults:TriggerResultInserter        2.197e-05    3.63371e-05   8.9221e-05     3.268e-05    1.3558e-05       27     
end_path:out1:RootOutput                          3.46e-06     7.83163e-06    2.369e-05     7.23e-06     3.69099e-06      27     
end_path:out1:RootOutput(write)                    2.91489       4.24051       5.08906       4.59012      0.641681        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4570.23 MB
  Peak resident set size usage (VmHWM): 2581.54 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0140_df-s03-d0_dw_0_20260203T144643_reco_stage0_20260203T233137_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0140_df-s03-d0_dw_0_20260203T144643_reco_stage0_20260203T233137_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0140_df-s03-d0_dw_0_20260203T144643_reco_stage0_20260203T233137_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158830.0,
    "core.end_time": 1770161499.0,
    "core.events": [
      22754,
      22760,
      22766,
      22772,
      22778,
      22784,
      22790,
      22796,
      22802,
      22808,
      22814,
      22820,
      22826,
      22832,
      22838,
      22844,
      22850,
      22856,
      22862,
      22868,
      22874,
      22880,
      22886,
      22892,
      22898,
      22904,
      22910
    ],
    "core.event_count": 27,
    "core.first_event_number": 22754,
    "core.last_event_number": 22910,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0140_df-s03-d0_dw_0_20260203T144643.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:11 UTC       justIN version: 01.06.00