justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.64@dunegpschedd02.fnal.gov

Jobsub ID297921.64@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce508
Last heartbeat2026-02-04 00:27:32
From worker nodeHostnameb9p35p0022.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:51:43
Input filesvd-protodune:np02vd_raw_run042433_0147_df-s04-d1_dw_0_20260203T193759.hdf5
JobscriptExit code0
Real time33m (2028s)
CPU time32m (1963s = 96%)
Max RSS bytes3413504000 (3255 MiB)
Outputting started2026-02-04 00:25:33
Output files
Finished2026-02-04 00:27:32
Saved logsjustin-logs:297921.64-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x63ae5160, 0x6105f150
Caught exception in looking up space point ptr... 0x6105f150, 0x1772b030
Caught exception in looking up space point ptr... 0x11804070, 0x4be7e400
Caught exception in looking up space point ptr... 0x4be7e400, 0x793ca7c0
Caught exception in looking up space point ptr... 0x176704c0, 0x17670360
Caught exception in looking up space point ptr... 0x17670360, 0x176704c0
Caught exception in looking up space point ptr... 0xee5f730, 0xd973aa0
Caught exception in looking up space point ptr... 0xd973aa0, 0xee5f730
Caught exception in looking up space point ptr... 0x62c221f0, 0x62c22060
Caught exception in looking up space point ptr... 0x62c22060, 0x62c221f0
Caught exception in looking up space point ptr... 0x62c221f0, 0x62c21ed0
Caught exception in looking up space point ptr... 0x62c21ed0, 0x62c221f0
Caught exception in looking up space point ptr... 0x3a3c3b40, 0x6df0c680
Caught exception in looking up space point ptr... 0x6df0c680, 0x3a3c3b40
Caught exception in looking up space point ptr... 0x6bbbb250, 0x18c42fe0
Caught exception in looking up space point ptr... 0x6e981e10, 0x6bbbb250
Caught exception in looking up space point ptr... 0x597ae6d0, 0x597a5080
Caught exception in looking up space point ptr... 0x597a5080, 0x597a3c90
Caught exception in looking up space point ptr... 0x1799a770, 0x4eec1880
Caught exception in looking up space point ptr... 0x4eec1880, 0x10f4a570
Caught exception in looking up space point ptr... 0x6f1d0880, 0x4eec12e0
Caught exception in looking up space point ptr... 0x4eec12e0, 0x1799a770
Caught exception in looking up space point ptr... 0x639cd630, 0x639cb7e0
Caught exception in looking up space point ptr... 0x639cb7e0, 0x639cd630
Caught exception in looking up space point ptr... 0x62a2d3c0, 0x46c7e0e0
Caught exception in looking up space point ptr... 0x3b4e4180, 0x62a2d3c0
Caught exception in looking up space point ptr... 0x4ec7af40, 0x1384ed50
Caught exception in looking up space point ptr... 0x1384ed50, 0x4ec7af40
Caught exception in looking up space point ptr... 0x18cb2160, 0x3b30eec0
Caught exception in looking up space point ptr... 0x3b30eec0, 0x18cb2160
Caught exception in looking up space point ptr... 0x1772e320, 0x1772e0c0
Caught exception in looking up space point ptr... 0x1772e0c0, 0x1772df90
Caught exception in looking up space point ptr... 0x1793b690, 0x17936740
Caught exception in looking up space point ptr... 0x17936740, 0x17936560
Caught exception in looking up space point ptr... 0x3df02590, 0x3df02770
Caught exception in looking up space point ptr... 0x3df02770, 0x3df02590
Caught exception in looking up space point ptr... 0x1545a4a0, 0x44675ac0
Caught exception in looking up space point ptr... 0x44675ac0, 0x1545a4a0
Caught exception in looking up space point ptr... 0x1545a4a0, 0x44675e30
Caught exception in looking up space point ptr... 0x44675e30, 0x1545a4a0
Caught exception in looking up space point ptr... 0x1795f180, 0x1795f280
Caught exception in looking up space point ptr... 0x1795f280, 0x1795f180
Caught exception in looking up space point ptr... 0x7b3cfbe0, 0x371a9d60
Caught exception in looking up space point ptr... 0x371a9d60, 0x7b3cfbe0
Caught exception in looking up space point ptr... 0x61059080, 0x61058f50
Caught exception in looking up space point ptr... 0x61058f50, 0x61059080
Caught exception in looking up space point ptr... 0x73054910, 0x6e2049a0
Caught exception in looking up space point ptr... 0x6e2049a0, 0x73054910
Caught exception in looking up space point ptr... 0x1746e710, 0x1746e580
Caught exception in looking up space point ptr... 0x1746e580, 0x1746e710
Caught exception in looking up space point ptr... 0x639dd960, 0x63a59920
Caught exception in looking up space point ptr... 0x63a59920, 0x639dd960
Caught exception in looking up space point ptr... 0x4e742ef0, 0x4e742c80
Caught exception in looking up space point ptr... 0x4e742c80, 0x4e741590
Caught exception in looking up space point ptr... 0x14440200, 0x14440070
Caught exception in looking up space point ptr... 0x14440070, 0x14440200
Caught exception in looking up space point ptr... 0x639ec160, 0x639ebf60
Caught exception in looking up space point ptr... 0x639ebf60, 0x639ec160
Caught exception in looking up space point ptr... 0x4effc470, 0x4effd060
Caught exception in looking up space point ptr... 0x4effd060, 0x4effc210
Caught exception in looking up space point ptr... 0x176389e0, 0x176d1010
Caught exception in looking up space point ptr... 0x176d1010, 0x176d1110
Caught exception in looking up space point ptr... 0x176d1110, 0x17638da0
Caught exception in looking up space point ptr... 0x4e749fe0, 0x4e748c30
Caught exception in looking up space point ptr... 0x4e748c30, 0x4e74a320
Caught exception in looking up space point ptr... 0x4e663c80, 0x4e663af0
Caught exception in looking up space point ptr... 0x4e663af0, 0x4e663960
++++>>>> total num hits: 75143, num free: 71783
04-Feb-2026 01:25:27 CET  Closed output file "np02vd_raw_run042433_0147_df-s04-d1_dw_0_20260203T193759_reco_stage0_20260204T002527_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         94.5894       104.618       113.195       104.457       5.47879        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.111e-05    6.76674e-05   0.000164011   6.1061e-05    2.3948e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                4.08413       5.24802       12.534        4.59149       1.98403        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00819473     0.0133381     0.0357787     0.0111428    0.00679564       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000332482   0.000584198   0.00141219    0.000479422   0.00028968       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000310932   0.000597717   0.00205957    0.000388242   0.000422832      19     
produce:beamevent:BeamEvent                      5.6081e-05    0.000110856   0.000759444    7.02e-05     0.000153891      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.53945       1.88815       2.1466        1.94363      0.187762        19     
produce:ophit:OpHitFinder                         0.0196852     0.0313825     0.0408302     0.0313387    0.00482597       19     
produce:wclsdatavd:WireCellToolkit                 65.7653       72.9773       83.5539       72.1109       5.02205        19     
produce:gaushit:GausHitFinder                     0.720222      0.967889       1.18952      0.975366      0.143254        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000186151   0.000250419   0.000431822   0.000240671   5.63995e-05      19     
produce:reco3d:SpacePointSolver                    11.215        16.4649       20.5457       16.5839       2.37552        19     
produce:hitpdune:DisambigFromSpacePoints          0.093581      0.148161      0.256904      0.132153      0.0460919       19     
produce:cluster3d:Cluster3D                        1.06656       1.91992       2.87154       1.72884      0.499683        19     
[art]:TriggerResults:TriggerResultInserter        2.246e-05    3.2119e-05     8.052e-05     2.935e-05    1.23619e-05      19     
end_path:out1:RootOutput                           4.6e-06     6.54905e-06   1.8361e-05      5.4e-06     3.61192e-06      19     
end_path:out1:RootOutput(write)                    3.70151       4.92186       5.77572       5.03808       0.48889        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5529.95 MB
  Peak resident set size usage (VmHWM): 3413.5 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0147_df-s04-d1_dw_0_20260203T193759_reco_stage0_20260204T002527_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0147_df-s04-d1_dw_0_20260203T193759_reco_stage0_20260204T002527_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0147_df-s04-d1_dw_0_20260203T193759_reco_stage0_20260204T002527_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162709.0,
    "core.end_time": 1770164728.0,
    "core.events": [
      16769,
      16775,
      16781,
      16787,
      16793,
      16799,
      16805,
      16811,
      16817,
      16823,
      16829,
      16835,
      16841,
      16847,
      16853,
      16859,
      16865,
      16871,
      16877
    ],
    "core.event_count": 19,
    "core.first_event_number": 16769,
    "core.last_event_number": 16877,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0147_df-s04-d1_dw_0_20260203T193759.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:13:01 UTC       justIN version: 01.06.00