justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.151@dunegpschedd01.fnal.gov

Jobsub ID303652.151@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-04 00:23:17
From worker nodeHostnameb9g24p4713.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:33:04
Input filesvd-protodune:np02vd_raw_run042433_0028_df-s02-d1_dw_0_20260203T185245.hdf5
JobscriptExit code0
Real time47m (2874s)
CPU time46m (2780s = 96%)
Max RSS bytes3412504576 (3254 MiB)
Outputting started2026-02-04 00:20:59
Output files
Finished2026-02-04 00:23:17
Saved logsjustin-logs:303652.151-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ion in looking up space point ptr... 0x9e811c0, 0x9e877a0
Caught exception in looking up space point ptr... 0x9e877a0, 0x9e87480
Caught exception in looking up space point ptr... 0x57b8f280, 0x57b8f370
Caught exception in looking up space point ptr... 0x57b8f370, 0x57b8f280
Caught exception in looking up space point ptr... 0x4e6e08b0, 0x4e6e03b0
Caught exception in looking up space point ptr... 0x4e6e03b0, 0x2f622ff0
Caught exception in looking up space point ptr... 0x52582a50, 0x525824a0
Caught exception in looking up space point ptr... 0x525824a0, 0x3ad2a300
Caught exception in looking up space point ptr... 0x2c4b38c0, 0x2c4b3790
Caught exception in looking up space point ptr... 0x4795d930, 0x2c4b38c0
Caught exception in looking up space point ptr... 0x38c947f0, 0x77aeeb70
Caught exception in looking up space point ptr... 0x77aeeb70, 0x38c16160
Caught exception in looking up space point ptr... 0x85de7c0, 0x2e080e80
Caught exception in looking up space point ptr... 0x2e081600, 0x85de7c0
Caught exception in looking up space point ptr... 0x4797cea0, 0x6984d640
Caught exception in looking up space point ptr... 0x6984de10, 0x4797cea0
Caught exception in looking up space point ptr... 0x31f42840, 0x3d0b9cf0
Caught exception in looking up space point ptr... 0x3d0b9cf0, 0x31f42840
Caught exception in looking up space point ptr... 0x501dd210, 0x501dd0c0
Caught exception in looking up space point ptr... 0x501dd0c0, 0x501dd210
Caught exception in looking up space point ptr... 0x523e04c0, 0x523e0350
Caught exception in looking up space point ptr... 0x523e0350, 0x523e04c0
Caught exception in looking up space point ptr... 0x35b377c0, 0x4f1dcb40
Caught exception in looking up space point ptr... 0x4d176570, 0x35b377c0
Caught exception in looking up space point ptr... 0x42240a30, 0x4223e660
Caught exception in looking up space point ptr... 0x4223e660, 0x4223e7b0
Caught exception in looking up space point ptr... 0x13d84790, 0x13d84530
Caught exception in looking up space point ptr... 0x13d84530, 0x13d84790
Caught exception in looking up space point ptr... 0x822a5150, 0x822a5280
Caught exception in looking up space point ptr... 0x822a5280, 0x822a54e0
Caught exception in looking up space point ptr... 0x822a54e0, 0x822a5610
Caught exception in looking up space point ptr... 0x3afe0560, 0x3afdde10
Caught exception in looking up space point ptr... 0x3afdde10, 0x3afe0560
Caught exception in looking up space point ptr... 0x13e609a0, 0x13e60b30
Caught exception in looking up space point ptr... 0x13e60b30, 0x13e609a0
Caught exception in looking up space point ptr... 0x3a4399b0, 0x3a439b10
Caught exception in looking up space point ptr... 0x3a439b10, 0x3a4399b0
Caught exception in looking up space point ptr... 0x5266c640, 0x52669920
Caught exception in looking up space point ptr... 0x5266a770, 0x5266c640
Caught exception in looking up space point ptr... 0x3b6ede60, 0x3b6edff0
Caught exception in looking up space point ptr... 0x3b6edff0, 0x61cc1ab0
Caught exception in looking up space point ptr... 0x52680830, 0x52688b00
Caught exception in looking up space point ptr... 0x52688b00, 0x52680830
Caught exception in looking up space point ptr... 0x3b4112a0, 0x4259df90
Caught exception in looking up space point ptr... 0x4259df90, 0x3b4112a0
Caught exception in looking up space point ptr... 0x665f65c0, 0x665f6740
Caught exception in looking up space point ptr... 0x665f6740, 0x665f65c0
Caught exception in looking up space point ptr... 0x5233e240, 0x5233f4c0
Caught exception in looking up space point ptr... 0x5233f4c0, 0x5233e240
Caught exception in looking up space point ptr... 0x66883e90, 0x66882c90
Caught exception in looking up space point ptr... 0x66882c90, 0x66884740
Caught exception in looking up space point ptr... 0x17857260, 0x178574c0
Caught exception in looking up space point ptr... 0x178574c0, 0x17857260
Caught exception in looking up space point ptr... 0x57b93a00, 0x80a55db0
Caught exception in looking up space point ptr... 0x80a55db0, 0x57b932f0
Caught exception in looking up space point ptr... 0x5701a670, 0x525bb010
Caught exception in looking up space point ptr... 0x525bb010, 0x5701a5a0
Caught exception in looking up space point ptr... 0x65d58630, 0x4ec37e90
Caught exception in looking up space point ptr... 0x4ec37e90, 0x65d58630
Caught exception in looking up space point ptr... 0x66893e30, 0x66893fc0
Caught exception in looking up space point ptr... 0x66893fc0, 0x66894150
Caught exception in looking up space point ptr... 0x19917160, 0x360057c0
Caught exception in looking up space point ptr... 0x360057c0, 0x19917160
Caught exception in looking up space point ptr... 0x610eb1b0, 0x5159dff0
Caught exception in looking up space point ptr... 0x5159dff0, 0x610eb550
Caught exception in looking up space point ptr... 0xd6ac2e0, 0xd6a8c60
Caught exception in looking up space point ptr... 0xd6a8c60, 0xd6ac2e0
++++>>>> total num hits: 67542, num free: 64504
04-Feb-2026 01:20:52 CET  Closed output file "np02vd_raw_run042433_0028_df-s02-d1_dw_0_20260203T185245_reco_stage0_20260204T002052_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         131.629       148.031       178.548       145.394       11.5788        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.4541e-05    8.38758e-05   0.000176643   6.8132e-05    3.36258e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.06116       7.08604       9.38034       6.84702      0.694809        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0101722     0.0189815     0.0400314     0.0130342     0.0102212       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000335776   0.000575705   0.000946738   0.000537221   0.000144577      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000357627   0.000498366   0.000876728   0.000477909   0.000122809      19     
produce:beamevent:BeamEvent                      5.0381e-05    0.000124248   0.000918628   7.2931e-05    0.000189284      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.14762       2.52406       3.15152       2.45857      0.288301        19     
produce:ophit:OpHitFinder                         0.0233789     0.0327232     0.0442064     0.0324308    0.00479954       19     
produce:wclsdatavd:WireCellToolkit                 99.7816       113.518       131.559       111.647       9.19152        19     
produce:gaushit:GausHitFinder                     0.868046       1.22305       2.30953       1.07178      0.352279        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000286145   0.000390584   0.000862497   0.000337727   0.000130272      19     
produce:reco3d:SpacePointSolver                    10.5327       15.701        23.5693       15.1308       3.80753        19     
produce:hitpdune:DisambigFromSpacePoints           0.14821      0.196889      0.408463      0.167572      0.0600507       19     
produce:cluster3d:Cluster3D                        1.62319       2.71377       6.24998       2.51782       1.14985        19     
[art]:TriggerResults:TriggerResultInserter        2.522e-05    3.64801e-05   9.2492e-05    3.0691e-05    1.54776e-05      19     
end_path:out1:RootOutput                          3.571e-06    7.89021e-06   2.2151e-05     7.97e-06     3.99143e-06      19     
end_path:out1:RootOutput(write)                    4.51477       4.98305       6.97303       4.75976      0.587334        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5352 MB
  Peak resident set size usage (VmHWM): 3412.5 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0028_df-s02-d1_dw_0_20260203T185245_reco_stage0_20260204T002052_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0028_df-s02-d1_dw_0_20260203T185245_reco_stage0_20260204T002052_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0028_df-s02-d1_dw_0_20260203T185245_reco_stage0_20260204T002052_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161595.0,
    "core.end_time": 1770164453.0,
    "core.events": [
      3199,
      3205,
      3211,
      3217,
      3223,
      3229,
      3235,
      3241,
      3247,
      3253,
      3259,
      3265,
      3271,
      3277,
      3283,
      3289,
      3295,
      3301,
      3307
    ],
    "core.event_count": 19,
    "core.first_event_number": 3199,
    "core.last_event_number": 3307,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0028_df-s02-d1_dw_0_20260203T185245.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:41:40 UTC       justIN version: 01.06.00