justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303651.129@dunegpschedd01.fnal.gov

Jobsub ID303651.129@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-03 23:57:27
From worker nodeHostnameb9p35p0043.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:22:45
Input filesvd-protodune:np02vd_raw_run042421_0005_df-s04-d1_dw_0_20260203T133335.hdf5
JobscriptExit code0
Real time32m (1955s)
CPU time31m (1872s = 95%)
Max RSS bytes2579795968 (2460 MiB)
Outputting started2026-02-03 23:55:22
Output files
Finished2026-02-03 23:57:27
Saved logsjustin-logs:303651.129-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ht exception in looking up space point ptr... 0x3f4a5d40, 0x9edfdb0
Caught exception in looking up space point ptr... 0x9edfdb0, 0x3f49a4c0
Caught exception in looking up space point ptr... 0x2c950720, 0x4e31a4e0
Caught exception in looking up space point ptr... 0x4e31a4e0, 0x2c950720
Caught exception in looking up space point ptr... 0x2a476430, 0x2a47feb0
Caught exception in looking up space point ptr... 0x2a47feb0, 0x2a47d410
Caught exception in looking up space point ptr... 0xeccfb70, 0x3f95c160
Caught exception in looking up space point ptr... 0x3f95c160, 0xeccfb70
Caught exception in looking up space point ptr... 0x42925590, 0x42925220
Caught exception in looking up space point ptr... 0x42925220, 0x42925590
Caught exception in looking up space point ptr... 0x41e075f0, 0x41e067b0
Caught exception in looking up space point ptr... 0x41e067b0, 0x41e04ee0
Caught exception in looking up space point ptr... 0x41e04ee0, 0x41e067b0
Caught exception in looking up space point ptr... 0x41e067b0, 0x41e01770
Caught exception in looking up space point ptr... 0x41e01770, 0x41e067b0
Caught exception in looking up space point ptr... 0x41e067b0, 0x41e075f0
Caught exception in looking up space point ptr... 0x417ef5a0, 0x4e541180
Caught exception in looking up space point ptr... 0x4e541180, 0x29c3e050
Caught exception in looking up space point ptr... 0x5ba81450, 0x5ba7ea50
Caught exception in looking up space point ptr... 0x5ba7ea50, 0x4e2d21f0
Caught exception in looking up space point ptr... 0x15462470, 0x15463f00
Caught exception in looking up space point ptr... 0x15463f00, 0x154626c0
Caught exception in looking up space point ptr... 0x3d9f5fd0, 0x10c99c90
Caught exception in looking up space point ptr... 0x10c99c90, 0x1475a6c0
Caught exception in looking up space point ptr... 0xf0a4890, 0xf0a4d30
Caught exception in looking up space point ptr... 0xf0a4d30, 0xf0a4890
Caught exception in looking up space point ptr... 0xf0a4890, 0xf0a4380
Caught exception in looking up space point ptr... 0xf0a4380, 0xf0a4890
Caught exception in looking up space point ptr... 0x506198a0, 0x50619650
Caught exception in looking up space point ptr... 0x50619650, 0x506198a0
Caught exception in looking up space point ptr... 0x387dd440, 0x387df9c0
Caught exception in looking up space point ptr... 0x387df9c0, 0x387dd0a0
Caught exception in looking up space point ptr... 0x2bee62e0, 0x2bee55d0
Caught exception in looking up space point ptr... 0x2bee55d0, 0x2bee62e0
Caught exception in looking up space point ptr... 0x15ca4b30, 0x34193b60
Caught exception in looking up space point ptr... 0x34193b60, 0x15ca4b30
Caught exception in looking up space point ptr... 0x61955aa0, 0x3fdd7210
Caught exception in looking up space point ptr... 0x3fdd7210, 0x61954060
Caught exception in looking up space point ptr... 0xd1fb640, 0xd1fad30
Caught exception in looking up space point ptr... 0xd1fad30, 0xd1fb640
Caught exception in looking up space point ptr... 0x2a26d810, 0x2a26d6c0
Caught exception in looking up space point ptr... 0x2a26d6c0, 0x2a26d810
Caught exception in looking up space point ptr... 0x37bf39e0, 0x37bf3890
Caught exception in looking up space point ptr... 0x37bf3890, 0x37bf39e0
Caught exception in looking up space point ptr... 0x29c61890, 0x29c613a0
Caught exception in looking up space point ptr... 0x29c613a0, 0x29c61640
Caught exception in looking up space point ptr... 0x3f6fc2c0, 0x3f6fc630
Caught exception in looking up space point ptr... 0x3f6fc630, 0x3f6fc2c0
Caught exception in looking up space point ptr... 0x2b47b420, 0x2b4769d0
Caught exception in looking up space point ptr... 0x2b47dc70, 0x2b47b420
Caught exception in looking up space point ptr... 0x5a453370, 0x3beb3140
Caught exception in looking up space point ptr... 0x3beb3140, 0x286e68f0
Caught exception in looking up space point ptr... 0x10b95aa0, 0x10b95f50
Caught exception in looking up space point ptr... 0x10b95f50, 0x10b95aa0
Caught exception in looking up space point ptr... 0x347cb190, 0x347cb640
Caught exception in looking up space point ptr... 0x347cb640, 0x347cb190
Caught exception in looking up space point ptr... 0x41179290, 0x41179420
Caught exception in looking up space point ptr... 0x41179420, 0x411795b0
Caught exception in looking up space point ptr... 0x30c458e0, 0x30c45a70
Caught exception in looking up space point ptr... 0x30c45a70, 0x30c458e0
Caught exception in looking up space point ptr... 0xa53e670, 0x506182a0
Caught exception in looking up space point ptr... 0x506182a0, 0xa53e670
Caught exception in looking up space point ptr... 0x5dbf8800, 0x5dbf86b0
Caught exception in looking up space point ptr... 0x5dbf86b0, 0x5dbf8800
Caught exception in looking up space point ptr... 0x408e1880, 0x408e4a20
Caught exception in looking up space point ptr... 0x408e4a20, 0x408e4cd0
++++>>>> total num hits: 113828, num free: 110079
04-Feb-2026 00:55:15 CET  Closed output file "np02vd_raw_run042421_0005_df-s04-d1_dw_0_20260203T133335_reco_stage0_20260203T235515_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         62.1279       70.1658       79.9276       69.9637       4.11632        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.785e-05    7.15981e-05   0.00018339     6.996e-05    2.28796e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.49284       4.18181       8.91569       3.84944       1.19786        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00822273     0.0174335     0.0753947     0.0112457     0.0144656       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00032278    0.000502449   0.00087192    0.00047037    0.000117729      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00121979    0.00156345    0.00290678    0.00148236    0.000310065      27     
produce:beamevent:BeamEvent                       6.42e-05     0.000106963   0.00070698     7.795e-05    0.000119202      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.74803       2.06624       2.39664       2.10613      0.150212        27     
produce:ophit:OpHitFinder                         0.0188818     0.033039      0.0415805     0.0329716    0.00547435       27     
produce:wclsdatavd:WireCellToolkit                 45.696        50.2841       57.8218       49.9974       3.23422        27     
produce:gaushit:GausHitFinder                     0.407418      0.712018       1.36304      0.706743      0.197939        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00010257    0.000173628    0.0003856    0.00016057    5.3325e-05       27     
produce:reco3d:SpacePointSolver                    3.83199       7.92246       16.4939       6.83157       3.15605        27     
produce:hitpdune:DisambigFromSpacePoints          0.0348155     0.0792377     0.171238      0.0714125     0.0319369       27     
produce:cluster3d:Cluster3D                       0.438038       1.23893       2.34591       1.14265      0.532671        27     
[art]:TriggerResults:TriggerResultInserter        2.268e-05     3.483e-05     8.891e-05     3.268e-05    1.20819e-05      27     
end_path:out1:RootOutput                          4.24e-06     6.08296e-06    2.098e-05     5.52e-06     3.0005e-06       27     
end_path:out1:RootOutput(write)                    2.77219       3.60717       4.9261        3.58363      0.447361        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4578.05 MB
  Peak resident set size usage (VmHWM): 2579.8 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0005_df-s04-d1_dw_0_20260203T133335_reco_stage0_20260203T235515_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0005_df-s04-d1_dw_0_20260203T133335_reco_stage0_20260203T235515_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0005_df-s04-d1_dw_0_20260203T133335_reco_stage0_20260203T235515_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160974.0,
    "core.end_time": 1770162916.0,
    "core.events": [
      815,
      821,
      827,
      833,
      839,
      845,
      851,
      857,
      863,
      869,
      875,
      881,
      887,
      893,
      899,
      905,
      911,
      917,
      923,
      929,
      935,
      941,
      947,
      953,
      959,
      965,
      971
    ],
    "core.event_count": 27,
    "core.first_event_number": 815,
    "core.last_event_number": 971,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0005_df-s04-d1_dw_0_20260203T133335.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:31 UTC       justIN version: 01.06.00