justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.22@dunegpschedd01.fnal.gov

Jobsub ID303653.22@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce509
Last heartbeat2026-02-04 00:11:13
From worker nodeHostnameb9p34p1022.cern.ch
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:37:16
Input filesvd-protodune:np02vd_raw_run042421_0144_df-s03-d0_dw_0_20260203T144856.hdf5
JobscriptExit code0
Real time31m (1904s)
CPU time30m (1843s = 96%)
Max RSS bytes2552270848 (2434 MiB)
Outputting started2026-02-04 00:09:01
Output files
Finished2026-02-04 00:11:13
Saved logsjustin-logs:303653.22-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Caught exception in looking up space point ptr... 0x47d658b0, 0x47d65b50
Caught exception in looking up space point ptr... 0x44b95fd0, 0x44b95e40
Caught exception in looking up space point ptr... 0x44b95e40, 0x44b95fd0
Caught exception in looking up space point ptr... 0x561bda00, 0x561baa20
Caught exception in looking up space point ptr... 0x561baa20, 0x561b3780
Caught exception in looking up space point ptr... 0x3dd0fab0, 0x3dd12a90
Caught exception in looking up space point ptr... 0x3dd12a90, 0x3dd12bd0
Caught exception in looking up space point ptr... 0x44fe1670, 0x44fe5780
Caught exception in looking up space point ptr... 0x44fe2e40, 0x25eecbb0
Caught exception in looking up space point ptr... 0x25eecbb0, 0x44fe1840
Caught exception in looking up space point ptr... 0x44fe1740, 0x44fe1670
Caught exception in looking up space point ptr... 0x565a4300, 0x13814190
Caught exception in looking up space point ptr... 0x13814190, 0x565a4300
Caught exception in looking up space point ptr... 0x25eccbb0, 0x25eccd00
Caught exception in looking up space point ptr... 0x25eccd00, 0x25ecce50
Caught exception in looking up space point ptr... 0x37018cf0, 0x37019010
Caught exception in looking up space point ptr... 0x37016710, 0x37018cf0
Caught exception in looking up space point ptr... 0x21f99c00, 0x465c8100
Caught exception in looking up space point ptr... 0x465c8100, 0x21f99c00
Caught exception in looking up space point ptr... 0x59560ac0, 0x26a01450
Caught exception in looking up space point ptr... 0x26a01450, 0x59560ac0
Caught exception in looking up space point ptr... 0x3791d730, 0x3791da50
Caught exception in looking up space point ptr... 0x3791da50, 0x3791d730
Caught exception in looking up space point ptr... 0x3791d730, 0x3791dc30
Caught exception in looking up space point ptr... 0x3791dc30, 0x3791d730
Caught exception in looking up space point ptr... 0x336b10a0, 0x65da6ea0
Caught exception in looking up space point ptr... 0x65da6ea0, 0x336b10a0
Caught exception in looking up space point ptr... 0x561a2480, 0x3db54930
Caught exception in looking up space point ptr... 0x3db54930, 0x561a2660
Caught exception in looking up space point ptr... 0x635c9340, 0x635c9dc0
Caught exception in looking up space point ptr... 0x635c9dc0, 0x635c9340
Caught exception in looking up space point ptr... 0x3db482f0, 0x3db48160
Caught exception in looking up space point ptr... 0x3db48160, 0x3db482f0
Caught exception in looking up space point ptr... 0x30b29a50, 0x3c0adc80
Caught exception in looking up space point ptr... 0x3c0adc80, 0x30b29a50
Caught exception in looking up space point ptr... 0x25edc500, 0x25ee0290
Caught exception in looking up space point ptr... 0x25ee0290, 0x25ee1ba0
Caught exception in looking up space point ptr... 0x3db4d3c0, 0x3db4d050
Caught exception in looking up space point ptr... 0x3db4d050, 0x3db4d1e0
Caught exception in looking up space point ptr... 0x5f789cf0, 0x5f788e00
Caught exception in looking up space point ptr... 0x5f788e00, 0x5f78b200
Caught exception in looking up space point ptr... 0x8692ae0, 0x8692be0
Caught exception in looking up space point ptr... 0x8692be0, 0x8692ae0
Caught exception in looking up space point ptr... 0x214006f0, 0x213ffda0
Caught exception in looking up space point ptr... 0x213ffda0, 0x214006f0
Caught exception in looking up space point ptr... 0x341b34e0, 0x341b77c0
Caught exception in looking up space point ptr... 0x341b77c0, 0x341b74a0
Caught exception in looking up space point ptr... 0x2c198a10, 0x2c199560
Caught exception in looking up space point ptr... 0x2c199560, 0x2c198a10
Caught exception in looking up space point ptr... 0x3dceaa20, 0x3dceafd0
Caught exception in looking up space point ptr... 0x3dce7a30, 0x3dceaa20
Caught exception in looking up space point ptr... 0x4e12a480, 0x90591c0
Caught exception in looking up space point ptr... 0x90591c0, 0x90592f0
Caught exception in looking up space point ptr... 0x5d224a90, 0x5d224960
Caught exception in looking up space point ptr... 0x5d224960, 0x5d224a90
Caught exception in looking up space point ptr... 0x336b36f0, 0x336b3560
Caught exception in looking up space point ptr... 0x336b3560, 0x336b36f0
Caught exception in looking up space point ptr... 0x561ba0b0, 0x561b5430
Caught exception in looking up space point ptr... 0x561b5430, 0x561ba0b0
Caught exception in looking up space point ptr... 0x28cdf280, 0x28cdee30
Caught exception in looking up space point ptr... 0x28cdee30, 0x28cde050
Caught exception in looking up space point ptr... 0x3dd6e750, 0x242ae860
Caught exception in looking up space point ptr... 0x242ae860, 0x242ae790
Caught exception in looking up space point ptr... 0x55bd5fc0, 0x29679960
Caught exception in looking up space point ptr... 0x29679960, 0x54277e40
++++>>>> total num hits: 61852, num free: 59117
04-Feb-2026 01:08:54 CET  Closed output file "np02vd_raw_run042421_0144_df-s03-d0_dw_0_20260203T144856_reco_stage0_20260204T000854_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         62.3409       68.3166       80.8191       66.9556       4.39179        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.8908e-05    8.56218e-05   0.00022012    8.1051e-05    2.89959e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                2.74555       3.52391       4.82296       3.36418      0.538146        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00924569     0.0148451     0.0350349     0.0124172    0.00734011       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000439027   0.000623054   0.00192628    0.000544835   0.000278787      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00127333    0.00223407    0.00549386    0.00174851    0.00107822       27     
produce:beamevent:BeamEvent                      7.6745e-05    0.000143482   0.000867999   0.000107561   0.000145783      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.75275       2.10029       2.52656       2.1089       0.164019        27     
produce:ophit:OpHitFinder                         0.022525      0.0334317     0.0417107     0.034444     0.00465355       27     
produce:wclsdatavd:WireCellToolkit                 44.1568       48.5806       54.5288       48.181        2.70538        27     
produce:gaushit:GausHitFinder                      0.44169      0.763694       1.10575      0.750625      0.173747        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000135894   0.00019268    0.000531886   0.000174772   7.24863e-05      27     
produce:reco3d:SpacePointSolver                    4.80191       8.26912       13.6904       7.85256       2.00952        27     
produce:hitpdune:DisambigFromSpacePoints          0.0474906     0.0858212     0.186188      0.0795711     0.0322333       27     
produce:cluster3d:Cluster3D                       0.749051       1.57198       2.86353       1.60998       0.56897        27     
[art]:TriggerResults:TriggerResultInserter       2.4237e-05    3.60998e-05   8.3185e-05    3.2378e-05    1.14545e-05      27     
end_path:out1:RootOutput                          4.947e-06    9.24011e-06   2.1913e-05     8.943e-06    2.91938e-06      27     
end_path:out1:RootOutput(write)                    2.73032       3.34071       4.72938       3.23228      0.453376        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4629.96 MB
  Peak resident set size usage (VmHWM): 2552.27 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0144_df-s03-d0_dw_0_20260203T144856_reco_stage0_20260204T000854_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0144_df-s03-d0_dw_0_20260203T144856_reco_stage0_20260204T000854_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0144_df-s03-d0_dw_0_20260203T144856_reco_stage0_20260204T000854_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161846.0,
    "core.end_time": 1770163735.0,
    "core.events": [
      23408,
      23414,
      23420,
      23426,
      23432,
      23438,
      23444,
      23450,
      23456,
      23462,
      23468,
      23474,
      23480,
      23486,
      23492,
      23498,
      23504,
      23510,
      23516,
      23522,
      23528,
      23534,
      23540,
      23546,
      23552,
      23558,
      23564
    ],
    "core.event_count": 27,
    "core.first_event_number": 23408,
    "core.last_event_number": 23564,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0144_df-s03-d0_dw_0_20260203T144856.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:59 UTC       justIN version: 01.06.00