justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.145@dunegpschedd02.fnal.gov

Jobsub ID297921.145@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2026-02-04 00:33:27
From worker nodeHostnameb9p35p0031.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:56:37
Input filesvd-protodune:np02vd_raw_run042433_0566_df-s04-d0_dw_0_20260203T221715.hdf5
JobscriptExit code0
Real time34m (2088s)
CPU time33m (2009s = 96%)
Max RSS bytes3381538816 (3224 MiB)
Outputting started2026-02-04 00:31:26
Output files
Finished2026-02-04 00:33:27
Saved logsjustin-logs:297921.145-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e point ptr... 0x541c3010, 0x541bf380
Caught exception in looking up space point ptr... 0x6e1eebe0, 0x3fb8bab0
Caught exception in looking up space point ptr... 0x3fb8bab0, 0x1558bbd0
Caught exception in looking up space point ptr... 0x4b2d94d0, 0x5820f020
Caught exception in looking up space point ptr... 0x5820f020, 0x4b2d94d0
Caught exception in looking up space point ptr... 0xb18dc90, 0x2cc87fe0
Caught exception in looking up space point ptr... 0x7e1f63f0, 0xb18dc90
Caught exception in looking up space point ptr... 0x86206970, 0x862072a0
Caught exception in looking up space point ptr... 0x862072a0, 0x332ee380
Caught exception in looking up space point ptr... 0x6e7d5000, 0x6e7dffd0
Caught exception in looking up space point ptr... 0x6e7dffd0, 0x6e7cc730
Caught exception in looking up space point ptr... 0x538833d0, 0x195194b0
Caught exception in looking up space point ptr... 0x195194b0, 0x538833d0
Caught exception in looking up space point ptr... 0x538833d0, 0x4b30ed90
Caught exception in looking up space point ptr... 0x4b30ed90, 0x538833d0
Caught exception in looking up space point ptr... 0x32d67500, 0x94c4080
Caught exception in looking up space point ptr... 0x94c4080, 0x478a1a40
Caught exception in looking up space point ptr... 0x478a1a40, 0x294d61c0
Caught exception in looking up space point ptr... 0x7c6c3a90, 0x7c6c1050
Caught exception in looking up space point ptr... 0x7c6c1050, 0x7c6c07e0
Caught exception in looking up space point ptr... 0x6e7efe90, 0x6e7f21a0
Caught exception in looking up space point ptr... 0x6e7f21a0, 0x6e7efe90
Caught exception in looking up space point ptr... 0x6e8da360, 0x6e8e3860
Caught exception in looking up space point ptr... 0x6e8e3860, 0x6e8e5ad0
Caught exception in looking up space point ptr... 0x4e5a76e0, 0x4e5c1ea0
Caught exception in looking up space point ptr... 0x4e5c1ea0, 0x4e5a76e0
Caught exception in looking up space point ptr... 0x862f4ad0, 0x862f42e0
Caught exception in looking up space point ptr... 0x13c30f30, 0x862f4ad0
Caught exception in looking up space point ptr... 0x5fc207c0, 0x5fc20670
Caught exception in looking up space point ptr... 0x5fc20670, 0x5fc207c0
Caught exception in looking up space point ptr... 0x7c6cf450, 0x7c6cf6d0
Caught exception in looking up space point ptr... 0x7c6cf6d0, 0x7c6cf450
Caught exception in looking up space point ptr... 0x7c6cf450, 0x7c6cf580
Caught exception in looking up space point ptr... 0x7c6cf580, 0x7c6cf450
Caught exception in looking up space point ptr... 0x86905d40, 0x86907650
Caught exception in looking up space point ptr... 0x86907650, 0x86905d40
Caught exception in looking up space point ptr... 0x4d8a49d0, 0x4d8a4050
Caught exception in looking up space point ptr... 0x4d8a4050, 0x4d8a4470
Caught exception in looking up space point ptr... 0x863430f0, 0x863432d0
Caught exception in looking up space point ptr... 0x863432d0, 0x863430f0
Caught exception in looking up space point ptr... 0x46d247e0, 0x46d24a80
Caught exception in looking up space point ptr... 0x46d24a80, 0x46d247e0
Caught exception in looking up space point ptr... 0x88cb020, 0x6024dc90
Caught exception in looking up space point ptr... 0x6024dc90, 0x6024de20
Caught exception in looking up space point ptr... 0x6024de20, 0x6eaf4c50
Caught exception in looking up space point ptr... 0x6841c910, 0x78a375d0
Caught exception in looking up space point ptr... 0x78a375d0, 0x6900caf0
Caught exception in looking up space point ptr... 0x6e8faf20, 0x6e8facd0
Caught exception in looking up space point ptr... 0x6e8f9b70, 0x6e8faf20
Caught exception in looking up space point ptr... 0x6eb55990, 0x37d25a30
Caught exception in looking up space point ptr... 0x37d25a30, 0x6eb55990
Caught exception in looking up space point ptr... 0x59cbee20, 0x59cc5100
Caught exception in looking up space point ptr... 0x59cc5100, 0x59cbee20
Caught exception in looking up space point ptr... 0x5af08d30, 0x8631ce60
Caught exception in looking up space point ptr... 0x8631ce60, 0x5af08be0
Caught exception in looking up space point ptr... 0xde3e230, 0xde3e490
Caught exception in looking up space point ptr... 0xde3e490, 0xde3afe0
Caught exception in looking up space point ptr... 0x19238d00, 0x4e572df0
Caught exception in looking up space point ptr... 0x4e572df0, 0x19238d00
Caught exception in looking up space point ptr... 0x6ac39410, 0x6ac395a0
Caught exception in looking up space point ptr... 0x6ac395a0, 0x6ac39730
Caught exception in looking up space point ptr... 0x6a9b2770, 0x6a9b0d70
Caught exception in looking up space point ptr... 0x6a9b0d70, 0x58155f90
Caught exception in looking up space point ptr... 0x61dee1d0, 0x61dee2d0
Caught exception in looking up space point ptr... 0x61dee2d0, 0x61dee1d0
Caught exception in looking up space point ptr... 0xe138ec0, 0xe1391e0
Caught exception in looking up space point ptr... 0xe1391e0, 0xe138ec0
++++>>>> total num hits: 64187, num free: 62208
04-Feb-2026 01:31:20 CET  Closed output file "np02vd_raw_run042433_0566_df-s04-d0_dw_0_20260203T221715_reco_stage0_20260204T003120_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         96.9309       106.989       117.336       106.495       6.15444        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.575e-05    7.42168e-05   0.00017296     7.007e-05    2.42623e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.97868       6.52987       6.99979       6.48153      0.263414        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00818719     0.015872      0.0429741     0.0147525    0.00828987       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00042543    0.000536032   0.00070782    0.000527091   7.6909e-05       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.000352     0.000461978   0.000645521   0.00042343    9.58435e-05      19     
produce:beamevent:BeamEvent                       6.603e-05    0.000117522   0.00078315     7.802e-05    0.000157446      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.65519       2.01052       2.34637       1.98136      0.207982        19     
produce:ophit:OpHitFinder                         0.0233032     0.0334505     0.0430159     0.0337877    0.00574949       19     
produce:wclsdatavd:WireCellToolkit                 68.3572       78.9064       87.9202       79.0509       5.16526        19     
produce:gaushit:GausHitFinder                      0.62139      0.898931       1.28883      0.879568      0.168568        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00016769    0.00022211    0.00037357    0.00021522    4.87236e-05      19     
produce:reco3d:SpacePointSolver                    8.91042       11.997        16.0257       11.7872       1.74205        19     
produce:hitpdune:DisambigFromSpacePoints          0.0756691     0.126373      0.202647      0.118779      0.0361453       19     
produce:cluster3d:Cluster3D                       0.820234       1.73421       3.15135       1.55952      0.593619        19     
[art]:TriggerResults:TriggerResultInserter        2.263e-05    3.39289e-05    9.39e-05      2.95e-05     1.5463e-05       19     
end_path:out1:RootOutput                          4.62e-06     6.89316e-06    2.232e-05      5.5e-06     4.35541e-06      19     
end_path:out1:RootOutput(write)                    4.22991       4.69378       5.62384       4.62606      0.388445        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5423.89 MB
  Peak resident set size usage (VmHWM): 3381.54 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0566_df-s04-d0_dw_0_20260203T221715_reco_stage0_20260204T003120_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0566_df-s04-d0_dw_0_20260203T221715_reco_stage0_20260204T003120_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0566_df-s04-d0_dw_0_20260203T221715_reco_stage0_20260204T003120_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163004.0,
    "core.end_time": 1770165081.0,
    "core.events": [
      64534,
      64540,
      64546,
      64552,
      64558,
      64564,
      64570,
      64576,
      64582,
      64588,
      64594,
      64600,
      64606,
      64612,
      64618,
      64624,
      64630,
      64636,
      64642
    ],
    "core.event_count": 19,
    "core.first_event_number": 64534,
    "core.last_event_number": 64642,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0566_df-s04-d0_dw_0_20260203T221715.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:37:45 UTC       justIN version: 01.06.00