justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.143@dunegpschedd02.fnal.gov

Jobsub ID297921.143@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce509
Last heartbeat2026-02-04 00:32:54
From worker nodeHostnameb9p34p1022.cern.ch
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:55:36
Input filesvd-protodune:np02vd_raw_run042433_0391_df-s02-d1_dw_0_20260203T211043.hdf5
JobscriptExit code0
Real time35m (2108s)
CPU time33m (2038s = 96%)
Max RSS bytes3405983744 (3248 MiB)
Outputting started2026-02-04 00:30:46
Output files
Finished2026-02-04 00:32:54
Saved logsjustin-logs:297921.143-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

int ptr... 0x60de63f0, 0x60dea6e0
Caught exception in looking up space point ptr... 0x66fc3810, 0x587a0530
Caught exception in looking up space point ptr... 0x587a0530, 0x3de6dfa0
Caught exception in looking up space point ptr... 0x6ef5d8c0, 0x2d1eedc0
Caught exception in looking up space point ptr... 0x2d1eedc0, 0x6ef5d8c0
Caught exception in looking up space point ptr... 0x54d73d30, 0x31a954c0
Caught exception in looking up space point ptr... 0x31a954c0, 0x4d351210
Caught exception in looking up space point ptr... 0x3fe33560, 0xfee4c30
Caught exception in looking up space point ptr... 0x6c7caa90, 0x3fe33560
Caught exception in looking up space point ptr... 0x2e6e5130, 0x2e6e1a80
Caught exception in looking up space point ptr... 0x2e6e1a80, 0x2e6e5130
Caught exception in looking up space point ptr... 0x2e6e2660, 0x2e6e28b0
Caught exception in looking up space point ptr... 0x2e6e7180, 0x2e6e2660
Caught exception in looking up space point ptr... 0x35b8cad0, 0x35b8c1f0
Caught exception in looking up space point ptr... 0x35b8cdf0, 0x35b8cad0
Caught exception in looking up space point ptr... 0x334bb540, 0x124cba90
Caught exception in looking up space point ptr... 0x334bb440, 0x334bb540
Caught exception in looking up space point ptr... 0x45b7eef0, 0x45b7eca0
Caught exception in looking up space point ptr... 0x1628c8c0, 0x45b7eef0
Caught exception in looking up space point ptr... 0x3605fb10, 0x3605c9f0
Caught exception in looking up space point ptr... 0x3605c9f0, 0x3605fb10
Caught exception in looking up space point ptr... 0x5ddb8250, 0x8249e310
Caught exception in looking up space point ptr... 0x8249e310, 0x32142610
Caught exception in looking up space point ptr... 0x31f94cc0, 0x594980e0
Caught exception in looking up space point ptr... 0x594980e0, 0x31f94a60
Caught exception in looking up space point ptr... 0x6cf27dc0, 0x6cf1a480
Caught exception in looking up space point ptr... 0x6cf1a480, 0x6cf27dc0
Caught exception in looking up space point ptr... 0x40722b50, 0x40716bd0
Caught exception in looking up space point ptr... 0x40716bd0, 0x40722830
Caught exception in looking up space point ptr... 0x9d3b710, 0x9d3b970
Caught exception in looking up space point ptr... 0x9d3b970, 0x9d3b710
Caught exception in looking up space point ptr... 0x7abc010, 0x7abcd20
Caught exception in looking up space point ptr... 0x7abcd20, 0x7abeea0
Caught exception in looking up space point ptr... 0x60459f40, 0x6045a790
Caught exception in looking up space point ptr... 0x6045a790, 0x6045a4b0
Caught exception in looking up space point ptr... 0x18d52e80, 0x3b418750
Caught exception in looking up space point ptr... 0x3a91f010, 0x18d52e80
Caught exception in looking up space point ptr... 0x360692f0, 0x36069160
Caught exception in looking up space point ptr... 0x36069160, 0x360692f0
Caught exception in looking up space point ptr... 0x317d5880, 0x5cb4e1e0
Caught exception in looking up space point ptr... 0x5cb4e1e0, 0x5cb4e330
Caught exception in looking up space point ptr... 0x5cb4cc90, 0x5cb4d0a0
Caught exception in looking up space point ptr... 0x5cb4d0a0, 0x5cb4d230
Caught exception in looking up space point ptr... 0x6cf20130, 0x6cf2d0b0
Caught exception in looking up space point ptr... 0x6cf2d0b0, 0x6cf20130
Caught exception in looking up space point ptr... 0x54e4bd00, 0x54e48d50
Caught exception in looking up space point ptr... 0x54e48d50, 0x54e4bd00
Caught exception in looking up space point ptr... 0x612a20c0, 0x61298310
Caught exception in looking up space point ptr... 0x1082d320, 0x612a20c0
Caught exception in looking up space point ptr... 0x5996b4c0, 0x59967db0
Caught exception in looking up space point ptr... 0x59967db0, 0x5996b4c0
Caught exception in looking up space point ptr... 0x4d2422c0, 0x6a3126d0
Caught exception in looking up space point ptr... 0x5b213d10, 0x4d2422c0
Caught exception in looking up space point ptr... 0x3913f100, 0x3913f420
Caught exception in looking up space point ptr... 0x3913f420, 0x3913f100
Caught exception in looking up space point ptr... 0x6def47e0, 0x6def49c0
Caught exception in looking up space point ptr... 0x6def49c0, 0x6def4600
Caught exception in looking up space point ptr... 0x5f9f9b80, 0x74bc6470
Caught exception in looking up space point ptr... 0x74bc6470, 0x5f9f9b80
Caught exception in looking up space point ptr... 0x122ceea0, 0x122ced30
Caught exception in looking up space point ptr... 0x122cbf10, 0x122ceea0
Caught exception in looking up space point ptr... 0x4586d080, 0x17e86880
Caught exception in looking up space point ptr... 0x6d6f6020, 0x4586d080
Caught exception in looking up space point ptr... 0x14261b50, 0x14262020
Caught exception in looking up space point ptr... 0x14262020, 0x14261b50
Caught exception in looking up space point ptr... 0x35ba5c00, 0x35ba0250
Caught exception in looking up space point ptr... 0x35ba0250, 0x35ba5950
++++>>>> total num hits: 86942, num free: 83782
04-Feb-2026 01:30:39 CET  Closed output file "np02vd_raw_run042433_0391_df-s02-d1_dw_0_20260203T211043_reco_stage0_20260204T003039_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         97.6114       108.352       129.936       107.649       6.53926        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.6019e-05    0.000146531   0.000719938   0.000111016   0.000149724      19     
produce:tpcrawdecoder:PDVDTPCReader                5.86387       6.20442       6.58292       6.21934      0.193632        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00941615     0.0207862     0.0492744     0.0170103     0.0111475       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000458656   0.000736812   0.00120799    0.000652366   0.000219718      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000406489   0.000618891   0.00120449    0.000564645   0.00019087       19     
produce:beamevent:BeamEvent                      8.4125e-05    0.000152524   0.00079492    0.000111987   0.000152848      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.78939       2.06792       2.47502       2.05702      0.161701        19     
produce:ophit:OpHitFinder                         0.0235073     0.0322108     0.0423559     0.0327547     0.0044301       19     
produce:wclsdatavd:WireCellToolkit                 74.9566       80.876        86.0701       80.679        3.33686        19     
produce:gaushit:GausHitFinder                     0.680976      0.941716       1.32224      0.919517      0.168319        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000168392   0.000229488   0.00050162    0.000212528   7.05294e-05      19     
produce:reco3d:SpacePointSolver                    7.3965        11.2192       15.5044       10.7459       2.00235        19     
produce:hitpdune:DisambigFromSpacePoints          0.0868283     0.152409      0.389104      0.138163      0.0645119       19     
produce:cluster3d:Cluster3D                       0.737874       2.58014       17.0527       1.76196       3.46882        19     
[art]:TriggerResults:TriggerResultInserter       2.2463e-05    3.85202e-05   0.000105207   3.4562e-05    1.77406e-05      19     
end_path:out1:RootOutput                          5.579e-06     9.218e-06    2.8112e-05     8.442e-06    4.73592e-06      19     
end_path:out1:RootOutput(write)                    3.6026        4.20696       5.88811       4.10324      0.534518        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5543.69 MB
  Peak resident set size usage (VmHWM): 3405.98 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0391_df-s02-d1_dw_0_20260203T211043_reco_stage0_20260204T003039_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0391_df-s02-d1_dw_0_20260203T211043_reco_stage0_20260204T003039_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0391_df-s02-d1_dw_0_20260203T211043_reco_stage0_20260204T003039_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162944.0,
    "core.end_time": 1770165040.0,
    "core.events": [
      44581,
      44587,
      44593,
      44599,
      44605,
      44611,
      44617,
      44623,
      44629,
      44635,
      44641,
      44647,
      44653,
      44659,
      44665,
      44671,
      44677,
      44683,
      44689
    ],
    "core.event_count": 19,
    "core.first_event_number": 44581,
    "core.last_event_number": 44689,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0391_df-s02-d1_dw_0_20260203T211043.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:45 UTC       justIN version: 01.06.00