justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297917.102@dunegpschedd02.fnal.gov

Jobsub ID297917.102@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:25
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2026-02-03 23:55:32
From worker nodeHostnameb9p35p0031.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:21:50
Input filesvd-protodune:np02vd_raw_run042421_0053_df-s04-d0_dw_0_20260203T135942.hdf5
JobscriptExit code0
Real time31m (1895s)
CPU time29m (1764s = 93%)
Max RSS bytes2608365568 (2487 MiB)
Outputting started2026-02-03 23:53:27
Output files
Finished2026-02-03 23:55:32
Saved logsjustin-logs:297917.102-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

r... 0x5a6f9f50, 0x5a6fa1e0
Caught exception in looking up space point ptr... 0x5a6fa1e0, 0x55314380
Caught exception in looking up space point ptr... 0x55314380, 0x2439ed40
Caught exception in looking up space point ptr... 0x2439ed40, 0x5a6f9f50
Caught exception in looking up space point ptr... 0x27f27060, 0x27f3ccf0
Caught exception in looking up space point ptr... 0x27f3ccf0, 0x27f29e50
Caught exception in looking up space point ptr... 0x2af7ce30, 0x24239f40
Caught exception in looking up space point ptr... 0x24239f40, 0x20e86df0
Caught exception in looking up space point ptr... 0x24aad140, 0x24aad600
Caught exception in looking up space point ptr... 0x24aad600, 0x4fd9df50
Caught exception in looking up space point ptr... 0x38895530, 0x388961b0
Caught exception in looking up space point ptr... 0x388961b0, 0x38896db0
Caught exception in looking up space point ptr... 0x56ec61a0, 0x260e0650
Caught exception in looking up space point ptr... 0x260e0650, 0x2b63f8d0
Caught exception in looking up space point ptr... 0x4f27aa90, 0x4f27a530
Caught exception in looking up space point ptr... 0x4f27a530, 0x4f27a260
Caught exception in looking up space point ptr... 0x4f27a260, 0x4f279d40
Caught exception in looking up space point ptr... 0xdb39a50, 0x55b6d3f0
Caught exception in looking up space point ptr... 0xd3fc5c0, 0xdb39a50
Caught exception in looking up space point ptr... 0x24a9ae10, 0x38b8da00
Caught exception in looking up space point ptr... 0x38b8da00, 0x24a9ae10
Caught exception in looking up space point ptr... 0x243ed040, 0x3d87e710
Caught exception in looking up space point ptr... 0x3d87e710, 0x243ed360
Caught exception in looking up space point ptr... 0x4c02c7c0, 0x4c02a0c0
Caught exception in looking up space point ptr... 0x4c02a0c0, 0x4c02c7c0
Caught exception in looking up space point ptr... 0x4c02c7c0, 0x3c06ad10
Caught exception in looking up space point ptr... 0x3c06ad10, 0x4c02c7c0
Caught exception in looking up space point ptr... 0x4fd28480, 0x4fd171a0
Caught exception in looking up space point ptr... 0x4fd171a0, 0x4fd30170
Caught exception in looking up space point ptr... 0x24a75850, 0x3ba59a40
Caught exception in looking up space point ptr... 0x3ba59a40, 0x24a75850
Caught exception in looking up space point ptr... 0x24a75850, 0x3ba59680
Caught exception in looking up space point ptr... 0x3ba59680, 0x24a75850
Caught exception in looking up space point ptr... 0x4c033560, 0x4c033410
Caught exception in looking up space point ptr... 0x4c033410, 0x4c0332f0
Caught exception in looking up space point ptr... 0x4c0319b0, 0x4c033560
Caught exception in looking up space point ptr... 0x3017b0b0, 0x3017b3b0
Caught exception in looking up space point ptr... 0x3017b3b0, 0x3017b0b0
Caught exception in looking up space point ptr... 0x3017b0b0, 0x3017b1b0
Caught exception in looking up space point ptr... 0x3017b1b0, 0x3017b0b0
Caught exception in looking up space point ptr... 0x4e94ce70, 0xc6c6290
Caught exception in looking up space point ptr... 0xc6c6290, 0x38f7e500
Caught exception in looking up space point ptr... 0x5fa130d0, 0x5fa1b110
Caught exception in looking up space point ptr... 0x5fa1b110, 0x5fa13220
Caught exception in looking up space point ptr... 0x466565b0, 0x46656740
Caught exception in looking up space point ptr... 0x46656740, 0x466568d0
Caught exception in looking up space point ptr... 0x294d2e70, 0x294d32e0
Caught exception in looking up space point ptr... 0x294d32e0, 0x294d3150
Caught exception in looking up space point ptr... 0x53dd43b0, 0x53dd4540
Caught exception in looking up space point ptr... 0x53dd4540, 0x53dd43b0
Caught exception in looking up space point ptr... 0x5043bce0, 0x4c03d110
Caught exception in looking up space point ptr... 0x4c03d110, 0x29548970
Caught exception in looking up space point ptr... 0x2adb6040, 0x13b26d50
Caught exception in looking up space point ptr... 0x13b26d50, 0x2adb6040
Caught exception in looking up space point ptr... 0x4f3660a0, 0x2a9023c0
Caught exception in looking up space point ptr... 0x2a9023c0, 0x2a902230
Caught exception in looking up space point ptr... 0x4fd14e10, 0x4fc640f0
Caught exception in looking up space point ptr... 0x4fc640f0, 0x4fd14e10
Caught exception in looking up space point ptr... 0x2b192c40, 0x2b192ab0
Caught exception in looking up space point ptr... 0x2b192ab0, 0x2b192c40
Caught exception in looking up space point ptr... 0x2b190f30, 0x2b191250
Caught exception in looking up space point ptr... 0x2b191250, 0x2b190f30
Caught exception in looking up space point ptr... 0x4ff03ab0, 0x4ff07ff0
Caught exception in looking up space point ptr... 0x4ff07ff0, 0x4ff03ab0
Caught exception in looking up space point ptr... 0x2b191570, 0x2b1913e0
Caught exception in looking up space point ptr... 0x2b1913e0, 0x2b191570
++++>>>> total num hits: 79437, num free: 76239
04-Feb-2026 00:53:20 CET  Closed output file "np02vd_raw_run042421_0053_df-s04-d0_dw_0_20260203T135942_reco_stage0_20260203T235320_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         59.7117       68.0435       85.9744       65.8287       5.89036        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.639e-05    7.74978e-05   0.00021983     7.276e-05    2.87854e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.0931        6.00065       22.0249       4.41485       4.01107        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00959869     0.0632216      0.50178      0.0276373     0.108004        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0004415    0.00368122     0.0579583    0.00061649     0.0111793       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00129873    0.00179673    0.00271986    0.00173527    0.000377496      27     
produce:beamevent:BeamEvent                       6.33e-05     0.000119329   0.00077734     9.102e-05    0.000130132      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.6192        2.05723       2.83729       2.06487      0.234741        27     
produce:ophit:OpHitFinder                         0.0243148     0.0329294     0.0409744     0.033246      0.0038139       27     
produce:wclsdatavd:WireCellToolkit                 43.0236       47.1994       53.3121       46.492        2.54141        27     
produce:gaushit:GausHitFinder                     0.428923      0.686124      0.911996      0.693406      0.140078        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00011638    0.000185227   0.00073742    0.00014393    0.000119106      27     
produce:reco3d:SpacePointSolver                    4.77031       7.0983        11.163         6.699        1.62245        27     
produce:hitpdune:DisambigFromSpacePoints          0.0410296     0.074876      0.151507      0.0653482     0.0288955       27     
produce:cluster3d:Cluster3D                        0.63362       1.21652       2.62762       1.08902      0.464474        27     
[art]:TriggerResults:TriggerResultInserter        2.25e-05     3.35178e-05    9.758e-05       3e-05      1.39766e-05      27     
end_path:out1:RootOutput                          4.47e-06     6.04926e-06    2.246e-05     5.26e-06     3.27558e-06      27     
end_path:out1:RootOutput(write)                    2.71857       3.5808        4.79147       3.61246      0.462562        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4623.45 MB
  Peak resident set size usage (VmHWM): 2608.37 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0053_df-s04-d0_dw_0_20260203T135942_reco_stage0_20260203T235320_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0053_df-s04-d0_dw_0_20260203T135942_reco_stage0_20260203T235320_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0053_df-s04-d0_dw_0_20260203T135942_reco_stage0_20260203T235320_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160918.0,
    "core.end_time": 1770162801.0,
    "core.events": [
      8638,
      8644,
      8650,
      8656,
      8662,
      8668,
      8674,
      8680,
      8686,
      8692,
      8698,
      8704,
      8710,
      8716,
      8722,
      8728,
      8734,
      8740,
      8746,
      8752,
      8758,
      8764,
      8770,
      8776,
      8782,
      8788,
      8794
    ],
    "core.event_count": 27,
    "core.first_event_number": 8638,
    "core.last_event_number": 8794,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0053_df-s04-d0_dw_0_20260203T135942.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:21 UTC       justIN version: 01.06.00