justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303651.105@dunegpschedd01.fnal.gov

Jobsub ID303651.105@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce509
Last heartbeat2026-02-03 23:54:10
From worker nodeHostnameb9p34p1022.cern.ch
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:21:04
Input filesvd-protodune:np02vd_raw_run042421_0005_df-s02-d1_dw_0_20260203T133335.hdf5
JobscriptExit code0
Real time30m (1854s)
CPU time29m (1772s = 95%)
Max RSS bytes2514591744 (2398 MiB)
Outputting started2026-02-03 23:51:59
Output files
Finished2026-02-03 23:54:10
Saved logsjustin-logs:303651.105-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

80
Caught exception in looking up space point ptr... 0x390b2130, 0x390a5f60
Caught exception in looking up space point ptr... 0x390a5f60, 0x390b24c0
Caught exception in looking up space point ptr... 0x5d0a0910, 0x5d0a0aa0
Caught exception in looking up space point ptr... 0x5d0a0aa0, 0x5d0a0910
Caught exception in looking up space point ptr... 0x408925f0, 0x40892740
Caught exception in looking up space point ptr... 0x40892740, 0x40892890
Caught exception in looking up space point ptr... 0x3dd63790, 0x3407ab40
Caught exception in looking up space point ptr... 0x3407ab40, 0x14e6ecf0
Caught exception in looking up space point ptr... 0x44163050, 0x2e46e130
Caught exception in looking up space point ptr... 0x2e46e130, 0x44163050
Caught exception in looking up space point ptr... 0x39f1e0f0, 0x39f1e960
Caught exception in looking up space point ptr... 0x39f2f490, 0x39f1e0f0
Caught exception in looking up space point ptr... 0x2b936a70, 0x4ba54950
Caught exception in looking up space point ptr... 0x4ba54950, 0x2b934dc0
Caught exception in looking up space point ptr... 0x4c167570, 0x408459b0
Caught exception in looking up space point ptr... 0x408459b0, 0x4c167570
Caught exception in looking up space point ptr... 0x52785cf0, 0x15242730
Caught exception in looking up space point ptr... 0x15242730, 0x52785cf0
Caught exception in looking up space point ptr... 0x2586cb90, 0x2586c820
Caught exception in looking up space point ptr... 0x2586c820, 0x2586cb90
Caught exception in looking up space point ptr... 0x86a0000, 0x869e5c0
Caught exception in looking up space point ptr... 0x869e5c0, 0x869ea20
Caught exception in looking up space point ptr... 0x22d36510, 0x22d36420
Caught exception in looking up space point ptr... 0x22d36420, 0x22d36510
Caught exception in looking up space point ptr... 0x76d5110, 0x13515cb0
Caught exception in looking up space point ptr... 0x13515cb0, 0x57a3c030
Caught exception in looking up space point ptr... 0x77d6560, 0x77cc7b0
Caught exception in looking up space point ptr... 0x77d1150, 0x77d6560
Caught exception in looking up space point ptr... 0xdafac00, 0x3aaeda60
Caught exception in looking up space point ptr... 0x3aaeda60, 0x28885250
Caught exception in looking up space point ptr... 0x55e46d50, 0x14750c50
Caught exception in looking up space point ptr... 0x14750c50, 0x55e46d50
Caught exception in looking up space point ptr... 0x49573d90, 0x49573f50
Caught exception in looking up space point ptr... 0x49573f50, 0x49573d90
Caught exception in looking up space point ptr... 0x147024b0, 0x57a72740
Caught exception in looking up space point ptr... 0x57a72740, 0x147024b0
Caught exception in looking up space point ptr... 0x4ba42eb0, 0x2e4af020
Caught exception in looking up space point ptr... 0x2e4af020, 0x4ba42eb0
Caught exception in looking up space point ptr... 0x77bfc90, 0x77c59e0
Caught exception in looking up space point ptr... 0x77c59e0, 0x77bfc90
Caught exception in looking up space point ptr... 0x2fffd080, 0x2fffa980
Caught exception in looking up space point ptr... 0x2fffa980, 0x228fe5e0
Caught exception in looking up space point ptr... 0x3bff7550, 0x2e4831a0
Caught exception in looking up space point ptr... 0x2e4831a0, 0x73c16f0
Caught exception in looking up space point ptr... 0xfe52f10, 0xfe52d80
Caught exception in looking up space point ptr... 0xfe52d80, 0xfe52a60
Caught exception in looking up space point ptr... 0x4461e170, 0x4461ddb0
Caught exception in looking up space point ptr... 0x4461ddb0, 0x4461e170
Caught exception in looking up space point ptr... 0x4bce98b0, 0x4bce9720
Caught exception in looking up space point ptr... 0x4bce9720, 0x4bce98b0
Caught exception in looking up space point ptr... 0x390c16f0, 0x3ab91720
Caught exception in looking up space point ptr... 0x390c2160, 0x390c16f0
Caught exception in looking up space point ptr... 0x34cc4af0, 0x22c2c680
Caught exception in looking up space point ptr... 0x22c2c680, 0x34cc4af0
Caught exception in looking up space point ptr... 0x797a7d0, 0x797aa10
Caught exception in looking up space point ptr... 0x797aa10, 0x797a7d0
Caught exception in looking up space point ptr... 0x3fe05b00, 0x4b6bcab0
Caught exception in looking up space point ptr... 0x4b6bcab0, 0x4b6bc510
Caught exception in looking up space point ptr... 0x39f263e0, 0x39f1dc60
Caught exception in looking up space point ptr... 0x39f1dc60, 0x39f263e0
Caught exception in looking up space point ptr... 0x3b34f1c0, 0x39f274e0
Caught exception in looking up space point ptr... 0x39f274e0, 0x3b34f1c0
Caught exception in looking up space point ptr... 0x44608360, 0x5edb07c0
Caught exception in looking up space point ptr... 0x5edb07c0, 0x44608700
Caught exception in looking up space point ptr... 0x2b941600, 0x135051d0
Caught exception in looking up space point ptr... 0x135051d0, 0x77d6690
++++>>>> total num hits: 72580, num free: 69763
04-Feb-2026 00:51:52 CET  Closed output file "np02vd_raw_run042421_0005_df-s02-d1_dw_0_20260203T133335_reco_stage0_20260203T235152_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         56.7133       66.5691       81.9012       65.9761       5.74651        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.052e-05    9.01266e-05   0.000239719   7.9839e-05    3.27379e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.7502        4.3024        8.38832       4.07614      0.827541        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0084753     0.0144076     0.0438654     0.0114181    0.00879676       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000412097   0.000561288   0.000820908   0.000543633   0.00011467       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00116255    0.00172078    0.00325336    0.00158672    0.000503624      27     
produce:beamevent:BeamEvent                      7.6144e-05    0.000136814   0.000924573   0.000109344   0.000155508      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.69784       2.04409       2.49534       2.07457      0.211512        27     
produce:ophit:OpHitFinder                         0.0238187     0.0334333     0.0407079     0.0328983     0.0040918       27     
produce:wclsdatavd:WireCellToolkit                 42.1458       47.0951       53.5756       46.4301       3.08634        27     
produce:gaushit:GausHitFinder                     0.319471      0.686499       1.04385      0.689298      0.190355        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000133139   0.00018489    0.000370815   0.000176945   4.71489e-05      27     
produce:reco3d:SpacePointSolver                    2.89068       7.80518       13.2394       6.94162       2.78727        27     
produce:hitpdune:DisambigFromSpacePoints          0.0306856     0.0771787     0.187714      0.0671566      0.03557        27     
produce:cluster3d:Cluster3D                       0.419681       1.31474       3.70814       1.22093      0.657763        27     
[art]:TriggerResults:TriggerResultInserter       2.7901e-05    4.35084e-05   0.000160901   3.4752e-05    2.84565e-05      27     
end_path:out1:RootOutput                          5.808e-06    8.93174e-06   2.5228e-05     8.372e-06    3.41212e-06      27     
end_path:out1:RootOutput(write)                    2.46659       3.16836       4.1977        2.97132       0.51663        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4641.12 MB
  Peak resident set size usage (VmHWM): 2514.59 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0005_df-s02-d1_dw_0_20260203T133335_reco_stage0_20260203T235152_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0005_df-s02-d1_dw_0_20260203T133335_reco_stage0_20260203T235152_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0005_df-s02-d1_dw_0_20260203T133335_reco_stage0_20260203T235152_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160873.0,
    "core.end_time": 1770162713.0,
    "core.events": [
      811,
      817,
      823,
      829,
      835,
      841,
      847,
      853,
      859,
      865,
      871,
      877,
      883,
      889,
      895,
      901,
      907,
      913,
      919,
      925,
      931,
      937,
      943,
      949,
      955,
      961,
      967
    ],
    "core.event_count": 27,
    "core.first_event_number": 811,
    "core.last_event_number": 967,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0005_df-s02-d1_dw_0_20260203T133335.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:06 UTC       justIN version: 01.06.00