justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297916.66@dunegpschedd02.fnal.gov

Jobsub ID297916.66@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2026-02-03 23:21:33
From worker nodeHostnameb9p35p0031.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:10
Input filesvd-protodune:np02vd_raw_run042422_0093_df-s04-d1_dw_0_20260203T160503.hdf5
JobscriptExit code0
Real time32m (1932s)
CPU time30m (1832s = 94%)
Max RSS bytes2609762304 (2488 MiB)
Outputting started2026-02-03 23:19:23
Output files
Finished2026-02-03 23:21:33
Saved logsjustin-logs:297916.66-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ught exception in looking up space point ptr... 0x12a765b0, 0x33469320
Caught exception in looking up space point ptr... 0x10f92180, 0x10f8dd90
Caught exception in looking up space point ptr... 0x10f8dd90, 0x10f8e010
Caught exception in looking up space point ptr... 0x10f8e010, 0x10f987c0
Caught exception in looking up space point ptr... 0x10f987c0, 0x10f989a0
Caught exception in looking up space point ptr... 0x10f989a0, 0x10f98d10
Caught exception in looking up space point ptr... 0x10f98ea0, 0x10f92180
Caught exception in looking up space point ptr... 0x3852ed60, 0x2c256c20
Caught exception in looking up space point ptr... 0x2c256c20, 0x3852ed60
Caught exception in looking up space point ptr... 0x2c55a7e0, 0x2c55a980
Caught exception in looking up space point ptr... 0x2c55a980, 0x2c55ac70
Caught exception in looking up space point ptr... 0x2c55ac70, 0x2c55af60
Caught exception in looking up space point ptr... 0xbe33110, 0x2e27f910
Caught exception in looking up space point ptr... 0xbe3ea10, 0xbe33110
Caught exception in looking up space point ptr... 0x2c5582b0, 0x2c557e20
Caught exception in looking up space point ptr... 0x2c557e20, 0x2c557fc0
Caught exception in looking up space point ptr... 0x2c557fc0, 0x2c5585f0
Caught exception in looking up space point ptr... 0x4f8c66f0, 0x1614ce90
Caught exception in looking up space point ptr... 0x1614ce90, 0x4f8c66f0
Caught exception in looking up space point ptr... 0x2c5567f0, 0x2c556500
Caught exception in looking up space point ptr... 0x2c556500, 0x2c5566a0
Caught exception in looking up space point ptr... 0x2c5566a0, 0x2c556b30
Caught exception in looking up space point ptr... 0x3eef0840, 0x3eef0aa0
Caught exception in looking up space point ptr... 0x2a9fd180, 0x3eef0840
Caught exception in looking up space point ptr... 0x3b93e4f0, 0x50027500
Caught exception in looking up space point ptr... 0x50027500, 0x3b93e4f0
Caught exception in looking up space point ptr... 0x3b0b81a0, 0x3b0aa030
Caught exception in looking up space point ptr... 0x3b0aa030, 0x3b0b81a0
Caught exception in looking up space point ptr... 0x4dec7230, 0x4deab020
Caught exception in looking up space point ptr... 0x165bd140, 0x4dec7230
Caught exception in looking up space point ptr... 0x10f8d760, 0x10f8d550
Caught exception in looking up space point ptr... 0x10f8d550, 0x31d26f80
Caught exception in looking up space point ptr... 0x458288c0, 0x45825010
Caught exception in looking up space point ptr... 0x4582dbf0, 0x458288c0
Caught exception in looking up space point ptr... 0x59fe3270, 0x266ca480
Caught exception in looking up space point ptr... 0x266ca480, 0xf484d20
Caught exception in looking up space point ptr... 0x4eb887e0, 0x4eb88ca0
Caught exception in looking up space point ptr... 0x4eb88ca0, 0x4eb88000
Caught exception in looking up space point ptr... 0x4eb88260, 0x4eb887e0
Caught exception in looking up space point ptr... 0x3b27b840, 0x3d22b8d0
Caught exception in looking up space point ptr... 0x3d22b8d0, 0x3b27b970
Caught exception in looking up space point ptr... 0x45bfbf50, 0x45bfdc10
Caught exception in looking up space point ptr... 0x45bfdc10, 0x45bfd970
Caught exception in looking up space point ptr... 0x45bfd970, 0x10733a20
Caught exception in looking up space point ptr... 0x10733b70, 0x45bfbf50
Caught exception in looking up space point ptr... 0x4deb2600, 0x2bf52540
Caught exception in looking up space point ptr... 0x4deb24d0, 0x2c5508d0
Caught exception in looking up space point ptr... 0x2c5508d0, 0x2bf52690
Caught exception in looking up space point ptr... 0x2bf52690, 0x2bf527e0
Caught exception in looking up space point ptr... 0x2c550a60, 0x4deb2600
Caught exception in looking up space point ptr... 0x2a222de0, 0x2a222cb0
Caught exception in looking up space point ptr... 0x2a222cb0, 0x2a222de0
Caught exception in looking up space point ptr... 0x45bea020, 0x45be86e0
Caught exception in looking up space point ptr... 0x45be86e0, 0x45be0ee0
Caught exception in looking up space point ptr... 0x2c13dca0, 0x50e15720
Caught exception in looking up space point ptr... 0x50e15720, 0x2c13dba0
Caught exception in looking up space point ptr... 0x2c55bb20, 0x2c55b980
Caught exception in looking up space point ptr... 0x2c55b980, 0x2c55b690
Caught exception in looking up space point ptr... 0x165c66c0, 0x3cd555e0
Caught exception in looking up space point ptr... 0x3cd555e0, 0x3b0c5860
Caught exception in looking up space point ptr... 0x3b0c5860, 0x3cd54b00
Caught exception in looking up space point ptr... 0x33a77980, 0x33a77830
Caught exception in looking up space point ptr... 0x33a77830, 0x33a77980
Caught exception in looking up space point ptr... 0x3fab8d30, 0x50058440
Caught exception in looking up space point ptr... 0x50058440, 0x3fab8d30
++++>>>> total num hits: 59401, num free: 57380
04-Feb-2026 00:19:18 CET  Closed output file "np02vd_raw_run042422_0093_df-s04-d1_dw_0_20260203T160503_reco_stage0_20260203T231918_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                          61.38        69.4783       76.1294       69.5672       3.6295         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.403e-05    7.62404e-05   0.00021972     7.162e-05    2.96309e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.35586       4.97923       7.60495       4.79349      0.710623        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00920473     0.0209508     0.059305      0.0139238     0.0139181       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00037952    0.000615559   0.00162638    0.00056392    0.000251878      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0012954    0.00164804    0.00282905    0.00156642    0.000323123      27     
produce:beamevent:BeamEvent                        6.2e-05     0.000115026   0.00077504     8.826e-05    0.000130657      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.72429       2.10005       2.26837       2.15038      0.136679        27     
produce:ophit:OpHitFinder                         0.0252165     0.0331398      0.04178      0.0331086    0.00433045       27     
produce:wclsdatavd:WireCellToolkit                 44.8544       49.0522       55.3789       48.6015       2.76775        27     
produce:gaushit:GausHitFinder                     0.434059      0.768385       1.45565      0.739968      0.235252        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00011248    0.000186458   0.00045217    0.00017082    6.71095e-05      27     
produce:reco3d:SpacePointSolver                    3.52942       7.73128       13.3048       7.38562       2.47843        27     
produce:hitpdune:DisambigFromSpacePoints          0.0338778     0.0762453     0.126678      0.0704199     0.0259535       27     
produce:cluster3d:Cluster3D                       0.654797       1.30507       2.75151       1.15475      0.497829        27     
[art]:TriggerResults:TriggerResultInserter        2.27e-05     3.20767e-05    9.813e-05     2.94e-05     1.3487e-05       27     
end_path:out1:RootOutput                           4.6e-06      5.88e-06      2.266e-05     5.03e-06     3.33849e-06      27     
end_path:out1:RootOutput(write)                     2.532        3.38545       4.05575       3.36332      0.285905        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4588.62 MB
  Peak resident set size usage (VmHWM): 2609.76 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0093_df-s04-d1_dw_0_20260203T160503_reco_stage0_20260203T231918_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0093_df-s04-d1_dw_0_20260203T160503_reco_stage0_20260203T231918_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0093_df-s04-d1_dw_0_20260203T160503_reco_stage0_20260203T231918_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158839.0,
    "core.end_time": 1770160759.0,
    "core.events": [
      15083,
      15089,
      15095,
      15101,
      15107,
      15113,
      15119,
      15125,
      15131,
      15137,
      15143,
      15149,
      15155,
      15161,
      15167,
      15173,
      15179,
      15185,
      15191,
      15197,
      15203,
      15209,
      15215,
      15221,
      15227,
      15233,
      15239
    ],
    "core.event_count": 27,
    "core.first_event_number": 15083,
    "core.last_event_number": 15239,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0093_df-s04-d1_dw_0_20260203T160503.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:08:00 UTC       justIN version: 01.06.00