justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.123@dunegpschedd01.fnal.gov

Jobsub ID303652.123@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce503
Last heartbeat2026-02-04 00:25:52
From worker nodeHostnameb9p05p0564.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:28:39
Input filesvd-protodune:np02vd_raw_run042433_0600_df-s03-d1_dw_0_20260203T223009.hdf5
JobscriptExit code0
Real time54m (3250s)
CPU time52m (3143s = 96%)
Max RSS bytes3624452096 (3456 MiB)
Outputting started2026-02-04 00:22:51
Output files
Finished2026-02-04 00:25:52
Saved logsjustin-logs:303652.123-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x42d241b0, 0x42d28130
Caught exception in looking up space point ptr... 0x18e17f00, 0x164e4670
Caught exception in looking up space point ptr... 0x164e4670, 0x164e7ba0
Caught exception in looking up space point ptr... 0x39bb2c60, 0x32b4c7e0
Caught exception in looking up space point ptr... 0x32b4c7e0, 0x62429ce0
Caught exception in looking up space point ptr... 0x3278db40, 0x327900e0
Caught exception in looking up space point ptr... 0x327900e0, 0x3278ffb0
Caught exception in looking up space point ptr... 0x3278ffb0, 0x60794520
Caught exception in looking up space point ptr... 0x51539fa0, 0x3278db40
Caught exception in looking up space point ptr... 0x43745b60, 0x43745cf0
Caught exception in looking up space point ptr... 0x437343b0, 0x43745b60
Caught exception in looking up space point ptr... 0x818beac0, 0xae232f0
Caught exception in looking up space point ptr... 0xae232f0, 0x818beac0
Caught exception in looking up space point ptr... 0x818beac0, 0x818be730
Caught exception in looking up space point ptr... 0x818be730, 0x818beac0
Caught exception in looking up space point ptr... 0xfa35950, 0x6aaa3d50
Caught exception in looking up space point ptr... 0x6aaa3d50, 0x6078e920
Caught exception in looking up space point ptr... 0x6078e920, 0xfa35860
Caught exception in looking up space point ptr... 0xfa35860, 0x35dc17a0
Caught exception in looking up space point ptr... 0x35dc1700, 0x3474d7e0
Caught exception in looking up space point ptr... 0xfa26490, 0x35dc1700
Caught exception in looking up space point ptr... 0x47cc9b00, 0x101789d0
Caught exception in looking up space point ptr... 0x43ac2540, 0x47cc9b00
Caught exception in looking up space point ptr... 0x4e97d0d0, 0x4e97c8b0
Caught exception in looking up space point ptr... 0x4e97c8b0, 0x4e97ce80
Caught exception in looking up space point ptr... 0x4d5ad0a0, 0x7c7ed8c0
Caught exception in looking up space point ptr... 0x7c7ed8c0, 0x4d5ad0a0
Caught exception in looking up space point ptr... 0x4ea3de20, 0x4ea2b9f0
Caught exception in looking up space point ptr... 0x4ea2b9f0, 0x4ea1b9b0
Caught exception in looking up space point ptr... 0x888d47c0, 0x6851a5c0
Caught exception in looking up space point ptr... 0x888d4950, 0x888d47c0
Caught exception in looking up space point ptr... 0x42e8c730, 0x14b719e0
Caught exception in looking up space point ptr... 0x42e8c810, 0x42e8c730
Caught exception in looking up space point ptr... 0x67a35ff0, 0x67a35e60
Caught exception in looking up space point ptr... 0x67a35e60, 0x37c03480
Caught exception in looking up space point ptr... 0x81b0cb20, 0x6a7baa90
Caught exception in looking up space point ptr... 0x6a7baa90, 0x6a7bac20
Caught exception in looking up space point ptr... 0x136b2fd0, 0x6bf3a310
Caught exception in looking up space point ptr... 0x61ff55d0, 0x136b2fd0
Caught exception in looking up space point ptr... 0x33be3140, 0x18dfe420
Caught exception in looking up space point ptr... 0x18dfe420, 0x6176b5c0
Caught exception in looking up space point ptr... 0x62d2f690, 0x62d1f4c0
Caught exception in looking up space point ptr... 0x62d1f4c0, 0x62d2f500
Caught exception in looking up space point ptr... 0x3244c890, 0x3245ee50
Caught exception in looking up space point ptr... 0x3245ee50, 0x32471410
Caught exception in looking up space point ptr... 0x60818410, 0x60806ad0
Caught exception in looking up space point ptr... 0x60806ad0, 0x833d8db0
Caught exception in looking up space point ptr... 0x485c9280, 0x485c9910
Caught exception in looking up space point ptr... 0x485c9910, 0x485c9280
Caught exception in looking up space point ptr... 0x485c9280, 0x485c9410
Caught exception in looking up space point ptr... 0x485c9410, 0x485c9280
Caught exception in looking up space point ptr... 0x888ce190, 0x74356250
Caught exception in looking up space point ptr... 0x2b2be110, 0x888ce190
Caught exception in looking up space point ptr... 0x40730d80, 0x87363b0
Caught exception in looking up space point ptr... 0x60fdc3b0, 0x40730d80
Caught exception in looking up space point ptr... 0x3592db60, 0x51626280
Caught exception in looking up space point ptr... 0x51626280, 0x3592db60
Caught exception in looking up space point ptr... 0x6ac2c450, 0x3daed3a0
Caught exception in looking up space point ptr... 0x3daed3a0, 0x6d6da530
Caught exception in looking up space point ptr... 0x6630e980, 0x3dd86040
Caught exception in looking up space point ptr... 0x3dd86040, 0x12bb15e0
Caught exception in looking up space point ptr... 0x461aa580, 0x1362eff0
Caught exception in looking up space point ptr... 0x1362eff0, 0x461aa580
Caught exception in looking up space point ptr... 0x597acc20, 0x7c177710
Caught exception in looking up space point ptr... 0x597aca80, 0x597acc20
Caught exception in looking up space point ptr... 0x4e973f00, 0x3b5ba640
Caught exception in looking up space point ptr... 0x3b5ba640, 0x40828c90
++++>>>> total num hits: 473148, num free: 463684
04-Feb-2026 01:22:41 CET  Closed output file "np02vd_raw_run042433_0600_df-s03-d1_dw_0_20260203T223009_reco_stage0_20260204T002241_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         131.413       168.158       197.403       168.383       17.1756        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.3581e-05    0.000130776   0.000254283   0.000119231   4.54836e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.65976       6.7359        7.40926       6.76521      0.389921        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0107347     0.0211348     0.0514798     0.0135537     0.0128378       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000412565   0.000579449   0.00114488    0.000534217   0.000169295      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000385815   0.000498725   0.000716729   0.000451735   9.67382e-05      19     
produce:beamevent:BeamEvent                      6.3811e-05    0.000149646   0.000962851   9.8512e-05    0.000192975      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.06204       2.65784       4.04993       2.51266      0.496273        19     
produce:ophit:OpHitFinder                         0.0219852     0.0337018     0.0422909     0.0334228    0.00514123       19     
produce:wclsdatavd:WireCellToolkit                 89.1773       98.2564       110.285       96.586        5.72563        19     
produce:gaushit:GausHitFinder                      1.04704       1.83769       3.06962       1.85955      0.423817        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000233923   0.000374955   0.000547086   0.000355364   7.39639e-05      19     
produce:reco3d:SpacePointSolver                    7.52455       18.0365       30.2186       17.7341       4.75676        19     
produce:hitpdune:DisambigFromSpacePoints          0.138453      0.244682      0.378947      0.232778      0.0587428       19     
produce:cluster3d:Cluster3D                        1.8727        34.7838       50.0877       41.1507       16.3697        19     
[art]:TriggerResults:TriggerResultInserter        2.449e-05    3.24676e-05   7.9421e-05     3.013e-05    1.14323e-05      19     
end_path:out1:RootOutput                          5.77e-06     8.71479e-06    2.719e-05     7.98e-06     4.53575e-06      19     
end_path:out1:RootOutput(write)                    4.61606       5.51999       7.46462       5.44155      0.622614        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5653.7 MB
  Peak resident set size usage (VmHWM): 3624.45 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0600_df-s03-d1_dw_0_20260203T223009_reco_stage0_20260204T002241_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0600_df-s03-d1_dw_0_20260203T223009_reco_stage0_20260204T002241_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0600_df-s03-d1_dw_0_20260203T223009_reco_stage0_20260204T002241_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161330.0,
    "core.end_time": 1770164562.0,
    "core.events": [
      68409,
      68415,
      68421,
      68427,
      68433,
      68439,
      68445,
      68451,
      68457,
      68463,
      68469,
      68475,
      68481,
      68487,
      68493,
      68499,
      68505,
      68511,
      68517
    ],
    "core.event_count": 19,
    "core.first_event_number": 68409,
    "core.last_event_number": 68517,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0600_df-s03-d1_dw_0_20260203T223009.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:35 UTC       justIN version: 01.06.00