justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303651.182@dunegpschedd01.fnal.gov

Jobsub ID303651.182@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce508
Last heartbeat2026-02-04 00:09:48
From worker nodeHostnameb9p35p0043.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:23:25
Input filesvd-protodune:np02vd_raw_run042421_0108_df-s04-d0_dw_0_20260203T142926.hdf5
JobscriptExit code0
Real time43m (2638s)
CPU time42m (2533s = 96%)
Max RSS bytes4587548672 (4375 MiB)
Outputting started2026-02-04 00:07:25
Output files
Finished2026-02-04 00:09:48
Saved logsjustin-logs:303651.182-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e0, 0x371e43d0
Caught exception in looking up space point ptr... 0x371e43d0, 0x517655e0
Caught exception in looking up space point ptr... 0x517655e0, 0x55b84500
Caught exception in looking up space point ptr... 0x55b84500, 0x45df1ba0
Caught exception in looking up space point ptr... 0x45df1ba0, 0x4c313a20
Caught exception in looking up space point ptr... 0x4c313a20, 0x5f4137d0
Caught exception in looking up space point ptr... 0x5f4137d0, 0x372734e0
Caught exception in looking up space point ptr... 0x57736d00, 0x57736860
Caught exception in looking up space point ptr... 0x57736860, 0x577363c0
Caught exception in looking up space point ptr... 0x577363c0, 0x57735c80
Caught exception in looking up space point ptr... 0x28fde000, 0x459d2ff0
Caught exception in looking up space point ptr... 0x459d2ff0, 0x11480f90
Caught exception in looking up space point ptr... 0x35ddd740, 0x2545a960
Caught exception in looking up space point ptr... 0x2545a960, 0x35ddd740
Caught exception in looking up space point ptr... 0x35ddd740, 0x51291b90
Caught exception in looking up space point ptr... 0x51291b90, 0x35ddd740
Caught exception in looking up space point ptr... 0x2c4d3880, 0x2befa0c0
Caught exception in looking up space point ptr... 0x1410dea0, 0x2c4d3880
Caught exception in looking up space point ptr... 0xcfb3720, 0xcfb34c0
Caught exception in looking up space point ptr... 0x30385ca0, 0xcfb3720
Caught exception in looking up space point ptr... 0x4fa4fcc0, 0x25735540
Caught exception in looking up space point ptr... 0x25735540, 0x4fa4fcc0
Caught exception in looking up space point ptr... 0x4fa4fcc0, 0x3b41fdd0
Caught exception in looking up space point ptr... 0x3b41fdd0, 0x4fa4fcc0
Caught exception in looking up space point ptr... 0x3fc9e3c0, 0x3fc9e560
Caught exception in looking up space point ptr... 0x3fc9e560, 0x3fc9ede0
Caught exception in looking up space point ptr... 0x13dc31d0, 0x4c7e9860
Caught exception in looking up space point ptr... 0x5bb20070, 0x13dc31d0
Caught exception in looking up space point ptr... 0x33db88e0, 0x3d5d5ae0
Caught exception in looking up space point ptr... 0x3d5d5ae0, 0x4988dd80
Caught exception in looking up space point ptr... 0x3cb991d0, 0x88172b0
Caught exception in looking up space point ptr... 0x88172b0, 0x4117e900
Caught exception in looking up space point ptr... 0x3902d4f0, 0x3902d640
Caught exception in looking up space point ptr... 0x3902d640, 0x3902d4f0
Caught exception in looking up space point ptr... 0x606f8080, 0x25e972a0
Caught exception in looking up space point ptr... 0x69921090, 0x606f8080
Caught exception in looking up space point ptr... 0x40210120, 0x474f0670
Caught exception in looking up space point ptr... 0x474f0670, 0x3ec2f0f0
Caught exception in looking up space point ptr... 0x499b99f0, 0x35bd3540
Caught exception in looking up space point ptr... 0x35d9a7b0, 0x499b99f0
Caught exception in looking up space point ptr... 0x140e3890, 0x41d27870
Caught exception in looking up space point ptr... 0x41d27870, 0x56d883e0
Caught exception in looking up space point ptr... 0xc101540, 0xc101700
Caught exception in looking up space point ptr... 0xc101700, 0xc101540
Caught exception in looking up space point ptr... 0xc101540, 0x3a87db20
Caught exception in looking up space point ptr... 0x3a87db20, 0xc101540
Caught exception in looking up space point ptr... 0x21d259f0, 0x37a4d550
Caught exception in looking up space point ptr... 0x37a4d550, 0x21d260f0
Caught exception in looking up space point ptr... 0x6e0638e0, 0x3bf2f2b0
Caught exception in looking up space point ptr... 0x3bf2f2b0, 0x6e0638e0
Caught exception in looking up space point ptr... 0x6e0638e0, 0x459e6bb0
Caught exception in looking up space point ptr... 0x459e6bb0, 0x6e0638e0
Caught exception in looking up space point ptr... 0x86564f0, 0x8656960
Caught exception in looking up space point ptr... 0x8656960, 0x86564f0
Caught exception in looking up space point ptr... 0x625a3960, 0x3f081e70
Caught exception in looking up space point ptr... 0x3f081e70, 0x3f082050
Caught exception in looking up space point ptr... 0x955eb00, 0x261f64a0
Caught exception in looking up space point ptr... 0x261f64a0, 0x955eb00
Caught exception in looking up space point ptr... 0x282d3320, 0x487a3dc0
Caught exception in looking up space point ptr... 0x487a3dc0, 0x282d3320
Caught exception in looking up space point ptr... 0x3b42ec30, 0x16322fc0
Caught exception in looking up space point ptr... 0x16322fc0, 0x3b42ec30
Caught exception in looking up space point ptr... 0x8655160, 0xe680a90
Caught exception in looking up space point ptr... 0xe680a90, 0x8655160
Caught exception in looking up space point ptr... 0x47745ff0, 0x24ae17f0
Caught exception in looking up space point ptr... 0x24ae17f0, 0x2c4d3920
++++>>>> total num hits: 37558, num free: 36405
04-Feb-2026 01:07:18 CET  Closed output file "np02vd_raw_run042421_0108_df-s04-d0_dw_0_20260203T142926_reco_stage0_20260204T000718_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         62.8843       95.4931       277.118       73.7927       54.3089        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.674e-05    7.44152e-05   0.00014487     7.127e-05    1.64622e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.3124        4.89914       5.5127        4.83277      0.295304        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00827011     0.0219526     0.0761222     0.0129635     0.0180753       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0004316    0.00180415     0.0326393    0.00056742    0.00605025       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00122187    0.00180213    0.00312988    0.00161944    0.00049776       27     
produce:beamevent:BeamEvent                       6.662e-05    0.000114521   0.00080423     8.396e-05    0.000136075      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.89461       2.13767       2.38497       2.12342      0.135106        27     
produce:ophit:OpHitFinder                         0.024892      0.034235      0.0391496     0.0350036    0.00362863       27     
produce:wclsdatavd:WireCellToolkit                 44.7582       52.2174       59.6626       52.363        3.73641        27     
produce:gaushit:GausHitFinder                     0.479004       1.11886       2.85725        0.806       0.690109        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00012686    0.000205019   0.00045163    0.00017229    8.1099e-05       27     
produce:reco3d:SpacePointSolver                    4.51573       9.4482        24.1059       7.4024        4.96075        27     
produce:hitpdune:DisambigFromSpacePoints          0.042519      0.182867      0.767677      0.0870987     0.220207        27     
produce:cluster3d:Cluster3D                       0.609699       21.1803       179.995       1.31226       47.6718        27     
[art]:TriggerResults:TriggerResultInserter        2.374e-05    4.00511e-05    9.024e-05     3.686e-05    1.4358e-05       27     
end_path:out1:RootOutput                          4.15e-06     7.21519e-06    3.653e-05     5.63e-06     5.99668e-06      27     
end_path:out1:RootOutput(write)                    2.91183       4.2153        7.82828       3.68277       1.37622        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6700.59 MB
  Peak resident set size usage (VmHWM): 4587.55 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0108_df-s04-d0_dw_0_20260203T142926_reco_stage0_20260204T000718_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0108_df-s04-d0_dw_0_20260203T142926_reco_stage0_20260204T000718_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0108_df-s04-d0_dw_0_20260203T142926_reco_stage0_20260204T000718_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161013.0,
    "core.end_time": 1770163639.0,
    "core.events": [
      17560,
      17566,
      17572,
      17578,
      17584,
      17590,
      17596,
      17602,
      17608,
      17614,
      17620,
      17626,
      17632,
      17638,
      17644,
      17650,
      17656,
      17662,
      17668,
      17674,
      17680,
      17686,
      17692,
      17698,
      17704,
      17710,
      17716
    ],
    "core.event_count": 27,
    "core.first_event_number": 17560,
    "core.last_event_number": 17716,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0108_df-s04-d0_dw_0_20260203T142926.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:37:04 UTC       justIN version: 01.06.00