justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.13@dunegpschedd01.fnal.gov

Jobsub ID303656.13@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce503
Last heartbeat2026-02-04 00:49:57
From worker nodeHostnameb9p02p6824.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:10:57
Input filesvd-protodune:np02vd_raw_run042421_0002_df-s04-d1_dw_0_20260203T133158.hdf5
JobscriptExit code0
Real time36m (2193s)
CPU time35m (2101s = 95%)
Max RSS bytes2621378560 (2499 MiB)
Outputting started2026-02-04 00:47:32
Output files
Finished2026-02-04 00:49:57
Saved logsjustin-logs:303656.13-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

n looking up space point ptr... 0x29beaa70, 0x29be8590
Caught exception in looking up space point ptr... 0x29be8590, 0x603c9cd0
Caught exception in looking up space point ptr... 0x603c9cd0, 0x5d3c95d0
Caught exception in looking up space point ptr... 0x5d3c95d0, 0x603c7360
Caught exception in looking up space point ptr... 0x40ce60b0, 0x3d407400
Caught exception in looking up space point ptr... 0x3d407400, 0x2a830710
Caught exception in looking up space point ptr... 0x2a830710, 0x845e1d0
Caught exception in looking up space point ptr... 0x44910090, 0x4490eff0
Caught exception in looking up space point ptr... 0x4490eff0, 0x44910090
Caught exception in looking up space point ptr... 0x58bf31a0, 0x58bf3e00
Caught exception in looking up space point ptr... 0x58bf3e00, 0x30450910
Caught exception in looking up space point ptr... 0x4bc5f5c0, 0x3ce15340
Caught exception in looking up space point ptr... 0x3ce15340, 0x4d83f7a0
Caught exception in looking up space point ptr... 0x4d83f7a0, 0x11b85340
Caught exception in looking up space point ptr... 0x11b85340, 0x49711be0
Caught exception in looking up space point ptr... 0x231abdf0, 0x3ff1f410
Caught exception in looking up space point ptr... 0x3ff1f410, 0x34d833b0
Caught exception in looking up space point ptr... 0x34d833b0, 0x2e9580f0
Caught exception in looking up space point ptr... 0x5594f270, 0x3510b980
Caught exception in looking up space point ptr... 0x3510b980, 0x40a4d7d0
Caught exception in looking up space point ptr... 0x602a6460, 0x602a6140
Caught exception in looking up space point ptr... 0x602a6140, 0x602a6460
Caught exception in looking up space point ptr... 0x59d58220, 0x59d5fed0
Caught exception in looking up space point ptr... 0x59d5fed0, 0x59d60dd0
Caught exception in looking up space point ptr... 0x3cb3f6d0, 0x46b964f0
Caught exception in looking up space point ptr... 0x46b964f0, 0x3cb3f6d0
Caught exception in looking up space point ptr... 0x3cb3f6d0, 0x46b93f50
Caught exception in looking up space point ptr... 0x46b93f50, 0x3cb3f6d0
Caught exception in looking up space point ptr... 0x3cb3f6d0, 0x46b90a80
Caught exception in looking up space point ptr... 0x46b90a80, 0x3cb3f6d0
Caught exception in looking up space point ptr... 0x3cb3f6d0, 0x46b96680
Caught exception in looking up space point ptr... 0x46b96680, 0x3cb3f6d0
Caught exception in looking up space point ptr... 0x28fa98c0, 0x4fdc9d90
Caught exception in looking up space point ptr... 0x4fdc9d90, 0x28fa98c0
Caught exception in looking up space point ptr... 0x47b93590, 0x47b93270
Caught exception in looking up space point ptr... 0x47b93270, 0x47b93590
Caught exception in looking up space point ptr... 0x5a212c60, 0x5a211bc0
Caught exception in looking up space point ptr... 0x5a211bc0, 0x5a212c60
Caught exception in looking up space point ptr... 0x602a46e0, 0x602a4550
Caught exception in looking up space point ptr... 0x602a4550, 0x602a46e0
Caught exception in looking up space point ptr... 0x2ec46060, 0x4eedfdd0
Caught exception in looking up space point ptr... 0x4eedfdd0, 0x239b2a20
Caught exception in looking up space point ptr... 0x4c5325e0, 0x26c3fcc0
Caught exception in looking up space point ptr... 0x26c3fdc0, 0x4c5325e0
Caught exception in looking up space point ptr... 0x29809ff0, 0x4c7f3470
Caught exception in looking up space point ptr... 0x4c7f3470, 0x58d53e00
Caught exception in looking up space point ptr... 0x26c552d0, 0x46ac0610
Caught exception in looking up space point ptr... 0x46ac0610, 0x50119f90
Caught exception in looking up space point ptr... 0x59d52110, 0x59d51e70
Caught exception in looking up space point ptr... 0x59d51e70, 0x59d51c00
Caught exception in looking up space point ptr... 0x59d51fc0, 0x59d52260
Caught exception in looking up space point ptr... 0x59d52260, 0x59d51fc0
Caught exception in looking up space point ptr... 0x62784230, 0x62782b20
Caught exception in looking up space point ptr... 0x62782b20, 0x62784190
Caught exception in looking up space point ptr... 0x59d52a80, 0x59d52f30
Caught exception in looking up space point ptr... 0x59d52f30, 0x59d52da0
Caught exception in looking up space point ptr... 0x59d54ed0, 0x59d53f10
Caught exception in looking up space point ptr... 0x59d53f10, 0x59d54ed0
Caught exception in looking up space point ptr... 0x5d3cb1f0, 0x5d2d6240
Caught exception in looking up space point ptr... 0x5d2d6240, 0x5d3cb1f0
Caught exception in looking up space point ptr... 0x3cb2b570, 0x46dbc410
Caught exception in looking up space point ptr... 0x46dbc410, 0x6029afd0
Caught exception in looking up space point ptr... 0x35196aa0, 0x8393bc0
Caught exception in looking up space point ptr... 0x8393bc0, 0x35196aa0
Caught exception in looking up space point ptr... 0x59d5cd70, 0x59d58b10
Caught exception in looking up space point ptr... 0x59d58b10, 0x59d5cd70
++++>>>> total num hits: 33920, num free: 32471
04-Feb-2026 01:47:23 CET  Closed output file "np02vd_raw_run042421_0002_df-s04-d1_dw_0_20260203T133158_reco_stage0_20260204T004723_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         65.4914       79.2543       90.1286       79.4623       6.05135        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       3.8871e-05    8.21463e-05   0.000220602    6.541e-05    4.57919e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.6686        4.34675       4.65329       4.37447      0.231106        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0111018     0.0156948     0.0492173     0.0137815    0.00712293       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000441203   0.000600879   0.00115985    0.000539375   0.00018075       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00164041     0.0021342    0.00348961    0.00205505    0.000376834      27     
produce:beamevent:BeamEvent                      6.0831e-05    0.000115828   0.000854597    8.466e-05    0.000146002      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.0988        2.75823       3.1954        2.75638      0.296296        27     
produce:ophit:OpHitFinder                         0.0250585     0.0354827     0.0439419     0.0353508    0.00494864       27     
produce:wclsdatavd:WireCellToolkit                 49.8544       57.4794       64.1377       56.9191       3.6182         27     
produce:gaushit:GausHitFinder                     0.418925      0.831415       1.16399      0.831833      0.238638        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000112361   0.000224149   0.000471174   0.000212881   7.27249e-05      27     
produce:reco3d:SpacePointSolver                    2.68235       8.60959       13.1273       9.3959        2.80962        27     
produce:hitpdune:DisambigFromSpacePoints          0.0291225     0.0827094     0.157765      0.0801185     0.0328288       27     
produce:cluster3d:Cluster3D                        0.48311       1.7683        5.68306       1.55157       1.24128        27     
[art]:TriggerResults:TriggerResultInserter        1.444e-05    2.55417e-05   8.7811e-05    2.1281e-05    1.52102e-05      27     
end_path:out1:RootOutput                          3.24e-06     8.11226e-06    3.109e-05      7.5e-06     5.36656e-06      27     
end_path:out1:RootOutput(write)                    2.73617       3.30489       4.37547       3.22688      0.406173        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4566.99 MB
  Peak resident set size usage (VmHWM): 2621.38 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0002_df-s04-d1_dw_0_20260203T133158_reco_stage0_20260204T004723_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0002_df-s04-d1_dw_0_20260203T133158_reco_stage0_20260204T004723_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0002_df-s04-d1_dw_0_20260203T133158_reco_stage0_20260204T004723_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163867.0,
    "core.end_time": 1770166045.0,
    "core.events": [
      329,
      335,
      341,
      347,
      353,
      359,
      365,
      371,
      377,
      383,
      389,
      395,
      401,
      407,
      413,
      419,
      425,
      431,
      437,
      443,
      449,
      455,
      461,
      467,
      473,
      479,
      485
    ],
    "core.event_count": 27,
    "core.first_event_number": 329,
    "core.last_event_number": 485,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0002_df-s04-d1_dw_0_20260203T133158.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:37:06 UTC       justIN version: 01.06.00