justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.183@dunegpschedd01.fnal.gov

Jobsub ID303647.183@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2026-02-03 23:28:41
From worker nodeHostnameb9p06p2822.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:23
Input filesvd-protodune:np02vd_raw_run042421_0123_df-s03-d0_dw_0_20260203T143734.hdf5
JobscriptExit code0
Real time38m (2326s)
CPU time36m (2211s = 95%)
Max RSS bytes2608070656 (2487 MiB)
Outputting started2026-02-03 23:26:10
Output files
Finished2026-02-03 23:28:41
Saved logsjustin-logs:303647.183-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Caught exception in looking up space point ptr... 0x4ddf1dd0, 0x4ddf1930
Caught exception in looking up space point ptr... 0x290909f0, 0x4ddf1dd0
Caught exception in looking up space point ptr... 0x57d5dea0, 0x57d67bc0
Caught exception in looking up space point ptr... 0x57d67fc0, 0x57d5dea0
Caught exception in looking up space point ptr... 0x42acccb0, 0x42acd5e0
Caught exception in looking up space point ptr... 0x42acd5e0, 0x42ac92a0
Caught exception in looking up space point ptr... 0x119cbcd0, 0x119c5df0
Caught exception in looking up space point ptr... 0x119c5df0, 0x119c5cf0
Caught exception in looking up space point ptr... 0x298f91d0, 0x3855ca00
Caught exception in looking up space point ptr... 0x3855ca00, 0x298f4ce0
Caught exception in looking up space point ptr... 0x35c4dd80, 0x298f91d0
Caught exception in looking up space point ptr... 0x57d64b60, 0x57d636c0
Caught exception in looking up space point ptr... 0x57d636c0, 0x57d64b60
Caught exception in looking up space point ptr... 0x4dbafd90, 0x2b35d260
Caught exception in looking up space point ptr... 0x4462e390, 0x37f01580
Caught exception in looking up space point ptr... 0x37f01580, 0x29781c30
Caught exception in looking up space point ptr... 0x2b1cd560, 0x4dbafd90
Caught exception in looking up space point ptr... 0x3856d0a0, 0x3dcefc40
Caught exception in looking up space point ptr... 0x3dcefc40, 0x3856d0a0
Caught exception in looking up space point ptr... 0x267306c0, 0xed953b0
Caught exception in looking up space point ptr... 0x66ea2b80, 0x267306c0
Caught exception in looking up space point ptr... 0x10c225e0, 0x3738b740
Caught exception in looking up space point ptr... 0x3738b740, 0x66b526b0
Caught exception in looking up space point ptr... 0x59cab830, 0x27630440
Caught exception in looking up space point ptr... 0x27630440, 0x2e6c4b90
Caught exception in looking up space point ptr... 0x119c2db0, 0x119c1a70
Caught exception in looking up space point ptr... 0x119c5bf0, 0x119c2db0
Caught exception in looking up space point ptr... 0xd145e40, 0x2d6d0f90
Caught exception in looking up space point ptr... 0x2d6d0f90, 0xd145e40
Caught exception in looking up space point ptr... 0x2848d330, 0x59d006b0
Caught exception in looking up space point ptr... 0x59d006b0, 0x2d6dec00
Caught exception in looking up space point ptr... 0x32f24e60, 0x32f252e0
Caught exception in looking up space point ptr... 0x32f25ec0, 0x32f24e60
Caught exception in looking up space point ptr... 0x31590fb0, 0x31590f10
Caught exception in looking up space point ptr... 0x31590f10, 0x14d8c220
Caught exception in looking up space point ptr... 0x1541db80, 0xf0bfe80
Caught exception in looking up space point ptr... 0x3311d990, 0x1541db80
Caught exception in looking up space point ptr... 0x5b138c50, 0x5b138ef0
Caught exception in looking up space point ptr... 0x5b138ef0, 0x5b138c50
Caught exception in looking up space point ptr... 0x151d7360, 0x4c2830c0
Caught exception in looking up space point ptr... 0x4c2830c0, 0x151d7360
Caught exception in looking up space point ptr... 0x53c435a0, 0x70ae1f0
Caught exception in looking up space point ptr... 0x70ae1f0, 0x42866960
Caught exception in looking up space point ptr... 0x2d186730, 0x41641150
Caught exception in looking up space point ptr... 0x416402d0, 0x2d186730
Caught exception in looking up space point ptr... 0x119bbd00, 0x119bac80
Caught exception in looking up space point ptr... 0x119c3f80, 0x119bbd00
Caught exception in looking up space point ptr... 0x2aa652f0, 0x2aa64b80
Caught exception in looking up space point ptr... 0x2aa64b80, 0x2aa635a0
Caught exception in looking up space point ptr... 0x5b547170, 0x11cc63d0
Caught exception in looking up space point ptr... 0x11cc63d0, 0x3843bf70
Caught exception in looking up space point ptr... 0x12449fe0, 0x3bca00a0
Caught exception in looking up space point ptr... 0x3bca00a0, 0x498ed120
Caught exception in looking up space point ptr... 0x4f4d33a0, 0x59000e70
Caught exception in looking up space point ptr... 0x59000e70, 0x4f4d33a0
Caught exception in looking up space point ptr... 0x10427d70, 0x104278b0
Caught exception in looking up space point ptr... 0x104278b0, 0x1042f1e0
Caught exception in looking up space point ptr... 0x3e294e60, 0x4287bef0
Caught exception in looking up space point ptr... 0x4287bef0, 0x2491bb60
Caught exception in looking up space point ptr... 0x35c5f980, 0x35c5fa20
Caught exception in looking up space point ptr... 0x35c5fa20, 0x35c5f980
Caught exception in looking up space point ptr... 0x4c190cf0, 0x4c1910e0
Caught exception in looking up space point ptr... 0x4c1910e0, 0x4c190ea0
Caught exception in looking up space point ptr... 0x4be864d0, 0x4c9577d0
Caught exception in looking up space point ptr... 0x4c9577d0, 0x2972e010
++++>>>> total num hits: 69894, num free: 67241
04-Feb-2026 00:25:59 CET  Closed output file "np02vd_raw_run042421_0123_df-s03-d0_dw_0_20260203T143734_reco_stage0_20260203T232559_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         69.7926       83.9443       96.0801       82.7429       7.42952        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.767e-05    0.000129473   0.000931994    7.806e-05    0.000163249      27     
produce:tpcrawdecoder:PDVDTPCReader                3.61296       4.19603       7.76723       4.0444       0.726195        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0108202     0.0189606     0.0739942     0.012859      0.0146261       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000415082   0.000566533   0.00155345    0.000497683   0.000229461      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00152821    0.00192809    0.00388499    0.00181026    0.000425353      27     
produce:beamevent:BeamEvent                      6.2721e-05    0.000145359   0.00137403    9.3721e-05    0.000243693      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.31304       2.82062       3.49802       2.77887      0.307966        27     
produce:ophit:OpHitFinder                         0.0263083     0.0371666     0.0444991     0.0376532    0.00514297       27     
produce:wclsdatavd:WireCellToolkit                 52.2829       61.3796       72.9368       61.3112       4.83519        27     
produce:gaushit:GausHitFinder                     0.445929      0.878792       1.42993      0.821303      0.242063        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00014994    0.000259161   0.000478012   0.000236721   7.81281e-05      27     
produce:reco3d:SpacePointSolver                    3.03701       9.28352       14.9696       9.56613       2.89839        27     
produce:hitpdune:DisambigFromSpacePoints          0.0361248     0.0882262     0.154199      0.0904677     0.0297824       27     
produce:cluster3d:Cluster3D                       0.556993       1.62411        2.557        1.45445      0.554863        27     
[art]:TriggerResults:TriggerResultInserter        1.396e-05    2.87438e-05   0.00010895     2.419e-05    1.72378e-05      27     
end_path:out1:RootOutput                          3.32e-06     8.08748e-06    3.472e-05     6.31e-06     7.54796e-06      27     
end_path:out1:RootOutput(write)                    2.8163        3.59576       4.67739       3.37516      0.562174        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4574.9 MB
  Peak resident set size usage (VmHWM): 2608.07 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0123_df-s03-d0_dw_0_20260203T143734_reco_stage0_20260203T232559_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0123_df-s03-d0_dw_0_20260203T143734_reco_stage0_20260203T232559_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0123_df-s03-d0_dw_0_20260203T143734_reco_stage0_20260203T232559_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158854.0,
    "core.end_time": 1770161160.0,
    "core.events": [
      20000,
      20006,
      20012,
      20018,
      20024,
      20030,
      20036,
      20042,
      20048,
      20054,
      20060,
      20066,
      20072,
      20078,
      20084,
      20090,
      20096,
      20102,
      20108,
      20114,
      20120,
      20126,
      20132,
      20138,
      20144,
      20150,
      20156
    ],
    "core.event_count": 27,
    "core.first_event_number": 20000,
    "core.last_event_number": 20156,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0123_df-s03-d0_dw_0_20260203T143734.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:57:55 UTC       justIN version: 01.06.00