justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.12@dunegpschedd01.fnal.gov

Jobsub ID303653.12@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2026-02-04 00:16:19
From worker nodeHostnameb9p06p2822.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:36:16
Input filesvd-protodune:np02vd_raw_run042421_0137_df-s03-d0_dw_0_20260203T144507.hdf5
JobscriptExit code0
Real time37m (2242s)
CPU time35m (2153s = 96%)
Max RSS bytes2626048000 (2504 MiB)
Outputting started2026-02-04 00:13:40
Output files
Finished2026-02-04 00:16:09
Saved logsjustin-logs:303653.12-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ht exception in looking up space point ptr... 0x3bd5f150, 0x35e82340
Caught exception in looking up space point ptr... 0x3d8fa280, 0x3d8f9990
Caught exception in looking up space point ptr... 0x3d8f9990, 0x3d8fa280
Caught exception in looking up space point ptr... 0x3d8fa280, 0x3d8f9b20
Caught exception in looking up space point ptr... 0x3d8f9b20, 0x3d8fa280
Caught exception in looking up space point ptr... 0x52ea34d0, 0x52ea1070
Caught exception in looking up space point ptr... 0x52ea1070, 0x52ea34d0
Caught exception in looking up space point ptr... 0x52ea34d0, 0x52ea33b0
Caught exception in looking up space point ptr... 0x52ea33b0, 0x52ea34d0
Caught exception in looking up space point ptr... 0x3acbb7d0, 0x24e9f660
Caught exception in looking up space point ptr... 0x24e9f660, 0x3acbb7d0
Caught exception in looking up space point ptr... 0x310fc690, 0x310fc3f0
Caught exception in looking up space point ptr... 0x310fc3f0, 0x310fc690
Caught exception in looking up space point ptr... 0x5566e8a0, 0x31102c40
Caught exception in looking up space point ptr... 0x31102c40, 0x55688c70
Caught exception in looking up space point ptr... 0x4370d920, 0x541ada50
Caught exception in looking up space point ptr... 0x541ada50, 0x4370da20
Caught exception in looking up space point ptr... 0x42206080, 0x42208440
Caught exception in looking up space point ptr... 0x42208440, 0x42208bd0
Caught exception in looking up space point ptr... 0x23a7df30, 0x23a80940
Caught exception in looking up space point ptr... 0x23a80940, 0x23a7df30
Caught exception in looking up space point ptr... 0x125409c0, 0x264eab20
Caught exception in looking up space point ptr... 0x264eab20, 0x264ea8d0
Caught exception in looking up space point ptr... 0x36866a30, 0x368668e0
Caught exception in looking up space point ptr... 0x368668e0, 0x36866a30
Caught exception in looking up space point ptr... 0x2f9da290, 0x2f9d9f70
Caught exception in looking up space point ptr... 0x2f9d9f70, 0x336fcac0
Caught exception in looking up space point ptr... 0x55c90060, 0x375b85c0
Caught exception in looking up space point ptr... 0x375b85c0, 0x375bab60
Caught exception in looking up space point ptr... 0x4f797f30, 0x4f796e80
Caught exception in looking up space point ptr... 0x4f796e80, 0x4f797f30
Caught exception in looking up space point ptr... 0x473f29c0, 0x53153a10
Caught exception in looking up space point ptr... 0x53153a10, 0x473f29c0
Caught exception in looking up space point ptr... 0x5ef241e0, 0x5ef23ec0
Caught exception in looking up space point ptr... 0x5ef23ec0, 0x13d4a4f0
Caught exception in looking up space point ptr... 0x169a0380, 0x1699f500
Caught exception in looking up space point ptr... 0x1699f500, 0x169a6010
Caught exception in looking up space point ptr... 0x3cbe30e0, 0x4a9a5400
Caught exception in looking up space point ptr... 0x4a9a5400, 0x3cbe30e0
Caught exception in looking up space point ptr... 0x13d38ad0, 0x13d385d0
Caught exception in looking up space point ptr... 0x13d385d0, 0x13d38ad0
Caught exception in looking up space point ptr... 0x3cbdb890, 0x3cbe2d90
Caught exception in looking up space point ptr... 0x3cbe2d90, 0x3cbd7c60
Caught exception in looking up space point ptr... 0x2e4e6c50, 0x9fd7f60
Caught exception in looking up space point ptr... 0x9fd7f60, 0x2e4e6c50
Caught exception in looking up space point ptr... 0x13d378b0, 0x13d37470
Caught exception in looking up space point ptr... 0x13d37470, 0x13d37610
Caught exception in looking up space point ptr... 0x31106d30, 0x31100500
Caught exception in looking up space point ptr... 0x31100500, 0x31106d30
Caught exception in looking up space point ptr... 0x4a999410, 0x4a9a7c90
Caught exception in looking up space point ptr... 0x4a9a7c90, 0x4a999410
Caught exception in looking up space point ptr... 0x4a9a8030, 0x4a9b5b70
Caught exception in looking up space point ptr... 0x4a9b5b70, 0x4a9a8030
Caught exception in looking up space point ptr... 0x4a9a6880, 0x3cbebf00
Caught exception in looking up space point ptr... 0x3cbebf00, 0x4a9b82e0
Caught exception in looking up space point ptr... 0x4a99a650, 0x3cbe5890
Caught exception in looking up space point ptr... 0x3cbe5890, 0x3cbef570
Caught exception in looking up space point ptr... 0x1699fda0, 0x1699a560
Caught exception in looking up space point ptr... 0x1699a560, 0x1699fda0
Caught exception in looking up space point ptr... 0x4a99a7d0, 0x3cbecbd0
Caught exception in looking up space point ptr... 0x3cbecbd0, 0x4a9a13c0
Caught exception in looking up space point ptr... 0x4d8cfb70, 0x4d8ed6c0
Caught exception in looking up space point ptr... 0x4d8ed6c0, 0x4d8cfb70
Caught exception in looking up space point ptr... 0x30719460, 0x1685d000
Caught exception in looking up space point ptr... 0x1685d000, 0x30719460
++++>>>> total num hits: 95770, num free: 91365
04-Feb-2026 01:13:30 CET  Closed output file "np02vd_raw_run042421_0137_df-s03-d0_dw_0_20260203T144507_reco_stage0_20260204T001330_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         71.1863       80.8564       94.4563       81.9004       6.89988        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.277e-05    8.29729e-05   0.000256301    7.091e-05    4.48016e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.72124       4.39804       5.03244       4.36635      0.303824        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00909994     0.0198502     0.053985      0.0113285     0.0134319       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000367961   0.00047448    0.000773754   0.000455693   8.80264e-05      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0012933    0.00166175    0.00213263    0.00167324    0.000187014      27     
produce:beamevent:BeamEvent                       4.621e-05    0.000102426   0.000960204    6.741e-05    0.000169011      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.10428       2.73529       3.47197       2.74777      0.354574        27     
produce:ophit:OpHitFinder                         0.0242653     0.0335081     0.0411768     0.0341569    0.00522233       27     
produce:wclsdatavd:WireCellToolkit                 49.455        56.8032       69.8333       56.4294       5.63282        27     
produce:gaushit:GausHitFinder                     0.541688      0.874425       1.62952      0.815913      0.246983        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000183581   0.000246259   0.000375892   0.000232791   5.0309e-05       27     
produce:reco3d:SpacePointSolver                    6.75995       10.7652       15.4244       11.0998       2.33911        27     
produce:hitpdune:DisambigFromSpacePoints          0.0568549     0.0875632     0.160359      0.0786529     0.0268459       27     
produce:cluster3d:Cluster3D                       0.821995       1.75498       7.49316       1.39236       1.26554        27     
[art]:TriggerResults:TriggerResultInserter        1.564e-05    2.80357e-05    9.158e-05     2.336e-05    1.49977e-05      27     
end_path:out1:RootOutput                          3.16e-06     6.21896e-06    3.017e-05     5.26e-06     4.93116e-06      27     
end_path:out1:RootOutput(write)                    2.84896       3.36487       4.94883       3.14305      0.537855        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4631.29 MB
  Peak resident set size usage (VmHWM): 2626.05 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0137_df-s03-d0_dw_0_20260203T144507_reco_stage0_20260204T001330_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0137_df-s03-d0_dw_0_20260203T144507_reco_stage0_20260204T001330_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0137_df-s03-d0_dw_0_20260203T144507_reco_stage0_20260204T001330_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161785.0,
    "core.end_time": 1770164012.0,
    "core.events": [
      22268,
      22274,
      22280,
      22286,
      22292,
      22298,
      22304,
      22310,
      22316,
      22322,
      22328,
      22334,
      22340,
      22346,
      22352,
      22358,
      22364,
      22370,
      22376,
      22382,
      22388,
      22394,
      22400,
      22406,
      22412,
      22418,
      22424
    ],
    "core.event_count": 27,
    "core.first_event_number": 22268,
    "core.last_event_number": 22424,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0137_df-s03-d0_dw_0_20260203T144507.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:25 UTC       justIN version: 01.06.00