justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.92@dunegpschedd01.fnal.gov

Jobsub ID303647.92@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-03 23:23:40
From worker nodeHostnameb9p12p0201.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:27
Input filesvd-protodune:np02vd_raw_run042421_0104_df-s04-d1_dw_0_20260203T142716.hdf5
JobscriptExit code0
Real time33m (2028s)
CPU time32m (1945s = 95%)
Max RSS bytes2770153472 (2641 MiB)
Outputting started2026-02-03 23:21:17
Output files
Finished2026-02-03 23:23:40
Saved logsjustin-logs:303647.92-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ht exception in looking up space point ptr... 0x5dd4bc30, 0x129ff540
Caught exception in looking up space point ptr... 0x129ff540, 0x129fd7e0
Caught exception in looking up space point ptr... 0x27df1820, 0x27dd7420
Caught exception in looking up space point ptr... 0x27dd7420, 0x634ad8c0
Caught exception in looking up space point ptr... 0x634ad8c0, 0x63553e70
Caught exception in looking up space point ptr... 0x3a3ab0f0, 0x2bc1bee0
Caught exception in looking up space point ptr... 0x2bc1bee0, 0x3d0baff0
Caught exception in looking up space point ptr... 0x40a4fa50, 0x5af03020
Caught exception in looking up space point ptr... 0x54979970, 0x40a4fa50
Caught exception in looking up space point ptr... 0x26b06610, 0x2f50ae60
Caught exception in looking up space point ptr... 0x2f50ae60, 0x2f50afb0
Caught exception in looking up space point ptr... 0x12ac48c0, 0x4d9d2860
Caught exception in looking up space point ptr... 0x4d9d2860, 0x406933d0
Caught exception in looking up space point ptr... 0x87dfe20, 0x75a1e30
Caught exception in looking up space point ptr... 0x75a1e30, 0x87dfe20
Caught exception in looking up space point ptr... 0x2bc1aa10, 0x2bc18510
Caught exception in looking up space point ptr... 0x2bc18510, 0x5460f0c0
Caught exception in looking up space point ptr... 0x593434c0, 0x593431a0
Caught exception in looking up space point ptr... 0x593431a0, 0x59342e80
Caught exception in looking up space point ptr... 0x5465d430, 0x62b152a0
Caught exception in looking up space point ptr... 0x62b152a0, 0x2bc164b0
Caught exception in looking up space point ptr... 0x32171c50, 0x3a313ff0
Caught exception in looking up space point ptr... 0x3a313ff0, 0x3a3141d0
Caught exception in looking up space point ptr... 0x11665a60, 0x11665bf0
Caught exception in looking up space point ptr... 0x11665bf0, 0x50bb6600
Caught exception in looking up space point ptr... 0x2c091760, 0x2c03b790
Caught exception in looking up space point ptr... 0x2c03b790, 0x12486f90
Caught exception in looking up space point ptr... 0x468edcb0, 0x2bc1dcf0
Caught exception in looking up space point ptr... 0x2bc1dcf0, 0x468edcb0
Caught exception in looking up space point ptr... 0x2be04080, 0x2bc199e0
Caught exception in looking up space point ptr... 0x2bc199e0, 0x2c05ee60
Caught exception in looking up space point ptr... 0x42f6b030, 0x409c6990
Caught exception in looking up space point ptr... 0x409c6990, 0x10fb1620
Caught exception in looking up space point ptr... 0x129fed20, 0x5dd649d0
Caught exception in looking up space point ptr... 0x5dd649d0, 0x5dd64b60
Caught exception in looking up space point ptr... 0x20585520, 0x2c0e86a0
Caught exception in looking up space point ptr... 0x2c0e86a0, 0x20585520
Caught exception in looking up space point ptr... 0x11917510, 0x3dff1d30
Caught exception in looking up space point ptr... 0x3dff1d30, 0x11917510
Caught exception in looking up space point ptr... 0x3cb3bde0, 0x2c054500
Caught exception in looking up space point ptr... 0x2c054500, 0x2c051910
Caught exception in looking up space point ptr... 0x5dd68850, 0x5dd686c0
Caught exception in looking up space point ptr... 0x5dd686c0, 0x5dd68530
Caught exception in looking up space point ptr... 0x39f98420, 0x42fd4690
Caught exception in looking up space point ptr... 0x42fd4690, 0x39f98420
Caught exception in looking up space point ptr... 0x129de700, 0x5dd65970
Caught exception in looking up space point ptr... 0x5dd65970, 0x129de700
Caught exception in looking up space point ptr... 0x129f1de0, 0x5dd4b2d0
Caught exception in looking up space point ptr... 0x5dd4b2d0, 0x5dd4b460
Caught exception in looking up space point ptr... 0x1250ae20, 0x2c053030
Caught exception in looking up space point ptr... 0x2c053030, 0x2c054e40
Caught exception in looking up space point ptr... 0x5a1a9fe0, 0x265fd5e0
Caught exception in looking up space point ptr... 0x265fd5e0, 0x5a1a9fe0
Caught exception in looking up space point ptr... 0x409c6b60, 0x33034670
Caught exception in looking up space point ptr... 0x33034670, 0x42f67960
Caught exception in looking up space point ptr... 0x2bccb1e0, 0x2bccb050
Caught exception in looking up space point ptr... 0x2bccb050, 0x126555d0
Caught exception in looking up space point ptr... 0x126555d0, 0x12655440
Caught exception in looking up space point ptr... 0x12655440, 0x2bccaa10
Caught exception in looking up space point ptr... 0x2bccaa10, 0x2bccaba0
Caught exception in looking up space point ptr... 0x2bccaba0, 0x2bccb1e0
Caught exception in looking up space point ptr... 0x544f9a00, 0x27e0ad40
Caught exception in looking up space point ptr... 0x27e0ad40, 0x544f9a00
Caught exception in looking up space point ptr... 0x1251d200, 0x1243ae80
Caught exception in looking up space point ptr... 0x35311950, 0x1251d200
++++>>>> total num hits: 651766, num free: 634543
04-Feb-2026 00:21:10 CET  Closed output file "np02vd_raw_run042421_0104_df-s04-d1_dw_0_20260203T142716_reco_stage0_20260203T232110_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         60.9328       73.1057       144.657       69.2911       15.2231        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.035e-05    6.51702e-05   0.000154011   5.3791e-05    2.94453e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.08277       3.39565       3.8666        3.39748      0.224589        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00894746     0.0143869     0.0430265     0.0124475    0.00681542       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000346342   0.000599819    0.0050241    0.000437051   0.000869542      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00133065    0.00169449    0.00256621     0.001637     0.000267182      27     
produce:beamevent:BeamEvent                       5.123e-05    0.000107079   0.000932543    6.885e-05    0.000163559      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.88453       2.24225       2.8864        2.21652      0.258064        27     
produce:ophit:OpHitFinder                         0.0187777     0.0304783     0.0418707     0.0307915    0.00510409       27     
produce:wclsdatavd:WireCellToolkit                 46.8776       50.4367       61.6886       48.4099       4.57875        27     
produce:gaushit:GausHitFinder                     0.396221      0.760612       1.20861      0.714278      0.186847        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00012551    0.000204426   0.000496982   0.000193301   6.32652e-05      27     
produce:reco3d:SpacePointSolver                    4.34171       9.09092       16.5319       9.49585       2.35431        27     
produce:hitpdune:DisambigFromSpacePoints          0.0409221     0.0835901     0.293052      0.0660484     0.0473101       27     
produce:cluster3d:Cluster3D                       0.874742       3.90073       60.8008       1.4934        11.1972        27     
[art]:TriggerResults:TriggerResultInserter        1.421e-05    2.4116e-05     8.134e-05     2.205e-05    1.18198e-05      27     
end_path:out1:RootOutput                          2.96e-06     4.84189e-06    2.227e-05     3.84e-06     3.54935e-06      27     
end_path:out1:RootOutput(write)                    2.56195       3.13075       4.90352       2.90047      0.578553        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4803.79 MB
  Peak resident set size usage (VmHWM): 2770.15 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0104_df-s04-d1_dw_0_20260203T142716_reco_stage0_20260203T232110_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0104_df-s04-d1_dw_0_20260203T142716_reco_stage0_20260203T232110_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0104_df-s04-d1_dw_0_20260203T142716_reco_stage0_20260203T232110_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158856.0,
    "core.end_time": 1770160871.0,
    "core.events": [
      16919,
      16925,
      16931,
      16937,
      16943,
      16949,
      16955,
      16961,
      16967,
      16973,
      16979,
      16985,
      16991,
      16997,
      17003,
      17009,
      17015,
      17021,
      17027,
      17033,
      17039,
      17045,
      17051,
      17057,
      17063,
      17069,
      17075
    ],
    "core.event_count": 27,
    "core.first_event_number": 16919,
    "core.last_event_number": 17075,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0104_df-s04-d1_dw_0_20260203T142716.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:44:26 UTC       justIN version: 01.06.00