justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303651.183@dunegpschedd01.fnal.gov

Jobsub ID303651.183@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-03 23:58:39
From worker nodeHostnameb9p12p0201.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:23:23
Input filesvd-protodune:np02vd_raw_run042421_0094_df-s03-d1_dw_0_20260203T142151.hdf5
JobscriptExit code0
Real time33m (1981s)
CPU time31m (1883s = 95%)
Max RSS bytes2567467008 (2448 MiB)
Outputting started2026-02-03 23:56:25
Output files
Finished2026-02-03 23:58:39
Saved logsjustin-logs:303651.183-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

57bced40
Caught exception in looking up space point ptr... 0x57bced40, 0x2c3c9f70
Caught exception in looking up space point ptr... 0x11e3c100, 0x1410e720
Caught exception in looking up space point ptr... 0x1410e720, 0x11e3c100
Caught exception in looking up space point ptr... 0x11e3c100, 0x2f4840a0
Caught exception in looking up space point ptr... 0x2f4840a0, 0x11e3c100
Caught exception in looking up space point ptr... 0x11e3c100, 0x3ed12ee0
Caught exception in looking up space point ptr... 0x3ed12ee0, 0x11e3c100
Caught exception in looking up space point ptr... 0x11e3c100, 0x332565c0
Caught exception in looking up space point ptr... 0x332565c0, 0x11e3c100
Caught exception in looking up space point ptr... 0x3ef13fc0, 0xdbb2410
Caught exception in looking up space point ptr... 0x61d84f60, 0x3ef13fc0
Caught exception in looking up space point ptr... 0x22c848c0, 0x246139c0
Caught exception in looking up space point ptr... 0x246138c0, 0x85b4390
Caught exception in looking up space point ptr... 0x85b4390, 0x22c83fb0
Caught exception in looking up space point ptr... 0x22c849f0, 0x22c848c0
Caught exception in looking up space point ptr... 0x3ee47460, 0x3ee47200
Caught exception in looking up space point ptr... 0x3ee47200, 0x3ee470d0
Caught exception in looking up space point ptr... 0x2989a1d0, 0x30c1db90
Caught exception in looking up space point ptr... 0x59ba4b10, 0x2989a1d0
Caught exception in looking up space point ptr... 0x2cb3c570, 0x2cb3bff0
Caught exception in looking up space point ptr... 0x2cb3cd30, 0x2cb3c570
Caught exception in looking up space point ptr... 0x50c1c240, 0x48d625d0
Caught exception in looking up space point ptr... 0x48d625d0, 0x50c1c240
Caught exception in looking up space point ptr... 0x2909dd60, 0x45e87f20
Caught exception in looking up space point ptr... 0x45e87f20, 0x1f7b4070
Caught exception in looking up space point ptr... 0x213b5260, 0x213b5440
Caught exception in looking up space point ptr... 0x213b5440, 0x213b8190
Caught exception in looking up space point ptr... 0x142ee560, 0x142ee600
Caught exception in looking up space point ptr... 0x142ee600, 0x142ee560
Caught exception in looking up space point ptr... 0x3aa22040, 0x3aa22140
Caught exception in looking up space point ptr... 0x3aa22140, 0x3de1ca00
Caught exception in looking up space point ptr... 0x5b27bbd0, 0x35fd7cf0
Caught exception in looking up space point ptr... 0x35fd7cf0, 0x35fd3e00
Caught exception in looking up space point ptr... 0x408fdf20, 0x4ff79410
Caught exception in looking up space point ptr... 0x4ff79410, 0x96571c0
Caught exception in looking up space point ptr... 0x375e0ee0, 0x2464b1d0
Caught exception in looking up space point ptr... 0x2464b1d0, 0x41b872e0
Caught exception in looking up space point ptr... 0x5b654c60, 0x3d3544e0
Caught exception in looking up space point ptr... 0x3d3544e0, 0x59bf71d0
Caught exception in looking up space point ptr... 0x4d4f9540, 0x263e72d0
Caught exception in looking up space point ptr... 0x263e72d0, 0x4d4f9540
Caught exception in looking up space point ptr... 0x4b17fd20, 0x41b19280
Caught exception in looking up space point ptr... 0x41b19280, 0x7f858d0
Caught exception in looking up space point ptr... 0x48d41250, 0x362af9a0
Caught exception in looking up space point ptr... 0x362af9a0, 0x48d41250
Caught exception in looking up space point ptr... 0x2a34bc40, 0xc70eb60
Caught exception in looking up space point ptr... 0xc70eb60, 0x2a34bc40
Caught exception in looking up space point ptr... 0x2a34bc40, 0x395c6cd0
Caught exception in looking up space point ptr... 0x395c6cd0, 0x2a34bc40
Caught exception in looking up space point ptr... 0xbeaeff0, 0xbe0ead0
Caught exception in looking up space point ptr... 0x24582880, 0xbeaeff0
Caught exception in looking up space point ptr... 0x894d190, 0x22bf5810
Caught exception in looking up space point ptr... 0x22bf5810, 0x30dced60
Caught exception in looking up space point ptr... 0x50485d70, 0x48833cf0
Caught exception in looking up space point ptr... 0x48833cf0, 0x50485d70
Caught exception in looking up space point ptr... 0x508f45f0, 0x508f4280
Caught exception in looking up space point ptr... 0x508f4280, 0x508f4460
Caught exception in looking up space point ptr... 0x7cb2b40, 0x28ee6340
Caught exception in looking up space point ptr... 0x55f37d90, 0x7cb2b40
Caught exception in looking up space point ptr... 0x1314c2e0, 0x1314bf70
Caught exception in looking up space point ptr... 0x1314bf70, 0x1314bde0
Caught exception in looking up space point ptr... 0x32f6f090, 0x9a17eb0
Caught exception in looking up space point ptr... 0x9a17eb0, 0x32f6f090
Caught exception in looking up space point ptr... 0x12376cb0, 0x59bba1c0
Caught exception in looking up space point ptr... 0x27662500, 0x12376cb0
++++>>>> total num hits: 51212, num free: 49130
04-Feb-2026 00:56:18 CET  Closed output file "np02vd_raw_run042421_0094_df-s03-d1_dw_0_20260203T142151_reco_stage0_20260203T235618_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         59.089        71.0315       83.9101       69.3314       6.77154        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.307e-05    7.7189e-05    0.00022468     6.438e-05    4.06257e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                 3.072        3.58148       4.14272       3.50644      0.270921        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00999369     0.0260133     0.105989      0.0145994     0.024186        27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000382421   0.000883019    0.0057981    0.000467521   0.00130096       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00154126    0.00208929    0.00550647     0.001867     0.00077202       27     
produce:beamevent:BeamEvent                       5.216e-05    0.000122782   0.00110413     6.833e-05    0.000196463      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.85576       2.12638       2.54137       2.08533      0.163021        27     
produce:ophit:OpHitFinder                         0.0193843     0.0306499     0.0367002     0.0314718    0.00352378       27     
produce:wclsdatavd:WireCellToolkit                 45.5778       50.6631       63.0471       48.8283       4.57303        27     
produce:gaushit:GausHitFinder                     0.313129      0.760825       1.3665       0.767819      0.176205        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000143381   0.000226033   0.000524922   0.000215491   6.52652e-05      27     
produce:reco3d:SpacePointSolver                    3.98456       9.21718       13.687        9.17846       2.42971        27     
produce:hitpdune:DisambigFromSpacePoints          0.0338503     0.0804801     0.151163      0.080549      0.0241127       27     
produce:cluster3d:Cluster3D                       0.346781       1.48976       3.23582       1.4553       0.606723        27     
[art]:TriggerResults:TriggerResultInserter        1.947e-05    2.95907e-05   9.3781e-05     2.794e-05    1.33265e-05      27     
end_path:out1:RootOutput                          3.48e-06     7.61519e-06    2.927e-05     7.16e-06     4.59847e-06      27     
end_path:out1:RootOutput(write)                    2.57773       3.03507       4.03053       2.92819      0.346211        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4525.01 MB
  Peak resident set size usage (VmHWM): 2567.47 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0094_df-s03-d1_dw_0_20260203T142151_reco_stage0_20260203T235618_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0094_df-s03-d1_dw_0_20260203T142151_reco_stage0_20260203T235618_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0094_df-s03-d1_dw_0_20260203T142151_reco_stage0_20260203T235618_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161017.0,
    "core.end_time": 1770162979.0,
    "core.events": [
      15291,
      15297,
      15303,
      15309,
      15315,
      15321,
      15327,
      15333,
      15339,
      15345,
      15351,
      15357,
      15363,
      15369,
      15375,
      15381,
      15387,
      15393,
      15399,
      15405,
      15411,
      15417,
      15423,
      15429,
      15435,
      15441,
      15447
    ],
    "core.event_count": 27,
    "core.first_event_number": 15291,
    "core.last_event_number": 15447,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0094_df-s03-d1_dw_0_20260203T142151.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:41 UTC       justIN version: 01.06.00