justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303651.125@dunegpschedd01.fnal.gov

Jobsub ID303651.125@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce509
Last heartbeat2026-02-03 23:56:33
From worker nodeHostnameb9p34p1022.cern.ch
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:21:44
Input filesvd-protodune:np02vd_raw_run042421_0132_df-s02-d1_dw_0_20260203T144223.hdf5
JobscriptExit code0
Real time32m (1956s)
CPU time31m (1880s = 96%)
Max RSS bytes2673025024 (2549 MiB)
Outputting started2026-02-03 23:54:22
Output files
Finished2026-02-03 23:56:33
Saved logsjustin-logs:303651.125-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

90, 0x375868c0
Caught exception in looking up space point ptr... 0x3ad79ae0, 0x32a26290
Caught exception in looking up space point ptr... 0xbf6e760, 0xbf6b900
Caught exception in looking up space point ptr... 0xbf6b900, 0xbf6e760
Caught exception in looking up space point ptr... 0xbf6e760, 0xbf6c680
Caught exception in looking up space point ptr... 0xbf6c680, 0xbf6e760
Caught exception in looking up space point ptr... 0x2a447920, 0x4839b050
Caught exception in looking up space point ptr... 0x2cb80ec0, 0x2a447920
Caught exception in looking up space point ptr... 0x115b1d70, 0x2c088ba0
Caught exception in looking up space point ptr... 0x2c088ba0, 0x115b1d70
Caught exception in looking up space point ptr... 0x47ba2e10, 0x47ba3440
Caught exception in looking up space point ptr... 0x47ba3440, 0x47ba06f0
Caught exception in looking up space point ptr... 0x410e76e0, 0x410e6360
Caught exception in looking up space point ptr... 0x410e6360, 0x410e76e0
Caught exception in looking up space point ptr... 0xbf68ca0, 0xbf68b10
Caught exception in looking up space point ptr... 0xbf68b10, 0xbf68ca0
Caught exception in looking up space point ptr... 0x426df410, 0x426df730
Caught exception in looking up space point ptr... 0x426df730, 0x426df410
Caught exception in looking up space point ptr... 0xe4d2f60, 0x312aeb90
Caught exception in looking up space point ptr... 0x312aeb90, 0xe4d2f60
Caught exception in looking up space point ptr... 0x3dc38940, 0x3dc3e460
Caught exception in looking up space point ptr... 0x3dc3e460, 0x3dc3eb20
Caught exception in looking up space point ptr... 0x14bf58c0, 0x14bf60e0
Caught exception in looking up space point ptr... 0x14bf7930, 0x14bf58c0
Caught exception in looking up space point ptr... 0x272bc550, 0x272c00b0
Caught exception in looking up space point ptr... 0x272c00b0, 0x272ba060
Caught exception in looking up space point ptr... 0x31367820, 0x34b53e60
Caught exception in looking up space point ptr... 0x34b53e60, 0x4263fdf0
Caught exception in looking up space point ptr... 0x30bd67e0, 0x30bd68e0
Caught exception in looking up space point ptr... 0x30bd68e0, 0x30bd67e0
Caught exception in looking up space point ptr... 0xbf641d0, 0xbf64030
Caught exception in looking up space point ptr... 0xbf64030, 0xbf641d0
Caught exception in looking up space point ptr... 0x2cc8d950, 0x2702feb0
Caught exception in looking up space point ptr... 0x5992a590, 0x2cc8d950
Caught exception in looking up space point ptr... 0x483c59b0, 0x483c3c00
Caught exception in looking up space point ptr... 0x483c3c00, 0x483c3d70
Caught exception in looking up space point ptr... 0x1570ab60, 0xe44dc20
Caught exception in looking up space point ptr... 0xe44dc20, 0x554480c0
Caught exception in looking up space point ptr... 0x609db250, 0x609db000
Caught exception in looking up space point ptr... 0x609db000, 0x609daf30
Caught exception in looking up space point ptr... 0x274bad60, 0x12985490
Caught exception in looking up space point ptr... 0x12985490, 0x274bad60
Caught exception in looking up space point ptr... 0x22b80950, 0x274bae90
Caught exception in looking up space point ptr... 0x489ad270, 0x22b80950
Caught exception in looking up space point ptr... 0x2caebad0, 0x2caebb70
Caught exception in looking up space point ptr... 0x2caebb70, 0x2caebad0
Caught exception in looking up space point ptr... 0x569d8870, 0x228427c0
Caught exception in looking up space point ptr... 0x228427c0, 0x569d8870
Caught exception in looking up space point ptr... 0x2db10b40, 0x21fea680
Caught exception in looking up space point ptr... 0x21fea680, 0x2db10b40
Caught exception in looking up space point ptr... 0x272b39c0, 0x272af5e0
Caught exception in looking up space point ptr... 0x272af5e0, 0x272b39c0
Caught exception in looking up space point ptr... 0x25dbfb70, 0x25dbf880
Caught exception in looking up space point ptr... 0x25dbf880, 0x25dbf760
Caught exception in looking up space point ptr... 0x410e4700, 0x410e5050
Caught exception in looking up space point ptr... 0x410e5050, 0x410e4700
Caught exception in looking up space point ptr... 0x3fd38df0, 0x3fd38d20
Caught exception in looking up space point ptr... 0x3fd38d20, 0x42801620
Caught exception in looking up space point ptr... 0x21585350, 0x345dad20
Caught exception in looking up space point ptr... 0x345dad20, 0x21585600
Caught exception in looking up space point ptr... 0x1730f1a0, 0x30e8bf10
Caught exception in looking up space point ptr... 0x30e8bf10, 0x5edacd90
Caught exception in looking up space point ptr... 0x2146d270, 0x2146bfc0
Caught exception in looking up space point ptr... 0x2146c9b0, 0x2146d270
Caught exception in looking up space point ptr... 0x15e15b50, 0x15e14ef0
Caught exception in looking up space point ptr... 0x15e14ef0, 0x15e15b50
++++>>>> total num hits: 41003, num free: 39502
04-Feb-2026 00:54:15 CET  Closed output file "np02vd_raw_run042421_0132_df-s02-d1_dw_0_20260203T144223_reco_stage0_20260203T235415_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         57.8367       70.3342       104.315       67.8545       13.3583        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.6509e-05    9.4201e-05    0.000217045   8.4116e-05    2.94101e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.62235       4.14172       4.95902       4.07651      0.360325        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0111843     0.0423475     0.0957584     0.0451176     0.0245909       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000390534    0.0036224     0.0267378    0.000660949   0.00618685       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00136359    0.00246154     0.0114249    0.00192562     0.001834        27     
produce:beamevent:BeamEvent                      8.6529e-05    0.000159031   0.00117703    0.000118468   0.000201579      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.68093       2.07109       2.80944       2.03494      0.269636        27     
produce:ophit:OpHitFinder                         0.0221022     0.0319291     0.0383376     0.0334628    0.00432461       27     
produce:wclsdatavd:WireCellToolkit                 42.7671       47.2644       54.5256       47.5292       3.06531        27     
produce:gaushit:GausHitFinder                     0.404353      0.688318       1.1623       0.692808       0.19019        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00012661    0.000210227   0.000812756   0.000189133   0.000125008      27     
produce:reco3d:SpacePointSolver                    3.54935       8.29484       15.5145       8.64853       2.99709        27     
produce:hitpdune:DisambigFromSpacePoints          0.030044      0.0807895     0.188951      0.0704745     0.036454        27     
produce:cluster3d:Cluster3D                       0.394345       4.34981       28.9643       1.22961       7.95527        27     
[art]:TriggerResults:TriggerResultInserter       2.4907e-05    4.03137e-05   0.000120561   3.3991e-05    1.85644e-05      27     
end_path:out1:RootOutput                          4.747e-06    7.33437e-06   2.5598e-05     6.33e-06     3.88343e-06      27     
end_path:out1:RootOutput(write)                    2.61942       3.33222       5.04141       3.13593      0.635715        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4782.1 MB
  Peak resident set size usage (VmHWM): 2673.03 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0132_df-s02-d1_dw_0_20260203T144223_reco_stage0_20260203T235415_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0132_df-s02-d1_dw_0_20260203T144223_reco_stage0_20260203T235415_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0132_df-s02-d1_dw_0_20260203T144223_reco_stage0_20260203T235415_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160913.0,
    "core.end_time": 1770162856.0,
    "core.events": [
      21445,
      21451,
      21457,
      21463,
      21469,
      21475,
      21481,
      21487,
      21493,
      21499,
      21505,
      21511,
      21517,
      21523,
      21529,
      21535,
      21541,
      21547,
      21553,
      21559,
      21565,
      21571,
      21577,
      21583,
      21589,
      21595,
      21601
    ],
    "core.event_count": 27,
    "core.first_event_number": 21445,
    "core.last_event_number": 21601,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0132_df-s02-d1_dw_0_20260203T144223.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:08:11 UTC       justIN version: 01.06.00