justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297980.8@dunegpschedd02.fnal.gov

Jobsub ID297980.8@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 00:48:30
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2026-02-04 02:19:46
From worker nodeHostnameheplnc160.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:50:10
Input filesvd-protodune:np02vd_raw_run042421_0081_df-s03-d1_dw_0_20260203T141449.hdf5
JobscriptExit code0
Real time1h (5184s)
CPU time1h (4745s = 91%)
Max RSS bytes3730722816 (3557 MiB)
Outputting started2026-02-04 02:16:36
Output files
Finished2026-02-04 02:19:46
Saved logsjustin-logs:297980.8-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

2f0
Caught exception in looking up space point ptr... 0x2b6b62f0, 0x2b6b6480
Caught exception in looking up space point ptr... 0x72880b40, 0x2107a5b0
Caught exception in looking up space point ptr... 0x72880cd0, 0x72880b40
Caught exception in looking up space point ptr... 0x5862fab0, 0x4e261230
Caught exception in looking up space point ptr... 0x4e261230, 0x4e2610a0
Caught exception in looking up space point ptr... 0x78eb80a0, 0x44311ec0
Caught exception in looking up space point ptr... 0x44311ec0, 0x44311d30
Caught exception in looking up space point ptr... 0x155d41d0, 0x81822570
Caught exception in looking up space point ptr... 0x4b4ee4f0, 0x155d41d0
Caught exception in looking up space point ptr... 0x3c9bed20, 0x3c9bee00
Caught exception in looking up space point ptr... 0x3c9bee00, 0x3c9bed20
Caught exception in looking up space point ptr... 0x58373e00, 0x254fa790
Caught exception in looking up space point ptr... 0x254fa790, 0xdd11130
Caught exception in looking up space point ptr... 0x2bfa42c0, 0x2bf96cb0
Caught exception in looking up space point ptr... 0xe38db10, 0x2bfa42c0
Caught exception in looking up space point ptr... 0x2bfa3e10, 0x2bf96800
Caught exception in looking up space point ptr... 0xe38d660, 0x2bfa3e10
Caught exception in looking up space point ptr... 0x1590e140, 0xf0ba6d0
Caught exception in looking up space point ptr... 0xd1439d0, 0x1590e140
Caught exception in looking up space point ptr... 0x4f0d6760, 0x3917a480
Caught exception in looking up space point ptr... 0x3917a480, 0x3917a610
Caught exception in looking up space point ptr... 0x584beb90, 0x584bea00
Caught exception in looking up space point ptr... 0x584bea00, 0x31b24000
Caught exception in looking up space point ptr... 0x9071020, 0x9060040
Caught exception in looking up space point ptr... 0x9060040, 0x9070e90
Caught exception in looking up space point ptr... 0x12ce5b30, 0x743da620
Caught exception in looking up space point ptr... 0x743da620, 0x12ce5b30
Caught exception in looking up space point ptr... 0x34f516f0, 0x6c94c000
Caught exception in looking up space point ptr... 0x6c94c000, 0x34f51560
Caught exception in looking up space point ptr... 0x3724b030, 0x37238750
Caught exception in looking up space point ptr... 0x37238750, 0x3724aea0
Caught exception in looking up space point ptr... 0x29578970, 0x34b7f6d0
Caught exception in looking up space point ptr... 0x34b7f6d0, 0x4011e9b0
Caught exception in looking up space point ptr... 0x3ed18950, 0x3ed074c0
Caught exception in looking up space point ptr... 0x3ed074c0, 0x3ed187c0
Caught exception in looking up space point ptr... 0x1036ca90, 0x1036c7f0
Caught exception in looking up space point ptr... 0x1036c7f0, 0x1036c6a0
Caught exception in looking up space point ptr... 0x25c22c40, 0x40464620
Caught exception in looking up space point ptr... 0x40464620, 0x111e7010
Caught exception in looking up space point ptr... 0x111e7010, 0x4e5deec0
Caught exception in looking up space point ptr... 0x4e5deec0, 0x25c22c40
Caught exception in looking up space point ptr... 0xec113e0, 0x14f1c970
Caught exception in looking up space point ptr... 0x14f1c970, 0x14f1c500
Caught exception in looking up space point ptr... 0x73d5d1b0, 0x25b674e0
Caught exception in looking up space point ptr... 0x25b674e0, 0x73d5d1b0
Caught exception in looking up space point ptr... 0x9e48210, 0x72dc8960
Caught exception in looking up space point ptr... 0x72dc8960, 0x9e48210
Caught exception in looking up space point ptr... 0x35094280, 0x35093f90
Caught exception in looking up space point ptr... 0x35093f90, 0x35094130
Caught exception in looking up space point ptr... 0x73dbb060, 0x73dbb1b0
Caught exception in looking up space point ptr... 0x73dbb1b0, 0x73dbb060
Caught exception in looking up space point ptr... 0x3d582d30, 0x3d582bd0
Caught exception in looking up space point ptr... 0x3d582bd0, 0x45ae90e0
Caught exception in looking up space point ptr... 0x570bd210, 0x570bd3a0
Caught exception in looking up space point ptr... 0x570bd3a0, 0x570bd210
Caught exception in looking up space point ptr... 0x3956aa80, 0x20f705b0
Caught exception in looking up space point ptr... 0x20f705b0, 0x3956aa80
Caught exception in looking up space point ptr... 0x54533fe0, 0x54534490
Caught exception in looking up space point ptr... 0x54534490, 0x54534170
Caught exception in looking up space point ptr... 0x42874e60, 0x42874980
Caught exception in looking up space point ptr... 0x42874980, 0x42874cd0
Caught exception in looking up space point ptr... 0x151a3880, 0x151a36f0
Caught exception in looking up space point ptr... 0x151a36f0, 0x151a3880
Caught exception in looking up space point ptr... 0x23b1da50, 0x2d509a70
Caught exception in looking up space point ptr... 0x114682c0, 0x23b1da50
++++>>>> total num hits: 341618, num free: 334595
04-Feb-2026 02:16:25 GMT  Closed output file "np02vd_raw_run042421_0081_df-s03-d1_dw_0_20260203T141449_reco_stage0_20260204T021625_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         122.392       188.302       355.376       185.366       47.4733        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        7.463e-05    0.000140969   0.00024086    0.000121177   4.34059e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                11.5062       11.929        13.3491       11.9243      0.327706        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.528262      0.552206      0.598299      0.551275      0.0160281       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176776     0.0181068     0.0194725     0.0180999    0.000376805      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0872297     0.0907359      0.10623      0.0890268    0.00543338       27     
produce:beamevent:BeamEvent                      0.000120445   0.000258515   0.00147604    0.000194734   0.000245109      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.39483       4.13627       4.86938       4.18633      0.337575        27     
produce:ophit:OpHitFinder                         0.0362166     0.0473373     0.0577417     0.0484912    0.00552266       27     
produce:wclsdatavd:WireCellToolkit                 81.5569       92.6808       101.75        93.3357       5.48146        27     
produce:gaushit:GausHitFinder                     0.982566       2.06155       3.90326       2.05029      0.760637        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000339105   0.000548353   0.00135048    0.000489196   0.000229257      27     
produce:reco3d:SpacePointSolver                    6.17746       17.0975       34.3336       17.579        5.79046        27     
produce:hitpdune:DisambigFromSpacePoints          0.0753406     0.418222       1.20789      0.381595      0.276278        27     
produce:cluster3d:Cluster3D                        1.28624       52.9109       207.906       51.6797       41.6084        27     
[art]:TriggerResults:TriggerResultInserter       3.3423e-05    4.80499e-05   0.000129032   4.3351e-05    1.79007e-05      27     
end_path:out1:RootOutput                          5.71e-06     1.16454e-05    5.351e-05     1.028e-05    8.58207e-06      27     
end_path:out1:RootOutput(write)                    3.62543       6.30772       11.4782       6.42924       1.55068        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5874.96 MB
  Peak resident set size usage (VmHWM): 3730.72 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0081_df-s03-d1_dw_0_20260203T141449_reco_stage0_20260204T021625_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0081_df-s03-d1_dw_0_20260203T141449_reco_stage0_20260204T021625_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0081_df-s03-d1_dw_0_20260203T141449_reco_stage0_20260204T021625_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770166226.0,
    "core.end_time": 1770171386.0,
    "core.events": [
      13185,
      13191,
      13197,
      13203,
      13209,
      13215,
      13221,
      13227,
      13233,
      13239,
      13245,
      13251,
      13257,
      13263,
      13269,
      13275,
      13281,
      13287,
      13293,
      13299,
      13305,
      13311,
      13317,
      13323,
      13329,
      13335,
      13341
    ],
    "core.event_count": 27,
    "core.first_event_number": 13185,
    "core.last_event_number": 13341,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0081_df-s03-d1_dw_0_20260203T141449.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:29:43 UTC       justIN version: 01.06.00