justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.28@dunegpschedd01.fnal.gov

Jobsub ID303656.28@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat2026-02-04 01:06:42
From worker nodeHostnameheplnc147.pp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:11:25
Input filesvd-protodune:np02vd_raw_run042421_0068_df-s03-d1_dw_0_20260203T140748.hdf5
JobscriptExit code0
Real time52m (3173s)
CPU time45m (2712s = 85%)
Max RSS bytes2571767808 (2452 MiB)
Outputting started2026-02-04 01:04:20
Output files
Finished2026-02-04 01:06:42
Saved logsjustin-logs:303656.28-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

b20, 0x43babad0
Caught exception in looking up space point ptr... 0x404321a0, 0x4bc69b20
Caught exception in looking up space point ptr... 0x4901d640, 0x4901d190
Caught exception in looking up space point ptr... 0x4901d190, 0x4901d640
Caught exception in looking up space point ptr... 0x4901d640, 0x4901cef0
Caught exception in looking up space point ptr... 0x4901cef0, 0x4901d640
Caught exception in looking up space point ptr... 0x4901d640, 0x4901d320
Caught exception in looking up space point ptr... 0x4901d320, 0x4901d640
Caught exception in looking up space point ptr... 0x4901d640, 0x4901d4b0
Caught exception in looking up space point ptr... 0x4901d4b0, 0x4901d640
Caught exception in looking up space point ptr... 0x2893b290, 0x2893af70
Caught exception in looking up space point ptr... 0x2893af70, 0x2893ae10
Caught exception in looking up space point ptr... 0x60f45700, 0x60f45850
Caught exception in looking up space point ptr... 0x60f45850, 0x2893b290
Caught exception in looking up space point ptr... 0x60ef41c0, 0xe99fd20
Caught exception in looking up space point ptr... 0xe99fd20, 0x39d9ef20
Caught exception in looking up space point ptr... 0x35701120, 0xf4a3df0
Caught exception in looking up space point ptr... 0xf4a3df0, 0x35701320
Caught exception in looking up space point ptr... 0x344f7cc0, 0x344f7b30
Caught exception in looking up space point ptr... 0x240cc8b0, 0x344f7cc0
Caught exception in looking up space point ptr... 0x1468fe30, 0x6214c880
Caught exception in looking up space point ptr... 0x6214c880, 0x6214ca10
Caught exception in looking up space point ptr... 0x6214ca10, 0x6214cba0
Caught exception in looking up space point ptr... 0x47ad4d60, 0x26e9edb0
Caught exception in looking up space point ptr... 0x26e9edb0, 0x47ad4e60
Caught exception in looking up space point ptr... 0xd62d4e0, 0xd62c720
Caught exception in looking up space point ptr... 0x358163c0, 0xd62d4e0
Caught exception in looking up space point ptr... 0xda29360, 0x85aea50
Caught exception in looking up space point ptr... 0x85aea50, 0xd9ca920
Caught exception in looking up space point ptr... 0x3b571d40, 0x3b570630
Caught exception in looking up space point ptr... 0x3b570630, 0x3b574c30
Caught exception in looking up space point ptr... 0x29cffc50, 0xdbae8b0
Caught exception in looking up space point ptr... 0xdbae8b0, 0x29cffc50
Caught exception in looking up space point ptr... 0x12039a80, 0x4e194790
Caught exception in looking up space point ptr... 0x25ad5f80, 0x12039a80
Caught exception in looking up space point ptr... 0x52729f50, 0x28613480
Caught exception in looking up space point ptr... 0x28613480, 0x52729f50
Caught exception in looking up space point ptr... 0x40a9d690, 0x40a9d590
Caught exception in looking up space point ptr... 0x411986e0, 0x40a9d690
Caught exception in looking up space point ptr... 0x4a6bcdd0, 0x395a0bf0
Caught exception in looking up space point ptr... 0x395a0bf0, 0x395a0950
Caught exception in looking up space point ptr... 0x8a9de20, 0x14914e80
Caught exception in looking up space point ptr... 0x14914e80, 0x8a9fa50
Caught exception in looking up space point ptr... 0x8a9fa50, 0x2c377bd0
Caught exception in looking up space point ptr... 0x32a5a5f0, 0x360f0f20
Caught exception in looking up space point ptr... 0x360f0f20, 0x28e7d3f0
Caught exception in looking up space point ptr... 0x5250eb10, 0x43924780
Caught exception in looking up space point ptr... 0x43924780, 0x5054ac40
Caught exception in looking up space point ptr... 0x3da1d090, 0x46ac10a0
Caught exception in looking up space point ptr... 0x46ac10a0, 0x3da1d090
Caught exception in looking up space point ptr... 0x628b5590, 0x4f2f5d60
Caught exception in looking up space point ptr... 0x628b5400, 0x628b5590
Caught exception in looking up space point ptr... 0x24b0fe70, 0x24b0fce0
Caught exception in looking up space point ptr... 0x24b0fce0, 0x24b0fb50
Caught exception in looking up space point ptr... 0xf650100, 0x1252f110
Caught exception in looking up space point ptr... 0x1252f110, 0xf650100
Caught exception in looking up space point ptr... 0x602cd2e0, 0x286ca1a0
Caught exception in looking up space point ptr... 0x286ca1a0, 0x286c96f0
Caught exception in looking up space point ptr... 0x405b0d00, 0x3b507040
Caught exception in looking up space point ptr... 0x3b507040, 0x405b0d00
Caught exception in looking up space point ptr... 0xecbf2c0, 0x3b506b50
Caught exception in looking up space point ptr... 0x3b506b50, 0xecbf2c0
Caught exception in looking up space point ptr... 0x24c1e9a0, 0x131cd820
Caught exception in looking up space point ptr... 0x131cd820, 0x24c1e9a0
Caught exception in looking up space point ptr... 0xa6e7410, 0x21fcbbf0
Caught exception in looking up space point ptr... 0x21fcbbf0, 0xa6e7410
++++>>>> total num hits: 41600, num free: 40226
04-Feb-2026 01:04:12 GMT  Closed output file "np02vd_raw_run042421_0068_df-s03-d1_dw_0_20260203T140748_reco_stage0_20260204T010412_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         98.6558       113.671       138.401       110.377       11.2371        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.5251e-05    0.000140166   0.000285499   0.000108826   5.27674e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                12.2959       15.0667       17.677        15.1544       1.5003         27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.553536      0.628079      0.740789      0.623889      0.0571301       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0179525     0.0283593     0.0864838     0.0216097     0.0171851       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0871777     0.0943605     0.118742      0.0889371    0.00894418       27     
produce:beamevent:BeamEvent                      0.00011092    0.000209315   0.00146218    0.000151737   0.000248551      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.64949       3.96153       4.3113        4.00463      0.162102        27     
produce:ophit:OpHitFinder                         0.023058      0.0415888      0.05587      0.042577     0.00657407       27     
produce:wclsdatavd:WireCellToolkit                 59.6438       75.4715       89.6371       75.3172       8.0166         27     
produce:gaushit:GausHitFinder                     0.491751       1.08269       1.84906      0.961611      0.387116        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000207973   0.000359796   0.000896603   0.000340623   0.000128601      27     
produce:reco3d:SpacePointSolver                    5.27593       10.6964       17.8875       10.3258       3.34609        27     
produce:hitpdune:DisambigFromSpacePoints          0.0631455     0.115572      0.229929      0.107169      0.0383819       27     
produce:cluster3d:Cluster3D                       0.784065       2.24988       6.06045       1.87468       1.3177         27     
[art]:TriggerResults:TriggerResultInserter       2.2362e-05    3.40303e-05   0.000133703   3.0438e-05    2.01018e-05      27     
end_path:out1:RootOutput                          5.431e-06    1.06573e-05   3.8142e-05     9.788e-06    5.84474e-06      27     
end_path:out1:RootOutput(write)                    3.40996       4.2023        5.50778       3.84656      0.711707        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4536.81 MB
  Peak resident set size usage (VmHWM): 2571.77 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0068_df-s03-d1_dw_0_20260203T140748_reco_stage0_20260204T010412_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0068_df-s03-d1_dw_0_20260203T140748_reco_stage0_20260204T010412_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0068_df-s03-d1_dw_0_20260203T140748_reco_stage0_20260204T010412_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163899.0,
    "core.end_time": 1770167054.0,
    "core.events": [
      11079,
      11085,
      11091,
      11097,
      11103,
      11109,
      11115,
      11121,
      11127,
      11133,
      11139,
      11145,
      11151,
      11157,
      11163,
      11169,
      11175,
      11181,
      11187,
      11193,
      11199,
      11205,
      11211,
      11217,
      11223,
      11229,
      11235
    ],
    "core.event_count": 27,
    "core.first_event_number": 11079,
    "core.last_event_number": 11235,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0068_df-s03-d1_dw_0_20260203T140748.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:25 UTC       justIN version: 01.06.00