justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.138@dunegpschedd02.fnal.gov

Jobsub ID297923.138@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2026-02-04 02:28:46
From worker nodeHostnameheplnc162.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:04:41
Input filesvd-protodune:np02vd_raw_run042433_0614_df-s03-d0_dw_0_20260203T223528.hdf5
JobscriptExit code0
Real time2h (8391s)
CPU time2h (7882s = 93%)
Max RSS bytes14892773376 (14202 MiB)
Outputting started2026-02-04 02:24:37
Output files
Finished2026-02-04 02:28:46
Saved logsjustin-logs:297923.138-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x4189abe0, 0x4189ace0
Caught exception in looking up space point ptr... 0x3b7ecf50, 0x4189abe0
Caught exception in looking up space point ptr... 0xa8dd2830, 0xa8dd2590
Caught exception in looking up space point ptr... 0xa8dd2590, 0x81b6f650
Caught exception in looking up space point ptr... 0x81fb02d0, 0x14f6bd80
Caught exception in looking up space point ptr... 0x14f6bd80, 0x14f6bc00
Caught exception in looking up space point ptr... 0xd680910, 0xd680670
Caught exception in looking up space point ptr... 0xd6823b0, 0xd680910
Caught exception in looking up space point ptr... 0x13ec57150, 0x13ec573f0
Caught exception in looking up space point ptr... 0x13ec573f0, 0x13ec576e0
Caught exception in looking up space point ptr... 0x893e300, 0x893e100
Caught exception in looking up space point ptr... 0x893e100, 0x893e300
Caught exception in looking up space point ptr... 0x54dd13f0, 0x6345b090
Caught exception in looking up space point ptr... 0x6345b090, 0x75db4c30
Caught exception in looking up space point ptr... 0x5a3c3ab0, 0x5a3c3600
Caught exception in looking up space point ptr... 0x5a3c3600, 0x5a3c3ab0
Caught exception in looking up space point ptr... 0x443bba20, 0x443bb720
Caught exception in looking up space point ptr... 0x443bb720, 0x443bba20
Caught exception in looking up space point ptr... 0x4210fe20, 0x38f92970
Caught exception in looking up space point ptr... 0x38f92970, 0x4210fc40
Caught exception in looking up space point ptr... 0x1a3f82f0, 0x2a193b20
Caught exception in looking up space point ptr... 0x2a193b20, 0x1a3f82f0
Caught exception in looking up space point ptr... 0x7031f360, 0x7031f4d0
Caught exception in looking up space point ptr... 0x7031f4d0, 0x7031f660
Caught exception in looking up space point ptr... 0x33283760, 0x332838f0
Caught exception in looking up space point ptr... 0x332838f0, 0x33283760
Caught exception in looking up space point ptr... 0xa8dd2fa0, 0xa8dd32c0
Caught exception in looking up space point ptr... 0xa8dd32c0, 0xa8dd2fa0
Caught exception in looking up space point ptr... 0x124dcbce0, 0x79571060
Caught exception in looking up space point ptr... 0x79571060, 0x124dcbce0
Caught exception in looking up space point ptr... 0x151d3cd0, 0x151d3f80
Caught exception in looking up space point ptr... 0x151d3f80, 0x151d3cd0
Caught exception in looking up space point ptr... 0x56e4ef00, 0x56e4ed70
Caught exception in looking up space point ptr... 0x56e4ed70, 0x56e4ef00
Caught exception in looking up space point ptr... 0x3829e6e0, 0x56e95180
Caught exception in looking up space point ptr... 0x56e95180, 0x3829e6e0
Caught exception in looking up space point ptr... 0x572b1950, 0x572b17c0
Caught exception in looking up space point ptr... 0x572b17c0, 0x572b1950
Caught exception in looking up space point ptr... 0x5a40a140, 0x5a40a2d0
Caught exception in looking up space point ptr... 0x5a40a2d0, 0x5a40a460
Caught exception in looking up space point ptr... 0x5b4d74f0, 0x12f0c9c80
Caught exception in looking up space point ptr... 0x12f0c9c80, 0x12f0caed0
Caught exception in looking up space point ptr... 0xe3c19990, 0xe3c1a830
Caught exception in looking up space point ptr... 0xe3c1a830, 0xe3c19990
Caught exception in looking up space point ptr... 0x75e8b500, 0x3e6ac500
Caught exception in looking up space point ptr... 0x3e6ac500, 0x3e6ac3b0
Caught exception in looking up space point ptr... 0x9aecdd10, 0x9aecde60
Caught exception in looking up space point ptr... 0x9aecde60, 0x9aecdfb0
Caught exception in looking up space point ptr... 0x6341e030, 0x6341de90
Caught exception in looking up space point ptr... 0x7967e1c0, 0x6341e030
Caught exception in looking up space point ptr... 0x818362e0, 0x7fb5c950
Caught exception in looking up space point ptr... 0x7fb5c950, 0x7fb5b780
Caught exception in looking up space point ptr... 0x7034ea00, 0x703556a0
Caught exception in looking up space point ptr... 0x703556a0, 0x7034ea00
Caught exception in looking up space point ptr... 0x703230f0, 0x7031d9f0
Caught exception in looking up space point ptr... 0x7031d9f0, 0x703230f0
Caught exception in looking up space point ptr... 0xb0b12110, 0xb0b14a80
Caught exception in looking up space point ptr... 0xb0b14a80, 0xb0b17810
Caught exception in looking up space point ptr... 0xb0b13e00, 0xb0b12110
Caught exception in looking up space point ptr... 0xf6f22a10, 0xf6f22830
Caught exception in looking up space point ptr... 0xf6f22830, 0x18f777a0
Caught exception in looking up space point ptr... 0x12302f40, 0x12302df0
Caught exception in looking up space point ptr... 0x12302df0, 0x12302f40
Caught exception in looking up space point ptr... 0xdc8b100, 0x75ddc730
Caught exception in looking up space point ptr... 0x75ddc730, 0x75ddbd30
Caught exception in looking up space point ptr... 0x18366040, 0x18363e20
Caught exception in looking up space point ptr... 0x18363e20, 0x18366040
++++>>>> total num hits: 88378, num free: 85448
04-Feb-2026 02:24:22 GMT  Closed output file "np02vd_raw_run042433_0614_df-s03-d0_dw_0_20260203T223528_reco_stage0_20260204T022422_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         147.418       436.871       1582.5        181.342       463.659        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        7.455e-05    0.000142513   0.000333004   0.000109595   6.88811e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                13.9791       17.6407       31.1097       16.9555       4.36169        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.507103      0.536257      0.567067      0.535638      0.0134751       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.017598      0.0182674     0.0242018     0.0179339    0.00141789       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0174804     0.017925      0.0186156     0.0178306    0.00029128       19     
produce:beamevent:BeamEvent                      9.2905e-05    0.000235337   0.00145037    0.000155442   0.000290066      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.41061       4.09381       5.15173       4.07271       0.45459        19     
produce:ophit:OpHitFinder                         0.0376658     0.0440294     0.0508401     0.0451777    0.00411989       19     
produce:wclsdatavd:WireCellToolkit                 106.589       123.926       140.233       122.309       10.1609        19     
produce:gaushit:GausHitFinder                     0.758282       4.43339       22.8626       1.40652       6.02301        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000267632   0.000576089   0.00122572    0.000425979   0.000323055      19     
produce:reco3d:SpacePointSolver                    7.83245       35.8375       108.642       16.3205       37.2033        19     
produce:hitpdune:DisambigFromSpacePoints          0.0928611     0.977144       3.82999      0.185573       1.36071        19     
produce:cluster3d:Cluster3D                        1.02824       239.736       1296.49       2.54971       410.637        19     
[art]:TriggerResults:TriggerResultInserter       2.9164e-05    5.41595e-05   0.000152436    4.266e-05    2.91458e-05      19     
end_path:out1:RootOutput                          4.899e-06    1.19809e-05   3.5397e-05     9.117e-06    7.26863e-06      19     
end_path:out1:RootOutput(write)                    4.44117       9.53668       22.8854       5.13092       7.15336        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17663.1 MB
  Peak resident set size usage (VmHWM): 14892.8 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0614_df-s03-d0_dw_0_20260203T223528_reco_stage0_20260204T022422_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0614_df-s03-d0_dw_0_20260203T223528_reco_stage0_20260204T022422_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0614_df-s03-d0_dw_0_20260203T223528_reco_stage0_20260204T022422_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163498.0,
    "core.end_time": 1770171865.0,
    "core.events": [
      70004,
      70010,
      70016,
      70022,
      70028,
      70034,
      70040,
      70046,
      70052,
      70058,
      70064,
      70070,
      70076,
      70082,
      70088,
      70094,
      70100,
      70106,
      70112
    ],
    "core.event_count": 19,
    "core.first_event_number": 70004,
    "core.last_event_number": 70112,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0614_df-s03-d0_dw_0_20260203T223528.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:41 UTC       justIN version: 01.06.00