justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.22@dunegpschedd02.fnal.gov

Jobsub ID297921.22@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 00:40:59
From worker nodeHostnameheplnc171.pp.rl.ac.uk
cpuinfoAMD EPYC 9534 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:45:20
Input filesvd-protodune:np02vd_raw_run042433_0110_df-s04-d1_dw_0_20260203T192356.hdf5
JobscriptExit code0
Real time53m (3192s)
CPU time48m (2935s = 91%)
Max RSS bytes3384066048 (3227 MiB)
Outputting started2026-02-04 00:38:34
Output files
Finished2026-02-04 00:40:59
Saved logsjustin-logs:297921.22-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0xac77fc0, 0x16cb6ea0
Caught exception in looking up space point ptr... 0x16cb6ea0, 0x16cb6d50
Caught exception in looking up space point ptr... 0x16cb6d50, 0x16cb6ea0
Caught exception in looking up space point ptr... 0xe586c40, 0xf9ec2c0
Caught exception in looking up space point ptr... 0xf9ec2c0, 0xf126200
Caught exception in looking up space point ptr... 0xa74c430, 0xa74c190
Caught exception in looking up space point ptr... 0xa74c190, 0xa74c350
Caught exception in looking up space point ptr... 0x6b32da70, 0x6b32f450
Caught exception in looking up space point ptr... 0x6b32f450, 0x6b32da70
Caught exception in looking up space point ptr... 0x59812e80, 0x59818df0
Caught exception in looking up space point ptr... 0x59818df0, 0x598043e0
Caught exception in looking up space point ptr... 0xcb71840, 0xcb714f0
Caught exception in looking up space point ptr... 0xcb714f0, 0xcb71840
Caught exception in looking up space point ptr... 0x2c2d4f40, 0x2c2d4d60
Caught exception in looking up space point ptr... 0x2c2d4d60, 0x2c2d4f40
Caught exception in looking up space point ptr... 0x4f73d070, 0x4f738010
Caught exception in looking up space point ptr... 0x4f738010, 0x4f73d070
Caught exception in looking up space point ptr... 0x4f73d070, 0x4f73e0f0
Caught exception in looking up space point ptr... 0x4f73e0f0, 0x4f73d070
Caught exception in looking up space point ptr... 0x303d64c0, 0x303d7400
Caught exception in looking up space point ptr... 0x303d7400, 0x303dcc50
Caught exception in looking up space point ptr... 0x303dcc50, 0x303dcd90
Caught exception in looking up space point ptr... 0x81956240, 0x81956420
Caught exception in looking up space point ptr... 0x81956420, 0x81956240
Caught exception in looking up space point ptr... 0x81956240, 0x81956380
Caught exception in looking up space point ptr... 0x81956380, 0x81956240
Caught exception in looking up space point ptr... 0x3b723d40, 0x3b724e70
Caught exception in looking up space point ptr... 0x3b724e70, 0x3b724ae0
Caught exception in looking up space point ptr... 0x303ebc60, 0x303eaa60
Caught exception in looking up space point ptr... 0x303eaa60, 0x303ebe50
Caught exception in looking up space point ptr... 0x53213e60, 0x3b9adb40
Caught exception in looking up space point ptr... 0x3b9adb40, 0x53213e60
Caught exception in looking up space point ptr... 0x4e060420, 0x3215aae0
Caught exception in looking up space point ptr... 0x3215aae0, 0x3215cac0
Caught exception in looking up space point ptr... 0x4d3aa510, 0x4d3a53e0
Caught exception in looking up space point ptr... 0x4d3a53e0, 0x2c1b5d50
Caught exception in looking up space point ptr... 0x51dabd00, 0x51dacd90
Caught exception in looking up space point ptr... 0x51dacd90, 0x129b1140
Caught exception in looking up space point ptr... 0x9d0fe70, 0x43dbef30
Caught exception in looking up space point ptr... 0x43dbef30, 0x43dbee30
Caught exception in looking up space point ptr... 0x69cccf30, 0x10c9ac50
Caught exception in looking up space point ptr... 0x10c9ac50, 0x74ee75d0
Caught exception in looking up space point ptr... 0x408d4a00, 0x69cccf30
Caught exception in looking up space point ptr... 0x625720f0, 0x16d2b7f0
Caught exception in looking up space point ptr... 0x16d2b7f0, 0x625720f0
Caught exception in looking up space point ptr... 0x3a53d600, 0x863c6a60
Caught exception in looking up space point ptr... 0x863c6a60, 0x11ea0420
Caught exception in looking up space point ptr... 0x4f73eb40, 0x4f735740
Caught exception in looking up space point ptr... 0x4f735740, 0x4f73eb40
Caught exception in looking up space point ptr... 0x747b2b80, 0x747b29f0
Caught exception in looking up space point ptr... 0x747b29f0, 0x747b2860
Caught exception in looking up space point ptr... 0x11147170, 0x11d74eb0
Caught exception in looking up space point ptr... 0x11d74eb0, 0x19086080
Caught exception in looking up space point ptr... 0x3a529360, 0x32031740
Caught exception in looking up space point ptr... 0x32031740, 0x3be16320
Caught exception in looking up space point ptr... 0x2c278c00, 0x2c26fb50
Caught exception in looking up space point ptr... 0x2c26fb50, 0x2c278c00
Caught exception in looking up space point ptr... 0x2c278c00, 0x2c274d20
Caught exception in looking up space point ptr... 0x2c274d20, 0x2c278c00
Caught exception in looking up space point ptr... 0x853e350, 0x853e5e0
Caught exception in looking up space point ptr... 0x853e5e0, 0x2f61cef0
Caught exception in looking up space point ptr... 0xa743010, 0xa742ec0
Caught exception in looking up space point ptr... 0xa742ec0, 0xa742d70
Caught exception in looking up space point ptr... 0x3bc074a0, 0x10801160
Caught exception in looking up space point ptr... 0x10801160, 0x4ab26510
Caught exception in looking up space point ptr... 0x5e71cec0, 0x5b7c6fb0
Caught exception in looking up space point ptr... 0x5b7c6fb0, 0x5e71cec0
++++>>>> total num hits: 64562, num free: 61688
04-Feb-2026 00:38:25 GMT  Closed output file "np02vd_raw_run042433_0110_df-s04-d1_dw_0_20260203T192356_reco_stage0_20260204T003825_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         148.692       163.872       181.555       164.467       8.46965        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.6201e-05    0.000126469   0.000206083   9.6022e-05    4.64022e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                13.6552       14.1269       14.7942       14.0261       0.26515        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.525495      0.550664      0.620038      0.545061      0.0270971       19     
produce:timingrawdecoder:PDHDTimingRawDecoder      0.01763      0.0179283     0.0182531     0.0179356    0.000160598      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0176057     0.0179152     0.0190489     0.0178807    0.000304815      19     
produce:beamevent:BeamEvent                      0.000147012   0.000227845   0.000945542   0.000175403   0.000173983      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.76196       3.28774       3.71388       3.30845       0.25285        19     
produce:ophit:OpHitFinder                         0.0302943     0.040308      0.0453618     0.042025     0.00489884       19     
produce:wclsdatavd:WireCellToolkit                 104.374       115.881       125.331       115.459       6.41833        19     
produce:gaushit:GausHitFinder                     0.839063       1.51793       2.43117       1.53093      0.392936        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000214974   0.000358876   0.00067685    0.000384755   0.000114139      19     
produce:reco3d:SpacePointSolver                    7.51555       19.2441       29.7226       17.7244       5.82923        19     
produce:hitpdune:DisambigFromSpacePoints          0.102806      0.227255      0.368102      0.219794      0.0684928       19     
produce:cluster3d:Cluster3D                       0.879637       3.05171       5.73874       2.76273       1.1453         19     
[art]:TriggerResults:TriggerResultInserter        2.858e-05    4.21916e-05   0.000110211    3.756e-05    1.74199e-05      19     
end_path:out1:RootOutput                          4.32e-06     9.91163e-06    2.451e-05     9.62e-06     3.89136e-06      19     
end_path:out1:RootOutput(write)                    4.23623       5.85029       6.67732       6.06691      0.722011        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5372.17 MB
  Peak resident set size usage (VmHWM): 3384.07 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0110_df-s04-d1_dw_0_20260203T192356_reco_stage0_20260204T003825_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0110_df-s04-d1_dw_0_20260203T192356_reco_stage0_20260204T003825_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0110_df-s04-d1_dw_0_20260203T192356_reco_stage0_20260204T003825_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162332.0,
    "core.end_time": 1770165507.0,
    "core.events": [
      12551,
      12557,
      12563,
      12569,
      12575,
      12581,
      12587,
      12593,
      12599,
      12605,
      12611,
      12617,
      12623,
      12629,
      12635,
      12641,
      12647,
      12653,
      12659
    ],
    "core.event_count": 19,
    "core.first_event_number": 12551,
    "core.last_event_number": 12659,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0110_df-s04-d1_dw_0_20260203T192356.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:27 UTC       justIN version: 01.06.00