justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 298018.0@dunegpschedd02.fnal.gov

Jobsub ID298018.0@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:52:31
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 04:15:57
From worker nodeHostnamewn2208251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:53:22
Input filesvd-protodune:np02vd_raw_run042421_0093_df-s03-d0_dw_0_20260203T142119.hdf5
JobscriptExit code0
Real time2h (8394s)
CPU time51m (3079s = 36%)
Max RSS bytes2575446016 (2456 MiB)
Outputting started2026-02-04 04:13:19
Output files
Finished2026-02-04 04:15:57
Saved logsjustin-logs:298018.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

01e610
Caught exception in looking up space point ptr... 0x3c01e610, 0x3c0132e0
Caught exception in looking up space point ptr... 0x11e284c0, 0x11e294d0
Caught exception in looking up space point ptr... 0x11e294d0, 0x50311a50
Caught exception in looking up space point ptr... 0xa9bc0f0, 0xa9bc280
Caught exception in looking up space point ptr... 0x4dc6ed10, 0xa9bc0f0
Caught exception in looking up space point ptr... 0x40ad6f00, 0x40ad7220
Caught exception in looking up space point ptr... 0x40ad7220, 0x40ad6f00
Caught exception in looking up space point ptr... 0x423257a0, 0x42325610
Caught exception in looking up space point ptr... 0x42325610, 0x423252f0
Caught exception in looking up space point ptr... 0x56eb6500, 0x56eb6260
Caught exception in looking up space point ptr... 0x56eb6260, 0x56eb6500
Caught exception in looking up space point ptr... 0x4ede54b0, 0x4ede1580
Caught exception in looking up space point ptr... 0x4ede1580, 0x3dc98b40
Caught exception in looking up space point ptr... 0x3ad54220, 0xe4acee0
Caught exception in looking up space point ptr... 0xe4ad070, 0x3ad54220
Caught exception in looking up space point ptr... 0x4e930bd0, 0x261e87d0
Caught exception in looking up space point ptr... 0x261e87d0, 0x4e930bd0
Caught exception in looking up space point ptr... 0x3736ecc0, 0x2a8adbf0
Caught exception in looking up space point ptr... 0x2a8adbf0, 0x3736ecc0
Caught exception in looking up space point ptr... 0x2c7aa3c0, 0x2c7a2cc0
Caught exception in looking up space point ptr... 0x2c7a2cc0, 0x2c77fe20
Caught exception in looking up space point ptr... 0x289d4450, 0x289d41b0
Caught exception in looking up space point ptr... 0x289d41b0, 0x289d4060
Caught exception in looking up space point ptr... 0x259a7670, 0x8409580
Caught exception in looking up space point ptr... 0x8409580, 0x259a7670
Caught exception in looking up space point ptr... 0x35967e40, 0x35967c50
Caught exception in looking up space point ptr... 0x35967c50, 0x3596a5b0
Caught exception in looking up space point ptr... 0x4dc6f1c0, 0x4dc6f350
Caught exception in looking up space point ptr... 0x4dc6f350, 0x4dc6f1c0
Caught exception in looking up space point ptr... 0x335ff480, 0x4edbd220
Caught exception in looking up space point ptr... 0x4edbd220, 0x335fbf30
Caught exception in looking up space point ptr... 0x2e964260, 0x4fdc5340
Caught exception in looking up space point ptr... 0x4fdc5340, 0x2e964260
Caught exception in looking up space point ptr... 0x452545c0, 0x14f6fe30
Caught exception in looking up space point ptr... 0x14f6fe30, 0x452545c0
Caught exception in looking up space point ptr... 0x30442590, 0x5b58e670
Caught exception in looking up space point ptr... 0x5b58e670, 0x5b58e490
Caught exception in looking up space point ptr... 0x30c81990, 0x308d6150
Caught exception in looking up space point ptr... 0x308d6150, 0x30c81990
Caught exception in looking up space point ptr... 0x261d98d0, 0x261d9b30
Caught exception in looking up space point ptr... 0x261d9b30, 0x261d98d0
Caught exception in looking up space point ptr... 0x272f6cb0, 0x272f2ec0
Caught exception in looking up space point ptr... 0x272f2ec0, 0x272f6cb0
Caught exception in looking up space point ptr... 0x272f6cb0, 0x2ff9c650
Caught exception in looking up space point ptr... 0x2ff9c650, 0x272f6cb0
Caught exception in looking up space point ptr... 0xcf0c260, 0x3d2f1440
Caught exception in looking up space point ptr... 0x3d2f1440, 0xcf0c260
Caught exception in looking up space point ptr... 0xcf0c260, 0x6645e060
Caught exception in looking up space point ptr... 0x6645e060, 0xcf0c260
Caught exception in looking up space point ptr... 0x4b277f10, 0x4b278200
Caught exception in looking up space point ptr... 0x4b278200, 0x4b277f10
Caught exception in looking up space point ptr... 0x51f8f480, 0x51f8f610
Caught exception in looking up space point ptr... 0x51f8f610, 0x51f8f480
Caught exception in looking up space point ptr... 0x502fa700, 0x502fa4d0
Caught exception in looking up space point ptr... 0x502fa4d0, 0x502fa700
Caught exception in looking up space point ptr... 0x5a476970, 0x5a476650
Caught exception in looking up space point ptr... 0x5a476650, 0x5a4761a0
Caught exception in looking up space point ptr... 0x15c6b8b0, 0xa41dd10
Caught exception in looking up space point ptr... 0xa41dd10, 0x15c6b8b0
Caught exception in looking up space point ptr... 0x334ba600, 0x334ba4d0
Caught exception in looking up space point ptr... 0x334ba4d0, 0x334ba600
Caught exception in looking up space point ptr... 0x2ff9ab50, 0x2ff9a160
Caught exception in looking up space point ptr... 0x2ff9a160, 0x2ff9ab50
Caught exception in looking up space point ptr... 0x4b770630, 0x4b770790
Caught exception in looking up space point ptr... 0x4b770790, 0x4b770630
++++>>>> total num hits: 78191, num free: 75314
04-Feb-2026 04:13:08 GMT  Closed output file "np02vd_raw_run042421_0093_df-s03-d0_dw_0_20260203T142119_reco_stage0_20260204T041308_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         232.125       306.596       358.567       317.821       35.8788        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000100178   0.000189764   0.000448132   0.000177613   8.93223e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                113.567       178.721       216.091       184.239       28.9978        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.800251       1.10163       2.71673       1.01265      0.360448        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0220903     0.0439233     0.279924      0.0283141     0.0496866       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.108563      0.117313      0.162029       0.11128      0.0144757       27     
produce:beamevent:BeamEvent                      0.000141326   0.000260669   0.00122255    0.000214533   0.000197975      27     
produce:pdvddaphne:DAPHNEReaderPDVD                12.4373       22.0565       30.5821       21.9381       4.06774        27     
produce:ophit:OpHitFinder                         0.0349338     0.0463057     0.0569772     0.0469413    0.00479039       27     
produce:wclsdatavd:WireCellToolkit                 66.4988       84.6568       91.966        85.5844       5.26533        27     
produce:gaushit:GausHitFinder                      0.65956       1.2323        2.31808       1.23355      0.331835        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000215245   0.000340595   0.00068648    0.000314101   9.84825e-05      27     
produce:reco3d:SpacePointSolver                    5.86844       10.7021       17.0149       9.82485       3.41951        27     
produce:hitpdune:DisambigFromSpacePoints          0.0525476     0.124898      0.205716      0.123062      0.0382459       27     
produce:cluster3d:Cluster3D                        1.32765       2.62769       6.38445       2.29672       1.11169        27     
[art]:TriggerResults:TriggerResultInserter       2.5599e-05    4.39259e-05   9.0199e-05    4.0656e-05    1.32533e-05      27     
end_path:out1:RootOutput                          6.132e-06    1.25469e-05   3.1269e-05    1.1382e-05    5.70927e-06      27     
end_path:out1:RootOutput(write)                    3.99392       5.12013       5.87451       5.22064      0.530436        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4559.73 MB
  Peak resident set size usage (VmHWM): 2575.45 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0093_df-s03-d0_dw_0_20260203T142119_reco_stage0_20260204T041308_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0093_df-s03-d0_dw_0_20260203T142119_reco_stage0_20260204T041308_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0093_df-s03-d0_dw_0_20260203T142119_reco_stage0_20260204T041308_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770170015.0,
    "core.end_time": 1770178390.0,
    "core.events": [
      15134,
      15140,
      15146,
      15152,
      15158,
      15164,
      15170,
      15176,
      15182,
      15188,
      15194,
      15200,
      15206,
      15212,
      15218,
      15224,
      15230,
      15236,
      15242,
      15248,
      15254,
      15260,
      15266,
      15272,
      15278,
      15284,
      15290
    ],
    "core.event_count": 27,
    "core.first_event_number": 15134,
    "core.last_event_number": 15290,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0093_df-s03-d0_dw_0_20260203T142119.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:27:01 UTC       justIN version: 01.06.00