Project

General

Profile

Backport #15336

hammer: Memory leak and high CPU usage on OSD when system clock is in the future

Added by Loic Dachary over 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Target version:
Release:
hammer
Crash signature:


Related issues

Copied from Ceph - Bug #12065: Memory leak and high CPU usage on OSD when system clock is in the future Resolved 06/17/2015

Associated revisions

Revision ac3569cd (diff)
Added by Alexey Sheplyakov over 4 years ago

hammer: monclient: avoid key renew storm on clock skew

Refreshing rotating keys too often is a symptom of a clock skew, try to
detect it and don't cause extra problems:

  • MonClient::_check_auth_rotating:
    - detect and report premature keys expiration due to a time skew
    - rate limit refreshing the keys to avoid excessive RAM and CPU usage
    (both by OSD in question and monitors which have to process a lot
    of auth messages)
  • MonClient::wait_auth_rotating: wait for valid (not expired) keys
  • OSD::init(): bail out after 10 attempts to obtain the rotating keys

Fixes: #15336

Signed-off-by: Alexey Sheplyakov <>
(cherry picked from commit 918c12c2ab5d014d0623b1accf959b041aac5128)

Conflicts:
src/osd/OSD.cc
no loadavg checks in Hammer, hence the conflict. Move the
counter and max_attempts initialization to a suitable place.

History

#1 Updated by Loic Dachary over 4 years ago

  • Copied from Bug #12065: Memory leak and high CPU usage on OSD when system clock is in the future added

#3 Updated by Loic Dachary over 4 years ago

  • Description updated (diff)
  • Status changed from New to In Progress
  • Assignee set to Alexey Sheplyakov

#4 Updated by Sage Weil over 4 years ago

  • Status changed from In Progress to Resolved

#5 Updated by Loic Dachary almost 4 years ago

  • Target version set to v0.94.7

Also available in: Atom PDF