Feature #55693: Limit the Health Detail MSG log size in cluster logs - RADOS - Ceph

Actions

Copy link

Feature #55693

open

Limit the Health Detail MSG log size in cluster logs

Added by Vikhyat Umrao almost 2 years ago. Updated almost 2 years ago.

Status:

Fix Under Review

Priority:

Normal

Assignee:

Prashant D

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Reviewed:

Affected Versions:

Component(RADOS):

Pull request ID:

46959

Description

RHBZ# https://bugzilla.redhat.com/show_bug.cgi?id=2087527

Version-Release number of selected component (if applicable):
Upstream quincy

In a recent occurrence on one of the Ceph upstream clusters, the LRC one went into DU mode due to a bug https://tracker.ceph.com/issues/54132.

Issues caused by this problem:

- SSH errors logged massive hexdump output (cephadm bug already fixed)
- These logs got stored as part of ‘ceph health detail’ as well
- Periodically we dumped ‘ceph health detail’ to the cluster log (going through paxos, each mon db, etc.) causing the mons to lose quorum and become unresponsive