Project

General

Profile

Bug #17719

OSDs marked OUT wrongly after monitor failover

Added by Ridge Chen 10 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Monitor
Target version:
-
Start date:
10/27/2016
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
hammer,jewel
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

Recently we find an issue with our ceph cluster, the version is 0.94.6.

We want to add additional RAM to the ceph nodes, so we need to stop
the ceph service on the nodes first. When we did that on the first
node, we found the OSDs on that node marked OUT and backfill started
(DOWN is expected in this case). The first node is somewhat special
that it is also the location of the leader monitor.

Then checked the monitor log and found the following:

cluster [INF] osd.0 out (down for 3375169.141844)

Looks like the monitor (who just become leader) has wrong
"down_pending_out" records and computes out a a very long DOWN time ,
finally decides to mark them OUT.

After researching the related code, the reason could be that:

1. "down_pending_out" is set a month ago for those OSDs because of a
network issue.
2. The down OSDs up and join the cluster again. "down_pending_out" is
cleared in the "OSDMonitor::tick()" method. But only happened on
leader monitor.
3. When we stop the ceph service on the first node. The monitor group
failover. The new leader monitor will recognize the OSDs kept in DOWN
status for a a very long time, and mark them OUT wrongly.


Related issues

Copied to Ceph - Backport #17883: hammer: OSDs marked OUT wrongly after monitor failover Resolved
Copied to Ceph - Backport #17884: jewel: OSDs marked OUT wrongly after monitor failover Resolved

History

#2 Updated by Kefu Chai 10 months ago

  • Category set to Monitor
  • Status changed from New to Need Review

#3 Updated by Kefu Chai 10 months ago

  • Backport set to hammer,jewel

#4 Updated by Kefu Chai 9 months ago

  • Status changed from Need Review to Pending Backport

#5 Updated by Kefu Chai 9 months ago

would be idea if we can have a test for this bug in ceph-qa-suite rados suite.

#6 Updated by Nathan Cutler 9 months ago

  • Copied to Backport #17883: hammer: OSDs marked OUT wrongly after monitor failover added

#7 Updated by Nathan Cutler 9 months ago

  • Copied to Backport #17884: jewel: OSDs marked OUT wrongly after monitor failover added

#8 Updated by Nathan Cutler 7 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF