Project

General

Profile

Bug #15778

ceph.restart + ceph_manager.wait_for_clean is racy

Added by Sage Weil about 3 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
05/09/2016
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

  - ceph.restart:
    - osd.2
  - ceph_manager.wait_for_clean: null

can proceed before osd.2 has restarted. That's because restart just restarts the daemon, but it may not have marked itself down yet. And wait_for_clean just checks the state in the osdmap, which may still say all osds are still up (from before restart).

For example, /a/sage-2016-05-07_04:30:37-rados-wip-sage-testing2---basic-smithi/176301

I think the fix is for ceph.restart to explicitly mark the osd down, or wait until the daemon has gotten far enough to mark the old osd down.

Note that the revive_osd() method in ceph_manager (used by thrasher) narrows this race by looping until an admin socket command works.


Related issues

Related to ceph-qa-suite - Bug #17808: ceph.restart is racy Verified 11/07/2016
Copied to ceph-qa-suite - Backport #23066: jewel: ceph.restart + ceph_manager.wait_for_clean is racy Resolved

History

#1 Updated by Sage Weil about 3 years ago

  • Priority changed from High to Urgent

#2 Updated by Sage Weil about 3 years ago

/a/sage-2016-05-13_05:47:04-rados-master-distro-basic-smithi/190454

#3 Updated by Sage Weil about 3 years ago

/a/sage-2016-05-19_19:44:21-rados-wip-sage-testing-distro-basic-smithi/203206

#4 Updated by Warren Usui about 3 years ago

I tried the following change to ceph.py which explicitly marks the osd down in ceph.restart:

diff --git a/tasks/ceph.py b/tasks/ceph.py
index 114f25b..43735be 100644
--- a/tasks/ceph.py
+++ b/tasks/ceph.py
@@ -1214,6 +1214,13 @@ def restart(ctx, config):
     if config.get('wait-for-osds-up', False):
         for cluster in clusters:
             wait_for_osds_up(ctx=ctx, config=dict(cluster=cluster))
+    manager = ctx.managers['ceph']
+    for dmon in daemons:
+        if '.' in dmon:
+            dm_parts = dmon.split('.')
+            if dm_parts[1].isdigit():
+                if dm_parts[0] == 'osd':
+                    manager.mark_down_osd(int(dm_parts[1]))
     yield

This seemed to work on some tests, but occasionally I would get teuthology tests that end with:

2016-05-24T21:01:06.810 DEBUG:teuthology.parallel:result is None
2016-05-24T21:01:06.810 INFO:teuthology.task.install:Purging /var/lib/ceph on ubuntu@vpm130.front.sepia.ceph.com
2016-05-24T21:01:06.811 INFO:teuthology.orchestra.run.vpm130:Running: "sudo rm -rf --one-file-system -- /var/lib/ceph || true ; test -d /var/lib/ceph && sudo find /var/lib/ceph -mindepth 1 -maxdepth 2 -type d -exec umount '{}' ';' ; sudo rm -rf --one-file-system -- /var/lib/ceph" 
2016-05-24T21:01:06.830 DEBUG:teuthology.parallel:result is None
2016-05-24T21:01:06.830 INFO:teuthology.nuke:Installed packages removed.
2016-05-24T21:01:06.835 INFO:teuthology.run:Summary data:
{duration: 410.3270421028137, failure_reason: '"2016-05-25 03:56:47.081880 osd.0 172.21.2.130:6804/17917
    3 : cluster [WRN] map e17 wrongly marked me down" in cluster log', flavor: basic,
  owner: wusui@teuthology, success: false}

2016-05-24T21:01:06.836 WARNING:teuthology.report:No job_id found; not reporting results
2016-05-24T21:01:06.836 INFO:teuthology.run:FAIL

So this looks like something is marked down that should not be. So I am suspicious that the above change is causing this problem.

#5 Updated by Yuri Weinstein about 3 years ago

Warren Usui wrote:

I tried the following change to ceph.py which explicitly marks the osd down in ceph.restart:

[...]

This seemed to work on some tests, but occasionally I would get teuthology tests that end with:

[...]

So this looks like something is marked down that should not be. So I am suspicious that the above change is causing this problem.

@Warren "map e17 wrongly marked me down" may need to be white listed.

#6 Updated by Warren Usui about 3 years ago

  • Assignee set to Warren Usui

#7 Updated by Warren Usui about 3 years ago

This seems to work okay with 'wrongly mark me down' white-listed. I have pushed a new branch wip-ceph-restart-wusui.

#8 Updated by Warren Usui about 3 years ago

Pull request: tasks/ceph.restart osd fix #1014 is ready for review

#9 Updated by Warren Usui almost 3 years ago

  • Status changed from New to Resolved

commit 1b7552c9cb331978cb0bfd4d7dc4dcde4186c176

#10 Updated by Loic Dachary over 2 years ago

  • Project changed from teuthology to ceph-qa-suite

#11 Updated by Loic Dachary over 2 years ago

  • Related to Bug #17808: ceph.restart is racy added

#12 Updated by Nathan Cutler over 1 year ago

  • Status changed from Resolved to Pending Backport
  • Backport set to jewel

#13 Updated by Nathan Cutler over 1 year ago

  • Copied to Backport #23066: jewel: ceph.restart + ceph_manager.wait_for_clean is racy added

#14 Updated by Nathan Cutler about 1 year ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF