Feature #20606: mds: improve usability of cluster rank manipulation and setting cluster up/down - CephFS - Ceph

Actions

Copy link

Feature #20606

closed

mds: improve usability of cluster rank manipulation and setting cluster up/down

Added by Patrick Donnelly almost 7 years ago. Updated about 6 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Douglas Fuller

Category:

Administration/Usability

Target version:

Ceph - v13.0.0

% Done:

100%

Source:

Development

Tags:

Backport:

Reviewed:

Affected Versions:

Component(FS):

MDSMonitor

Labels (FS):

multimds

Pull request ID:

Description

Right now the procedure for bringing down a cluster is:

ceph fs set cephfs_a cluster_down 1
ceph mds fail 1:1 # rank 1 of 2
ceph mds fail 1:0 # rank 0 of 2
ceph status
  cluster:
    id:     4ef94796-a652-4e0f-ad4e-8f3aaa9b9d18
    health: HEALTH_ERR
            mds ranks 0,1 have failed
            mds cluster is degraded

  services:
    mon: 3 daemons, quorum a,b,c
    mgr: x(active)
    mds: 0/2/2 up, 2 up:standby, 2 failed
    osd: 3 osds: 3 up, 3 in

  data:
    pools:   2 pools, 16 pgs
    objects: 39 objects, 3558 bytes
    usage:   3265 MB used, 27646 MB / 30911 MB avail
    pgs:     16 active+clean

This leaves the journal unflushed and client sessions half-open. Also, disturbing notices are in `ceph status` showing "failed" mdss and unhelpful health warnings.

I would recommend several changes outlined in this issue's sub-tasks.

Subtasks 5 (0 open — 5 closed)

Actions

Copy link

Updated by Patrick Donnelly almost 7 years ago

Description updated (diff)

Actions

Copy link

Updated by Patrick Donnelly almost 7 years ago

Subject changed from mds: allow cluster to be shut down gently and without warnings/errors to mds: improve usability of cluster rank manipulation and setting cluster up/down
Release set to master

Actions

Copy link

Updated by John Spray almost 7 years ago

My thoughts on this:

maybe we should preface this class of command (manipulating the MDS ranks) with "cluster", so we'd have commands like "ceph fs cluster down", "ceph fs cluster set size", etc?
the 'deactivate' stuff is probably clearer if we re-cast it as operating on an FS rank rather than a daemon. So really we're saying "tear down this rank, whichever MDS daemon is holding it", rather than "MDS daemon xyz, please tear down the rank you hold". That might avoid the awkward naming of 'deactivate'.
I'm a bit fuzzy on the stuff here about bringing the cluster down, can't tell if it's about shrinking the cluster, or cleanly stopping daemons (to start them again later)?

Actions

Copy link

Updated by Patrick Donnelly almost 7 years ago

John Spray wrote:

My thoughts on this:

maybe we should preface this class of command (manipulating the MDS ranks) with "cluster", so we'd have commands like "ceph fs cluster down", "ceph fs cluster set size", etc?

I like it!

the 'deactivate' stuff is probably clearer if we re-cast it as operating on an FS rank rather than a daemon. So really we're saying "tear down this rank, whichever MDS daemon is holding it", rather than "MDS daemon xyz, please tear down the rank you hold". That might avoid the awkward naming of 'deactivate'.

I was also thinking similarly: let's move `ceph mds` commands that operate on ranks to `ceph fs`.

I'm a bit fuzzy on the stuff here about bringing the cluster down, can't tell if it's about shrinking the cluster, or cleanly stopping daemons (to start them again later)?

I don't think I get your question. Can you rephrase?

Actions

Copy link

Updated by John Spray almost 7 years ago

The last point about cluster down: looking at http://tracker.ceph.com/issues/20609, I'm not sure what the higher level goal is. If we wanted to free up daemons to do other work (while making this filesystem inaccessible), then that's what the existing "cluster down" does. If we wanted to deactivate ranks, then I'm not sure why we'd ever want to deactivate the last one.

Actions

Copy link

Updated by Patrick Donnelly over 6 years ago

John Spray wrote:

The last point about cluster down: looking at http://tracker.ceph.com/issues/20609, I'm not sure what the higher level goal is. If we wanted to free up daemons to do other work (while making this filesystem inaccessible), then that's what the existing "cluster down" does. If we wanted to deactivate ranks, then I'm not sure why we'd ever want to deactivate the last one.

The idea is to provide a mechanism for cleanly bringing the cluster down. Admittedly this is not something we expect people to be doing except in extraordinary cases or in testing. However, I thought we had an opportunity to improve this while thinking about the related issues.

Actions

Copy link