Project

General

Profile

Bug #16255

ceph-create-keys: sometimes blocks forever if mds "allow" is set

Added by Dietmar Maurer about 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Administration/Usability
Target version:
-
Start date:
06/13/2016
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
jewel
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:

Description

The documentations at:

http://docs.ceph.com/docs/master/dev/mon-bootstrap/

tells to create the client.admin key with:

  1. ceph-authtool /path/to/keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'

Please note the mds permission is 'allow', and not 'allow *'.

I never understood why this need to be different, but I used this everywhere. So all our
existing installations use it.

But this commit:

https://github.com/ceph/ceph/commit/c7e905e7e232a973abf7c6fa71a2ffbad7aa0ffd

suddenly requires 'allow *' for mds permissions. So with my old keys,
ceph-create-keys simply runs an endless loop logging the following
error message:

Jun 13 06:10:17 elsa ceph-create-keys10083: Error EINVAL: key for client.admin
exists but cap mds does not match
Jun 13 06:10:17 elsa ceph-create-keys10083: INFO:ceph-create-keys:Cannot get
or create admin key
Jun 13 06:10:18 elsa ceph-create-keys10083: INFO:ceph-create-keys:Talking to
monitor...

So either the documentation is wrong, or the commit should be reverted.


Related issues

Copied to fs - Backport #17347: jewel: ceph-create-keys: sometimes blocks forever if mds "allow" is set Resolved

History

#1 Updated by John Spray about 3 years ago

  • Project changed from Ceph to fs

The difference between "allow" and "allow *" is that the "*" is necessary in more recent versions to issue 'tell' commands. Having an existing key that only has "allow" isn't usually a problem until someone tries to do "ceph tell mds.foo ...".

The loop you're seeing presumably is only occurring when /etc/ceph/ceph.client-admin.keyring has been removed.

Folks on systemd-using systems will probably not be seeing this issue because the systemd service file checks for /var/lib/ceph/bootstrap-mds/ceph.keyring.

Anyway, this is a bug. ceph-create-keys should not loop forever if the admin key exists but has different caps than the default.

#2 Updated by John Spray about 3 years ago

  • Subject changed from ceph-create-keys: set mds "allow *" is wrong! to ceph-create-keys: sometimes blocks forever if mds "allow" is set

#3 Updated by Dietmar Maurer about 3 years ago

The loop you're seeing presumably is only occurring when /etc/ceph/ceph.client-admin.keyring has been removed.

exactly (we do not store the key there).

Folks on systemd-using systems will probably not be seeing this issue because the systemd service file checks for /var/lib/ceph/bootstrap-mds/ceph.keyring.

Anyway, this is a bug. ceph-create-keys should not loop forever if the admin key exists but has different caps than the default.

Yes - I thought it should simply use the existing key.

#4 Updated by Greg Farnum about 3 years ago

  • Category set to Administration/Usability

#5 Updated by Alfredo Deza about 3 years ago

We've encountered the same issue when upgrading. The " on systemd-using systems will probably not be seeing this" is not an issue, what we see on a systemd server is that upgrading will trigger it.

This might be because the initial keyring for the cluster was created with `mds allow` and now the newly upgraded cluster requests `mds allow *` which returns the:

Error EINVAL: moncap parse failed, stopped at 'allow' of 'allow'

Relevant BZ: https://bugzilla.redhat.com/show_bug.cgi?id=135729

Regardless of the deployment method (manual/ceph-ansible/ceph-deploy) a user who adds a new monitor to an upgraded cluster will hit this problem

#6 Updated by John Spray about 3 years ago

  • Status changed from New to Need Review
  • Assignee set to John Spray

#7 Updated by Nathan Cutler about 3 years ago

  • Backport set to jewel

#8 Updated by huanwen ren almost 3 years ago

I've had the same issue when use ceph-deploy gatherkeys(jewel)
if I change "mds 'allow'" to "mds 'allow *'", it's true.

#9 Updated by Sage Weil almost 3 years ago

  • Status changed from Need Review to Pending Backport

#10 Updated by Nathan Cutler almost 3 years ago

  • Copied to Backport #17347: jewel: ceph-create-keys: sometimes blocks forever if mds "allow" is set added

#11 Updated by Nathan Cutler almost 3 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF