Bug #3055
closedceph command stuck until all predefined monitors are up
0%
Description
I'm using ceph-0.48.1 and encountered a problem from which I'm not sure if it's a bug or a wrong setup sequence. So please feel free to close as invalid if this applies. So at first, the short problem description:
When I have three monitors in ceph.conf and I'm generating the initial monmap on one of them (which in turn includes all three monitors), all further calls to 'ceph *' are stuck until all three (remaining two) monitors become available. Is this intended behaviour or is the initial monmap setup wrong?
More lengthy description and exact command sequence:
When I do something like this on a (the first) monitor named clayton:
Authentication (I'm not sure if related to authentication at all):
/usr/bin/ceph-authtool --create-keyring /etc/ceph/mon.clayton.keyring
/usr/bin/ceph-authtool --gen-key -n mon. /etc/ceph/mon.clayton.keyring
MON setup:
mkdir -p /tmp/mon-init
/sbin/mkcephfs -d /tmp/mon-init -c /etc/ceph/ceph.conf --prepare-monmap
/usr/bin/ceph-mon -c /etc/ceph/ceph.conf --mkfs -i clayton --monmap /tmp/mon-init/monmap -k /etc/ceph/mon.clayton.keyring
And ceph.conf looks something like:
[global]
auth supported = cephx
keyring = /etc/ceph/$name.keyring
[mon]
mon data = /var/lib/ceph/mon/ceph-$id
[mds]
[mon.clayton]
host = clayton
mon addr = 10.11.1.48:6789
[mon.delius]
host = delius
mon addr = 10.11.1.14:6789
[mon.jacobi]
host = jacobi
mon addr = 10.11.1.43:6789
So when starting the first monitor, any subsequent 'ceph -s' call hangs until all three monitors become available. This might be never in some cases, though. Is this intended behaviour? I mean, I'm setting up the monmap with monitors which are not there yet, only in ceph.conf, so this could be considered a wrong setup. However, a call to ceph shouldn't block indefinitely, right? In the end, one monitor should be enough.
Updated by Greg Farnum over 11 years ago
- Status changed from New to Won't Fix
By putting all three monitors in the monmap, you've declared them to all be part of the monitor group, and you'll need at least two of them in the quorum to make progress.
In order to prevent providing old information and really confusing semantics, monitors which are outside of the quorum won't let clients connect (to a first approximation, anyway; there are some details to handle when they're in elections and such).
So the scenario you've presented here isn't expected to work and really can't.
Updated by Holger Macht over 11 years ago
Thanks for the clarification, that's all I wanted to know.
Updated by Holger Macht over 11 years ago
One thing that still bothers me is how the correct way of doing this looks like. '/sbin/mkcephfs -d /tmp/mon-init -c /etc/ceph/ceph.conf --prepare-monmap' always grabs all the monitors from the configuration. Are you just not allowed to put monitors in ceph.conf which possibly will never come up? So it wouldn't be possible to create a single configuration file which can be used and deployed on all nodes, no? Thanks.
Updated by Sage Weil over 11 years ago
Holger Macht wrote:
One thing that still bothers me is how the correct way of doing this looks like. '/sbin/mkcephfs -d /tmp/mon-init -c /etc/ceph/ceph.conf --prepare-monmap' always grabs all the monitors from the configuration. Are you just not allowed to put monitors in ceph.conf which possibly will never come up? So it wouldn't be possible to create a single configuration file which can be used and deployed on all nodes, no? Thanks.
That's a limitation with mkcephfs, but not the monitors (and one reason why you hear tv complaining about mkcephfs every chance he gets). See http://ceph.com/docs/master/dev/mon-bootstrap/ for information about what the monitors care capable of. The old chef and crowbar stuff does something pretty simple (start with single monitor, then expand), while the new chef/juju stuff will be multiple mons up in parallel using the 'initial members' to describe the initial quorum candidates.