Ceph » devops

I saw a similar behavior on a 12.04 (precise) installation. Here are my logs.

2013-05-07 17:21:17.546091 7f7ced601780 0 ceph version 0.61 (237f3f1e8d8c3b85666529860285dcdffdeda4c5), process ceph-mon, pid 1210
2013-05-07 17:21:18.571090 7f7ced601780 1 mon.issdm-44@-1(probing) e0 preinit fsid 32044951-1acc-468d-acb4-47aef08dd408
2013-05-07 17:21:18.571201 7f7ced601780 1 mon.issdm-44@-1(probing) e0 initial_members issdm-44-c, filtering seed monmap
2013-05-07 17:21:18.687516 7f7ced5ee700 0 -- 192.168.140.144:6789/0 >> 0.0.0.0:0/1 pipe(0x25b6280 sd=21 :0 s=1 pgs=0 cs=0 l=0).fault
2013-05-07 17:22:18.626053 7f7ce7ea8700 0 mon.issdm-44@-1(probing).data_health(0) update_stats avail 85% total 235530756 used 22536532 avail 201203736
2013-05-07 17:23:18.626187 7f7ce7ea8700 0 mon.issdm-44@-1(probing).data_health(0) update_stats avail 85% total 235530756 used 22536536 avail 201203732

Although, now that I look at it, my logs do have this line:
2013-05-07 17:21:18.687516 7f7ced5ee700 0 -- 192.168.140.144:6789/0 >> 0.0.0.0:0/1 pipe(0x25b6280 sd=21 :0 s=1 pgs=0 cs=0 l=0).fault

which does indicate a fault of some sort. I'll uninstall/reinstall and see if that behavior changes.

I hate to kick a dead horse, but did this make it into 0.63 or will it be available in a later release? Ran into this on Scientific Linux last night. Was able to get around it by manually starting ceph-mon on my three mon nodes and then waiting for quorum before creating keys. I assume it is the same race.

This fix landed in 0.61.1. Please try that (or a newer) version and see if you're still hitting it.

:/

0.61.2

[root@test-ceph-1001 ~]# yum list ceph
Loaded plugins: security
Installed Packages
ceph.x86_64 0.61.2-0.el6 @ceph

Scientific Linux 6.4

To reproduce, I installed ceph using ceph-deploy. Ran ceph-deploy new. Added MONs on all three of the nodes. Then running ceph-deploy gatherkeys saw the same error message above.

The ceph-create-keys process was running indefinitely on each of the three machines in my cluster. I probably should have straced it or something useful, but then I found this bug and figured out the way around it.

If I can provide any additional information, logs, help, reproducing with configuration options, etc, please let me know.

Okay so I tried duplicating this again today. And now I can't. I think it was due to an iptables issue at first, but was able to duplicate even after setting iptables to have a default allow policy. But now, no matter what I try I can't duplicate it. The first mon created takes a while to return, but once the cluster is in quorum it returns as expected.

So, nevermind.

I am seeing this same problem. I am using the latest master version of ceph-deploy, and the target node is Ubuntu 12.10.

After `mon create`, `gatherkeys` reports that it cannot find keys. I see that the `ceph-create-keys` process is still running after `mon create` returns.

nwatkins@kyoto:~$ ps aux | grep ceph
root     11847  0.0  0.0  35616  7040 ?        Ss   11:13   0:00 /usr/bin/python /usr/sbin/ceph-create-keys --cluster=ceph -i kyoto

Actually, this looks like it is being caused by the monitor crashing, and ceph-create-keys not being able to connect to admin socket. I submitted an error report on the monitor issue to the mailing list.

I'm still seeing this with the latest cuttlefish on ubuntu 13.04.

after doing ceph-deploy install ceph-dmon1 and then ceph-deploy mon create ceph-dmon1, gatherkeys fails.

logging in to ceph-dmon1 this is what I see:
root@ceph-dmon1:~# ps faux | grep ceph
root 9010 0.0 0.0 9436 912 pts/2 S+ 16:14 0:00 \_ grep --color=auto ceph
root 8764 0.1 0.0 132756 5972 ? Ssl 16:14 0:00 /usr/bin/ceph-mon --cluster=ceph -i ceph-dmon1 -f
root 8765 0.5 0.0 33808 7096 ? Ss 16:14 0:00 /usr/bin/python /usr/sbin/ceph-create-keys --cluster=ceph -i ceph-dmon1
root@ceph-dmon1:~#

So after reinstalling the server, this went away. Next time I run into this, I'll update.

runing ceph-create-keys manually gives:

INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
repeated ad-infinitum.

I can confirm the broken ceph-deploy / ceph-create-keys

I like to get familiar with ceph so I'm running an automatically freshly installed uptodate raring ringtail system.
Than I would like the following code to be executed from one of the mon systems ('nike' in this case)

------------------------------------------------------------------------

#!/bin/bash
# (c) bglomm@ecologic.eu 2013
# initialize the ceph cluster
#

# our ceph systems
ceph_osds="ping pong" 
ceph_mons="nike ping pong king kong" 
ceph_mds="nike king kong" 

# prepare them
#for host in $ceph_mons; do
#   getent hosts $ceph_mons | cut -f-4 -d\.| ssh cephmngmnt@$host sudo cat \>\> /etc/hosts
#done
# that has been done during installation meanwhile

# install ceph
ceph-deploy install $ceph_mons

# deploy ceph
ceph-deploy new $ceph_mons

# create the monitors
ceph-deploy --overwrite-conf mon create $ceph_mons

# get the keys
for host in $ceph_mons; do
    ceph-deploy gatherkeys $host
done

for host in $ceph_osds;do
    ceph-deploy osd create $host:/dev/sdb2
done

The first error is the missing keys when try to gather them
taking a look at the hosts, on all five there is the create-key process

/usr/bin/python /usr/sbin/ceph-create-keys --cluster=ceph -i ping

running infinetly, no keys are created.

all /var/lib/ceph/bootstrap-*
are empty
nothing else (ceph -s etc) works since:
ERROR: missing keyring, cannot use cephx for authentication

btw:
/usr/sbin/ceph-create-keys is starting automatically after a reboot again,
service ceph restart doesn't work at all?

................................

root@ping[/0]:~ # ps ax | egrep ceph
 3257 ?        Ssl    0:03 /usr/bin/ceph-mon --cluster=ceph -i ping -f
 3258 ?        Ss     0:03 /usr/bin/python /usr/sbin/ceph-create-keys --cluster=ceph -i ping
17774 pts/0    S+     0:00 egrep --color=auto ceph
root@ping[/0]:~ # service ceph restart
root@ping[/0]:~ # ps ax | egrep ceph
 3257 ?        Ssl    0:03 /usr/bin/ceph-mon --cluster=ceph -i ping -f
 3258 ?        Ss     0:03 /usr/bin/python /usr/sbin/ceph-create-keys --cluster=ceph -i ping
17816 pts/0    S+     0:00 egrep --color=auto ceph
root@ping[/0]:~ # kill -9 3257 3258 && sleep 5
root@ping[/0]:~ # ps ax | egrep ceph
17998 ?        Ss     0:00 /bin/sh -e -c /usr/bin/ceph-mon --cluster="${cluster:-ceph}" -i "$id" -f /bin/sh
17999 ?        Sl     0:00 /usr/bin/ceph-mon --cluster=ceph -i ping -f
18000 ?        Ss     0:00 /bin/sh -e -c /usr/sbin/ceph-create-keys --cluster="${cluster:-ceph}" -i "${id:-$(hostname)}" /bin/sh
18001 ?        S      0:00 /usr/bin/python /usr/sbin/ceph-create-keys --cluster=ceph -i ping
18073 pts/0    S+     0:00 egrep --color=auto ceph

Anybody any ideas how to deploy ceph at the moment

best regards

Bernhard

P.S.: 
root@nike[/1]:~ # ceph-deploy --version
1.2.1
root@nike[/1]:~ # ceph --version
ceph version 0.67 (e3b7bc5bce8ab330ec1661381072368af3c218a0)

bernhard: i think the problem in your case is that you have old keyrings in /etc/ceph from prior cluster instances. can you add, at the top of your script before the install,

ceph-deploy purge $all
ceph-dpeloy purgedata $all

to blow away /etc/ceph and /var/lib/ceph contents.

Hi,

I'm hitting the same bug on red hat 6.3 (Santiago), purging /var/lib/ceph and /etc/ceph doesn't help.

Getting the same thing on a fresh install of CentOS 6.4

[## ceph-cluster]# ceph-deploy mon create ##
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ##
[ceph_deploy.mon][DEBUG ] detecting platform for host ## ...
[ceph_deploy.mon][INFO  ] distro info: CentOS 6.4 Final
[##][DEBUG ] deploying mon to ##
[##][DEBUG ] remote hostname: ##
[##][INFO  ] write cluster configuration to /etc/ceph/{cluster}.conf
[##][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-##
[##][INFO  ] create a done file to avoid re-doing the mon deployment
[##][INFO  ] create the init path if it does not exist
[##][INFO  ] locating `service` executable...
[##][INFO  ] found `service` executable: /sbin/service
[##][INFO  ] Running command: /sbin/service ceph start mon.##
[root@## ceph-cluster]# ceph-deploy gatherkeys ##
[ceph_deploy.gatherkeys][DEBUG ] Checking ## for /etc/ceph/ceph.client.admin
[ceph_deploy.gatherkeys][WARNIN] Unable to find /etc/ceph/ceph.client.admin.keyring on ['##']
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Checking ## for /var/lib/ceph/bootstrap-osd
[ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring on ['##']
[ceph_deploy.gatherkeys][DEBUG ] Checking ## for /var/lib/ceph/bootstrap-mds
[ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['##']
[root@## ceph-cluster]# ceph-deploy --version
1.2.1
[root@## ceph-cluster]# ceph --version
ceph version 0.67 (e3b7bc5bce8ab330ec1661381072368af3c218a0)
[root@## ceph-cluster]#  /usr/bin/python /usr/sbin/ceph-create-keys -i ##
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'

Michael Potter wrote:

Getting the same thing on a fresh install of CentOS 6.4

[...]

Can you post the output of 'ceph daemon mon.### mon_status', and contents of /etc/ceph/ceph.conf?

Thanks!

I think the documentation is a little confusing. I had the same problems minutes ago I fixed it. In my escenario I have two nodes right now, ceph-admin and ceph-server-01.

Steps that failed:
root@ceph-admin:~/mycluster# ceph-deploy new ceph-admin
root@ceph-admin:~/mycluster# ceph-deploy install ceph-server-01
root@ceph-admin:~/mycluster# ceph-deploy mon create ceph-server-01
root@ceph-admin:~/mycluster# ceph-deploy gatherkeys ceph-server-01
Unable to find /etc/ceph/ceph.client.admin.keyring on ['ceph-server-01']
Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring on ['ceph-server-01']
Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['ceph-server-01']

To clean everything:
root@ceph-admin:~/mycluster# ceph-deploy purge ceph-server-01
root@ceph-admin:~/mycluster# ceph-deploy purgedata ceph-server-01

Steps that worked:
root@ceph-admin:~/mycluster# ceph-deploy new ceph-server-01
root@ceph-admin:~/mycluster# ceph-deploy install ceph-server-01
root@ceph-admin:~/mycluster# ceph-deploy mon create ceph-server-01
root@ceph-admin:~/mycluster# ceph-deploy gatherkeys ceph-server-01

Documentation isn't very clear about that we should specify as node when we execute "new".

What do you think? https://github.com/ceph/ceph/pull/510

Diego Woitasen wrote:

What do you think? https://github.com/ceph/ceph/pull/510

aha, i bet this is what is tripping up a lot of people. it's specifically the mon nodes that need to be named in the new step. i'll tweak the docs!

Sage Weil wrote:

Michael Potter wrote:

Getting the same thing on a fresh install of CentOS 6.4

[...]

Can you post the output of 'ceph daemon mon.### mon_status', and contents of /etc/ceph/ceph.conf?

Thanks!

Cheers Sage, hoping it's just an obvious trap some of us newcomers are falling into. Tried with and without osd/mon settings in the conf (Not got to the stage of making some osd's yet anyway!).
The "ceph-mon is not in quorum" error looks like it should be so obvious to resolve but when there's only one node I'm not sure how that's even possible.

[root@#subdomain-identifier# ceph-cluster]# ceph daemon mon.#subdomain-identifier# mon_status
{ "name": "#subdomain-identifier#",
  "rank": 0,
  "state": "leader",
  "election_epoch": 1,
  "quorum": [
        0],
  "outside_quorum": [],
  "extra_probe_peers": [],
  "sync_provider": [],
  "monmap": { "epoch": 1,
      "fsid": "f40acb72-33fb-43a9-bf62-691bb047f0b6",
      "modified": "0.000000",
      "created": "0.000000",
      "mons": [
            { "rank": 0,
              "name": "#subdomain-identifier#",
              "addr": "[::1]:6789\/0"}]}}

[root@#subdomain-identifier# ceph-cluster]# cat /etc/ceph/ceph.conf
[mon.a]
mon_addr = #subdomain-identifier#.#resolveable-domain#
host = #subdomain-identifier#

[osd.0]
host = #subdomain-identifier#

[global]
filestore_xattr_use_omap = true
mon_host = #ipaddr#
osd_journal_size = 1024
mon_initial_members = #subdomain-identifier#
auth_supported = cephx
osd_crush_chooseleaf_type = 0
fsid = 77cdf265-09d1-42a6-98b9-56a39fdefea1

[osd.1]
host = #subdomain-identifier#

[osd.2]
host = #subdomain-identifier#

Also -

Nice ideas Diego, I'm still on the 'Get a single node running Ceph' stage of usage though so there might be more than one way to cause this one (from my failed google-fu on solving the issue that seems quite probable) .

Ah, I think the problem is

{ "rank": 0,
              "name": "#subdomain-identifier#",
              "addr": "[::1]:6789\/0"}]}}

It needs to be the IP address that matches the mon addr line. My first guess is that there is an /etc/hosts entry on the mon machine that maps the host.domain to [::1]. Try changing that to whatever dns resolves to?

Hi Sage, took everything out of the host for except for #ipaddr# #subdomain-identifier#.#resolveable-domain#
Cleaned and re-installed everything. Now getting the below output.

[root@#subdomain-identifier# ceph-cluster]# ceph daemon mon.#subdomain-identifier# mon_status
{ "name": "#subdomain-identifier#",
  "rank": 0,
  "state": "leader",
  "election_epoch": 1,
  "quorum": [
        0],
  "outside_quorum": [],
  "extra_probe_peers": [],
  "sync_provider": [],
  "monmap": { "epoch": 1,
      "fsid": "77cdf265-09d1-42a6-98b9-56a39fdefea1",
      "modified": "0.000000",
      "created": "0.000000",
      "mons": [
            { "rank": 0,
              "name": "#subdomain-identifier#",
              "addr": "#ipaddr#:6789\/0"}]}}

[root@#subdomain-identifier# ceph-cluster]# cat /etc/ceph/ceph.conf
[mon.a]
mon_addr = #ipaddr#:6789
...

Noticed I was now getting OSD warnings on service ceph restart, huzzah progress! But it still wasn't working...

[root@#subdomain-identifier# ceph-cluster]# ps ax | egrep ceph
 4570 pts/0    Sl     0:00 /usr/bin/ceph-mon -i #subdomain-identifier# --pid-file /var/run/ceph/mon.#subdomain-identifier#.pid -c /etc/ceph/ceph.conf
 4794 pts/0    S      0:00 /usr/bin/python /usr/sbin/ceph-create-keys -i a
 5768 pts/0    S+     0:00 egrep ceph
[root@#subdomain-identifier# ceph-cluster]# ceph-create-keys -i #subdomain-identifier#.#resolveable-domain#
admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.

Hmm...
Starting to look a bit more like [[http://tracker.ceph.com/issues/5777]] now.

[root@#subdomain-identifier# ceph-cluster]# ls /var/run/ceph/
ceph-mon.#subdomain-identifier#.asok  mon.#subdomain-identifier#.pid

[root@#subdomain-identifier# ceph-cluster]# ceph --admin-daemon /var/run/ceph/ceph-mon.#subdomain-identifier#.asok mon_status
Returns the same as ceph daemon mon.#subdomain-identifier# mon_status

Michael Potter wrote:

Hi Sage, took everything out of the host for except for #ipaddr# #subdomain-identifier#.#resolveable-domain#
Cleaned and re-installed everything. Now getting the below output.

[...]

Noticed I was now getting OSD warnings on service ceph restart, huzzah progress!

Yay! So just to confirm, the problem was that /etc/hosts has the loopback addr?

But it still wasn't working...

[...]

Hmm...
Starting to look a bit more like [[http://tracker.ceph.com/issues/5777]] now.

[...]

[root@#subdomain-identifier# ceph-cluster]# ceph --admin-daemon /var/run/ceph/ceph-mon.#subdomain-identifier#.asok mon_status
Returns the same as ceph daemon mon.#subdomain-identifier# mon_status

What do you get if you run

ceph-create-keys -v -i `hostname` ?

Hi Sage, sorted it just before your reply. Was idly scrolling back through the thread when I spotted the word 'iptables' and smacked my head on the desk.

[root@#subdomain-identifier# ceph-cluster]# ceph-deploy gatherkeys #subdomain-identifier#.#resolveable-domain#
[ceph_deploy.gatherkeys][DEBUG ] Checking #subdomain-identifier#.#resolveable-domain# for /etc/ceph/ceph.client.admin.keyring
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from #subdomain-identifier#.#resolveable-domain#.
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Checking #subdomain-identifier#.#resolveable-domain# for /var/lib/ceph/bootstrap-osd/ceph.keyring
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-osd.keyring key from #subdomain-identifier#.#resolveable-domain#.
[ceph_deploy.gatherkeys][DEBUG ] Checking #subdomain-identifier#.#resolveable-domain# for /var/lib/ceph/bootstrap-mds/ceph.keyring
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-mds.keyring key from #subdomain-identifier#.#resolveable-domain#.

Sage Weil wrote:

bernhard: i think the problem in your case is that you have old keyrings in /etc/ceph from prior cluster instances. can you add, at the top of your script before the install,

ceph-deploy purge $all
ceph-dpeloy purgedata $all

to blow away /etc/ceph and /var/lib/ceph contents.

thnx Sage, but fresh install is fresh install ;-).
Install as in install, not image replay.
ceph-create-keys just runs on all mons without end, without doing anything...
Is there another way to create the keys?
Haven't found a straight howto for deploying the cluster without ceph-deploy/mkcephfs yet,
could use my fai/cfengine to do the job is I know what needs to be done.

bernhard glomm wrote:

Sage Weil wrote:

bernhard: i think the problem in your case is that you have old keyrings in /etc/ceph from prior cluster instances. can you add, at the top of your script before the install,

ceph-deploy purge $all
ceph-dpeloy purgedata $all

to blow away /etc/ceph and /var/lib/ceph contents.

thnx Sage, but fresh install is fresh install ;-).
Install as in install, not image replay.
ceph-create-keys just runs on all mons without end, without doing anything...

Can you post the contents of /etc/ceph/ceph.conf, /etc/hosts, and on a couple of mons the output from 'ceph --admin-daemon /var/run/ceph/ceph-mon.`hostname`.asok mon_status`? That should tell us what is going on.

Is there another way to create the keys?
Haven't found a straight howto for deploying the cluster without ceph-deploy/mkcephfs yet,
could use my fai/cfengine to do the job is I know what needs to be done.

Key creation can be done in a few different ways, but whatever is preventing ceph-create-keys from working may throw a wrench in other methods too, and I'd really like to get to the bottom of it :) Thanks!

Sage,

sorry for being late on this, other tasks kept me busy,
but here the infos you were asking for:

ceph-create-keys just runs on all mons without end, without doing anything...

Can you post the contents of /etc/ceph/ceph.conf, /etc/hosts, and on a couple of mons the output from 'ceph --admin-daemon /var/run/ceph/ceph-mon.`hostname`.asok mon_status`? That should tell us what is going on.

Is there another way to create the keys?
Haven't found a straight howto for deploying the cluster without ceph-deploy/mkcephfs yet,
could use my fai/cfengine to do the job is I know what needs to be done.

Key creation can be done in a few different ways, but whatever is preventing ceph-create-keys from working may throw a wrench in other methods too, and I'd really like to get to the bottom of it :) Thanks!

the results looks quite strange to me,
yes the 5 hosts names can be resolved back and forth through DNS
AND have /etc/hosts entries an all nodes:

# ceph --admin-daemon /var/run/ceph/ceph-mon.`hostname`.asok mon_status
# on:
# nike:
{ "name": "nike",
  "rank": 1,
  "state": "electing",
  "election_epoch": 1,
  "quorum": [],
  "outside_quorum": [],
  "extra_probe_peers": [
        "192.168.242.31:6789\/0",
        "192.168.242.32:6789\/0",
        "192.168.242.92:6789\/0",
        "192.168.242.93:6789\/0"],
  "sync_provider": [],
  "monmap": { "epoch": 0,
      "fsid": "65c1781b-95f0-40ec-b317-f2f0729a46ff",
      "modified": "0.000000",
      "created": "0.000000",
      "mons": [
            { "rank": 0,
              "name": "king",
              "addr": "192.168.242.31:6789\/0"},
            { "rank": 1,
              "name": "nike",
              "addr": "192.168.242.36:6789\/0"},
            { "rank": 2,
              "name": "ping",
              "addr": "192.168.242.92:6789\/0"},
            { "rank": 3,
              "name": "pong",
              "addr": "192.168.242.93:6789\/0"},
            { "rank": 4,
              "name": "kong",
              "addr": "0.0.0.0:0\/4"}]}}

# king
{ "name": "king",
  "rank": 0,
  "state": "electing",
  "election_epoch": 1,
  "quorum": [],
  "outside_quorum": [],
  "extra_probe_peers": [
        "192.168.242.32:6789\/0",
        "192.168.242.36:6789\/0",
        "192.168.242.92:6789\/0",
        "192.168.242.93:6789\/0"],
  "sync_provider": [],
  "monmap": { "epoch": 0,
      "fsid": "65c1781b-95f0-40ec-b317-f2f0729a46ff",
      "modified": "0.000000",
      "created": "0.000000",
      "mons": [
            { "rank": 0,
              "name": "king",
              "addr": "192.168.242.31:6789\/0"},
            { "rank": 1,
              "name": "ping",
              "addr": "192.168.242.92:6789\/0"},
            { "rank": 2,
              "name": "pong",
              "addr": "192.168.242.93:6789\/0"},
            { "rank": 3,
              "name": "nike",
              "addr": "0.0.0.0:0\/3"},
            { "rank": 4,
              "name": "kong",
              "addr": "0.0.0.0:0\/4"}]}}

# kong
{ "name": "kong",
  "rank": 0,
  "state": "electing",
  "election_epoch": 1,
  "quorum": [],
  "outside_quorum": [],
  "extra_probe_peers": [
        "192.168.242.31:6789\/0",
        "192.168.242.36:6789\/0",
        "192.168.242.92:6789\/0",
        "192.168.242.93:6789\/0"],
  "sync_provider": [],
  "monmap": { "epoch": 0,
      "fsid": "65c1781b-95f0-40ec-b317-f2f0729a46ff",
      "modified": "0.000000",
      "created": "0.000000",
      "mons": [
            { "rank": 0,
              "name": "kong",
              "addr": "192.168.242.32:6789\/0"},
            { "rank": 1,
              "name": "ping",
              "addr": "192.168.242.92:6789\/0"},
            { "rank": 2,
              "name": "pong",
              "addr": "192.168.242.93:6789\/0"},
            { "rank": 3,
              "name": "nike",
              "addr": "0.0.0.0:0\/3"},
            { "rank": 4,
              "name": "king",
              "addr": "0.0.0.0:0\/4"}]}}

# ping
{ "name": "ping",
  "rank": 1,
  "state": "electing",
  "election_epoch": 1,
  "quorum": [],
  "outside_quorum": [],
  "extra_probe_peers": [
        "192.168.242.31:6789\/0",
        "192.168.242.32:6789\/0",
        "192.168.242.36:6789\/0",
        "192.168.242.93:6789\/0"],
  "sync_provider": [],
  "monmap": { "epoch": 0,
      "fsid": "65c1781b-95f0-40ec-b317-f2f0729a46ff",
      "modified": "0.000000",
      "created": "0.000000",
      "mons": [
            { "rank": 0,
              "name": "nike",
              "addr": "192.168.242.36:6789\/0"},
            { "rank": 1,
              "name": "ping",
              "addr": "192.168.242.92:6789\/0"},
            { "rank": 2,
              "name": "pong",
              "addr": "192.168.242.93:6789\/0"},
            { "rank": 3,
              "name": "king",
              "addr": "0.0.0.0:0\/3"},
            { "rank": 4,
              "name": "kong",
              "addr": "0.0.0.0:0\/4"}]}}

# pong
{ "name": "pong",
  "rank": 2,
  "state": "electing",
  "election_epoch": 1,
  "quorum": [],
  "outside_quorum": [],
  "extra_probe_peers": [
        "192.168.242.31:6789\/0",
        "192.168.242.32:6789\/0",
        "192.168.242.36:6789\/0",
        "192.168.242.92:6789\/0"],
  "sync_provider": [],
  "monmap": { "epoch": 0,
      "fsid": "65c1781b-95f0-40ec-b317-f2f0729a46ff",
      "modified": "0.000000",
      "created": "0.000000",
      "mons": [
            { "rank": 0,
              "name": "nike",
              "addr": "192.168.242.36:6789\/0"},
            { "rank": 1,
              "name": "ping",
              "addr": "192.168.242.92:6789\/0"},
            { "rank": 2,
              "name": "pong",
              "addr": "192.168.242.93:6789\/0"},
            { "rank": 3,
              "name": "king",
              "addr": "0.0.0.0:0\/3"},
            { "rank": 4,
              "name": "kong",
              "addr": "0.0.0.0:0\/4"}]}}

# /etc/ceph/ceph.conf
[global]
fsid = 65c1781b-95f0-40ec-b317-f2f0729a46ff
mon_initial_members = ping, pong, nike, king, kong
mon_host = 192.168.242.92,192.168.242.93,192.168.242.36,192.168.242.31,192.168.242.32
auth_supported = cephx
osd_journal_size = 1024
filestore_xattr_use_omap = true

# files/folders created during ceph_deploy
# on ping=OSD

root@ping[/1]:~ # tree -A /var/lib/ceph/
/var/lib/ceph/
??? bootstrap-mds
??? bootstrap-osd
??? mds
??? mon
?   ??? ceph-ping
?       ??? done
?       ??? keyring
?       ??? store.db
?       ?   ??? 000005.sst
?       ?   ??? 000006.log
?       ?   ??? CURRENT
?       ?   ??? LOCK
?       ?   ??? LOG
?       ?   ??? LOG.old
?       ?   ??? MANIFEST-000004
?       ??? upstart
??? osd
??? tmp

root@ping[/0]:~ # tree -A /etc/ceph/
/etc/ceph/
??? ceph.conf
??? rbdmap

0 directories, 2 files

# on nike
/var/lib/ceph/
??? bootstrap-mds
??? bootstrap-osd
??? mds
??? mon
?   ??? ceph-nike
?       ??? done
?       ??? keyring
?       ??? store.db
?       ?   ??? 000005.sst
?       ?   ??? 000006.log
?       ?   ??? CURRENT
?       ?   ??? LOCK
?       ?   ??? LOG
?       ?   ??? LOG.old
?       ?   ??? MANIFEST-000004
?       ??? upstart
??? osd
??? tmp

8 directories, 10 files

root@nike[/0]:~ # tree -A /etc/ceph/
/etc/ceph/
??? ceph.conf
??? ceph.log
??? ceph.mon.keyring
??? rbdmap

0 directories, 4 files

Ooh, I think I know what this is. This is probably cuttlefish v0.61.7 or older, right? There is a fix in dumpling (and backported for v0.61.8 cuttlefish) that forces and internal refresh of the monmap and prevents this sort of hang-up during the initial quorum convergence.

Sage Weil wrote:

Ooh, I think I know what this is. This is probably cuttlefish v0.61.7 or older, right? There is a fix in dumpling (and backported for v0.61.8 cuttlefish) that forces and internal refresh of the monmap and prevents this sort of hang-up during the initial quorum convergence.

Sorry to dissapoint you:

root@nike[/0]:~ # ceph --version
ceph version 0.67.2 (eb4380dd036a0b644c6283869911d615ed729ac8)

Is there an alternativ howto to set the cluster up?
How can I generate the keys manually?

TIA

Sage Weil wrote:

Bernhard, thanks for those logs--I think I've identified the problem. Can you try with wip-4924 (based off of dumpling)? It will show up on gitbuilder.ceph.com shortly. Thanks!

sorry for my blankness ;-)
You got me two lines how I do that??

TIA

sorry, didn't realised on the first glance it was a normal package repository...
got it, will post results ASAP

Sage Weil wrote:

Bernhard, thanks for those logs--I think I've identified the problem. Can you try with wip-4924 (based off of dumpling)? It will show up on gitbuilder.ceph.com shortly. Thanks!

Hi Sage

wip-4924 seems to do the job,
now I run into another show stopper ;-)
OSD creation fails/disk preparation fails.

will have a look into that tomorrow,
here just the output of ceph-deploy

+ ceph-deploy -v gatherkeys atom02
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.client.admin.keyring
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.bootstrap-osd.keyring
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.bootstrap-mds.keyring
+ for host in '$ceph_osds'
+ ceph-deploy -v osd create ping:/dev/sdb
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ping:/dev/sdb:
[ceph_deploy.osd][DEBUG ] Deploying osd to ping
[ceph_deploy.osd][DEBUG ] Host ping is now ready for osd use.
[ceph_deploy.osd][DEBUG ] Preparing host ping disk /dev/sdb journal None activate True
[ceph_deploy.osd][ERROR ] ceph-disk-prepare --cluster ceph -- /dev/sdb returned 1

***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format.
***************************************************************

Information: Moved requested sector from 34 to 2048 in
order to align on 2048-sector boundaries.
Non-GPT disk; not saving changes. Use -g to override.

INFO:ceph-disk:Will colocate journal with data on /dev/sdb
ceph-disk: Error: Command '['sgdisk', '--new=2:0:1024M', '--change-name=2:ceph journal', '--partition-guid=2:81b63820-f2a9-433d-925d-d70b238292de', '--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--', '/dev/sdb']' returned non-zero exit status 3

[ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs

+ for host in '$ceph_osds'
+ ceph-deploy -v osd create pong:/dev/sdb
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks pong:/dev/sdb:
[ceph_deploy.osd][DEBUG ] Deploying osd to pong
[ceph_deploy.osd][DEBUG ] Host pong is now ready for osd use.
[ceph_deploy.osd][DEBUG ] Preparing host pong disk /dev/sdb journal None activate True
[ceph_deploy.osd][ERROR ] ceph-disk-prepare --cluster ceph -- /dev/sdb returned 1

***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format.
***************************************************************

Information: Moved requested sector from 34 to 2048 in
order to align on 2048-sector boundaries.
Non-GPT disk; not saving changes. Use -g to override.

INFO:ceph-disk:Will colocate journal with data on /dev/sdb
ceph-disk: Error: Command '['sgdisk', '--new=2:0:1024M', '--change-name=2:ceph journal', '--partition-guid=2:43eb9a2a-d315-4866-9fa7-e254ccc6d767', '--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--', '/dev/sdb']' returned non-zero exit status 3

[ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs

bernhard glomm wrote:

Sage Weil wrote:

Bernhard, thanks for those logs--I think I've identified the problem. Can you try with wip-4924 (based off of dumpling)? It will show up on gitbuilder.ceph.com shortly. Thanks!

Hi Sage

wip-4924 seems to do the job,
now I run into another show stopper ;-)
OSD creation fails/disk preparation fails.

will have a look into that tomorrow,
here just the output of ceph-deploy

[...]

add the --zap-disk argument to blow away the old partition table?

Sage Weil wrote:

thanks for helping test this!

better free software ;-)

add the --zap-disk argument to blow away the old partition table?

arggghhh, thnx

Yepp, now it works as expected.
Let's c what I can break next...

P.S.:

When does the fix will make it into the main branch (dumpling)?
I keep working with wip-4924 for now?

this will go into the next dumpling point release. thanks again for helping track it down!

Hi All,
I am still hitting this in RHEL 6.4 with latest dumpling release (ceph-0.67.3-0.el6.x86_64).
For my setup, I am trying to create 3 MONs, mon creation goes through but create-keys hangs indefinitely.

Regards,
Abhay

I have just managed to deploy 3 ceph nodes successfully in Ubuntu Raring using Dumpling 0.67.3. The above issue looks a lot like an issue I hit a while back whereby gather-keys was failing following mon creation. I resolved this issue by ensuring that the mons were in quorum before executing gather-keys and I no longer see this issue.