Ceph » devops

I've been trying to debug this problem that shows up when using ceph-deploy, and have found some pretty weird behavior:

[ceph_deploy.gatherkeys][WARNIN] Unable to find /etc/ceph/ceph.client.admin.keyring on ['issdm-2']

The below findings are on the monitor node after the failed gatherkeys.

The ceph-create-keys upstart script is run when ceph-mon starts, but there is an error in syslog:

Jan 24 10:58:38 issdm-2 kernel: [71327.813051] init: ceph-create-keys main process (26591) terminated with status 1

There is more info in /var/log/upstart/ceph-create-keys.log

INFO:ceph-create-keys:Key exists already: /etc/ceph/ceph.client.admin.keyring
INFO:ceph-create-keys:Key exists already: /var/lib/ceph/bootstrap-osd/ceph.keyring
INFO:ceph-create-keys:Key exists already: /var/lib/ceph/bootstrap-mds/ceph.keyring
INFO:ceph-create-keys:Talking to monitor...
Error initializing cluster client: Error
INFO:ceph-create-keys:Cannot get or create admin key, permission denied

However, there is still no ceph.client.admin.keyring in /etc/ceph.

Now I start and stop the monitor

sudo stop ceph-mon id=issdm-2
sudo start ceph-mon id=issdm-2

Now the following new output is in ceph-create-keys.log

no valid command found; 10 closest matches:
config show
help
log dump
get_command_descriptions
git_version
config set <var> <val> [<val>...]
version
2
config get <var>
0
admin_socket: invalid command
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Key exists already: /var/lib/ceph/bootstrap-osd/ceph.keyring
INFO:ceph-create-keys:Key exists already: /var/lib/ceph/bootstrap-mds/ceph.keyring

But the ceph.client.admin.key happens to now be in /etc/ceph/...