Ceph » RADOS

Trying to check the health status on a Ceph node:

$ sudo ceph health
2015-08-20 12:34:42.682036 7f10df6a3700  0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError

The contents of /etc/ceph seems correct to me:

$ ls /etc/ceph/
ceph.client.admin.keyring  ceph.conf  rbdmap

The cluster was created with ceph-deploy, three nodes, with three monitors in quorum:

vagrant@node1:~$ sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
{
    "name": "node1",
    "rank": 0,
    "state": "leader",
    "election_epoch": 6,
    "quorum": [
        0,
        1,
        2
    ],
    "outside_quorum": [],
    "extra_probe_peers": [
        "192.168.111.101:6789\/0",
        "192.168.111.102:6789\/0" 
    ],
    "sync_provider": [],
    "monmap": {
        "epoch": 1,
        "fsid": "cd0ef43d-5228-4189-8bcc-130b84e1c850",
        "modified": "0.000000",
        "created": "0.000000",
        "mons": [
            {
                "rank": 0,
                "name": "node1",
                "addr": "192.168.111.100:6789\/0" 
            },
            {
                "rank": 1,
                "name": "node2",
                "addr": "192.168.111.101:6789\/0" 
            },
            {
                "rank": 2,
                "name": "node3",
                "addr": "192.168.111.102:6789\/0" 
            }
        ]
    }
}

Not being able to tell what is going on, tailing the logs hint at the possible problem:

==> /var/log/ceph/ceph-mon.node1.log <==
2015-08-20 12:29:00.633779 7fef19594700  0 cephx server client.admin:  unexpected key: req.key=d70f9358ddd0245d expected_key=1b1c8ba3b3f3bc04

Expectation: The CLI should at least hint to what is the problem with authentication. If cephx is enabled and the keys don't match, or if permissions are wrong, or a file is missing. Similar to the work that was done for #10877