Ceph

We have two separate Ceph installations with five servers each.
Sometimes when a server is restarted the Ceph monitor on it does not
start automatically. The monitor is different each time and it does not
happen always.

We've upgraded from Cuttlefish to Dumpling and later to Emperor and it
seems that the issue is not totally resolved. Currently we are using
Emperor.

We've tried several ways [1] [2] [3] to to bring the monitor back up.
Usually only recreating the monitor helped. [3]

In Emperor usually the monitor can be started/restarted manually with:

sudo initctl start ceph-mon cluster=ceph id=_ceph_server_hostname

OS: Ubuntu 12.04
Ceph version: cuttlefish, dumpling, emperor
Kernel version: 3.5.x

[1]
sudo initctl start ceph-mon cluster=cluster_name id=nowhere-cmp-05

[2]
sudo restart ceph-mon-all
sudo initctl restart ceph-mon cluster=cluster_name id=nowhere-cmp-05

[3]
sudo initctl stop ceph-mon cluster=cluster_name id=nowhere-cmp-04
sudo ceph mon remove nowhere-cmp-04
sudo mv /var/lib/ceph/mon/ceph-nowhere-cmp-04/ /var/lib/ceph/mon/ceph-nowhere-cmp-04.bak
sudo mkdir /var/lib/ceph/mon/ceph-nowhere-cmp-04
sudo ceph auth get mon. -o /tmp/auth
sudo ceph mon getmap -o /tmp/map
sudo ceph-mon -i nowhere-cmp-04 --mkfs --monmap /tmp/map --keyring /tmp/auth
sudo ceph mon add nowhere-cmp-04 10.16.0.107:6789
sudo ceph-mon -i nowhere-cmp-04 --public-addr 10.16.0.107:6789