Ceph

Hello,

I have the following setup:
- 5 servers with 22 HDD disk (as OSD);
- 5 monitor daemons running on the same servers;
- 3 mds daemons (1 active and 2 standby);
- Linux OS on all servers - ubuntu 16.04.05 with kernel 4.15.0-43-generic -
- ceph cluster mimic 13.2.4

The crush map is :

ceph osd crush rule dump
[
    {
        "rule_id": 0,
        "rule_name": "replicated_rule",
        "ruleset": 0,
        "type": 1,
        "min_size": 1,
        "max_size": 10,
        "steps": [
            {
                "op": "take",
                "item": -1,
                "item_name": "default" 
            },
            {
                "op": "chooseleaf_firstn",
                "num": 0,
                "type": "host" 
            },
            {
                "op": "emit" 
            }
        ]
    }
]

My ceph.conf file looks like this on all ceph servers:

### Ceph SATA Cluster ###

[global]
fsid = 98295168-4f61-4660-82bf-94383ef747cf

public_network = 172.24.152.0/21
cluster_network = 172.24.160.0/24

mon_initial_members = ceph-hyperstore101-sata
mon_host = 172.24.152.33, 172.24.152.34, 172.24.152.35, 172.24.152.36, 172.24.152.37

auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

### End config file ###

and my conf file looks like this on clients (I am using RBD)

[global]
fsid = 98295168-4f61-4660-82bf-94383ef747cf

public_network = 172.24.152.0/21
cluster_network = 172.24.160.0/24
mon_initial_members = ceph-hyperstore101-sata
mon_host = 172.24.152.33
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

now.... if I power-off one server the entire cluster becomes unavailable and all the clients with RBD (I am using only RBD at the moment) cannot read data from mounted disk in RBD. The pool replication for RBD is 2 (one data and one copy).

I have checked the see if the data on OSD1 si replicated on a OSD44 on a different server for example so this is OK.

Am I missing something ?