Support #23254
closedClinet side write freeze for 15 Sec when one storage node rebooted
0%
Description
I am running ceph version 12.2.2, with bluestore. In my system I am running 3 storage nodes with 2 OSD each. Monitors and Mgr running on 3 controller
A. Test details -
1. Launch a fedora-24 VM. start writing with dd on attached cinder volume. and other window monitor the resultant file size.
2. Restart one storage node @ Tue Mar 6 14:22:59 EET 2018
B. Observation
1. File size is constant for 15 sec, after that size grows
Also tried with osd_heartbeat_grace = 2 and osd_mon_heartbeat_interval = 5 but write freeze time doesn't improve
System settings
[root@controller-1 ~]# sudo ceph -s
cluster:
id: 08762c5c-52fd-4d29-91ec-987d6ece068e
health: HEALTH_OK
services:
mon: 3 daemons, quorum controller-1,controller-2,controller-3
mgr: controller-3(active), standbys: controller-1, controller-2
osd: 6 osds: 6 up, 6 in
data:
pools: 4 pools, 736 pgs
objects: 34581 objects, 134 GB
usage: 275 GB used, 1959 GB / 2235 GB avail
pgs: 736 active+clean
[root@controller-1 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 2.18271 root default
-5 0.72757 host storage-1
0 ssd 0.36378 osd.0 up 1.00000 1.00000
3 ssd 0.36378 osd.3 up 1.00000 1.00000
-3 0.72757 host storage-2
1 ssd 0.36378 osd.1 up 1.00000 1.00000
4 ssd 0.36378 osd.4 up 1.00000 1.00000
-7 0.72757 host storage-3
2 ssd 0.36378 osd.2 up 1.00000 1.00000
5 ssd 0.36378 osd.5 up 1.00000 1.00000
Files