Project

General

Profile

Actions
Copy link

Support #23254

closed

Clinet side write freeze for 15 Sec when one storage node rebooted

Added by Suvendu Mitra about 6 years ago. Updated about 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Tags:
Reviewed:
Affected Versions:
v12.2.2
Pull request ID:

Description

I am running ceph version 12.2.2, with bluestore. In my system I am running 3 storage nodes with 2 OSD each. Monitors and Mgr running on 3 controller

A. Test details -
1. Launch a fedora-24 VM. start writing with dd on attached cinder volume. and other window monitor the resultant file size.
2. Restart one storage node @ Tue Mar 6 14:22:59 EET 2018

B. Observation
1. File size is constant for 15 sec, after that size grows

Also tried with osd_heartbeat_grace = 2 and osd_mon_heartbeat_interval = 5 but write freeze time doesn't improve

System settings

[root@controller-1 ~]# sudo ceph -s
cluster:
id: 08762c5c-52fd-4d29-91ec-987d6ece068e
health: HEALTH_OK

services:
    mon: 3 daemons, quorum controller-1,controller-2,controller-3
    mgr: controller-3(active), standbys: controller-1, controller-2
    osd: 6 osds: 6 up, 6 in
data:
    pools:   4 pools, 736 pgs
    objects: 34581 objects, 134 GB
    usage:   275 GB used, 1959 GB / 2235 GB avail
    pgs:     736 active+clean

[root@controller-1 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 2.18271 root default
-5 0.72757 host storage-1
0 ssd 0.36378 osd.0 up 1.00000 1.00000
3 ssd 0.36378 osd.3 up 1.00000 1.00000
-3 0.72757 host storage-2
1 ssd 0.36378 osd.1 up 1.00000 1.00000
4 ssd 0.36378 osd.4 up 1.00000 1.00000
-7 0.72757 host storage-3
2 ssd 0.36378 osd.2 up 1.00000 1.00000
5 ssd 0.36378 osd.5 up 1.00000 1.00000

Files

Download all files
storage_3.tar.gz (257 KB) storage_3.tar.gz Ceph log from storage-3 Suvendu Mitra, 03/07/2018 08:35 AM
controller-1.tar.gz (951 KB) controller-1.tar.gz Ceph log from controller-1 Suvendu Mitra, 03/07/2018 08:37 AM
ceph_w.txt (23.7 KB) ceph_w.txt Suvendu Mitra, 03/19/2018 08:55 AM
Actions
Copy link

Also available in: Atom PDF