Project

General

Profile

Actions

Bug #21180

closed

Bluestore throttler causes down OSD

Added by Henrik Korkuc over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
OSD
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Writing large amount of data to EC RBD pool via NBD causes down OSDs, PGs and drop in traffic due to unhealthy cluster. OSDs themself are running, disks seem to be idle. In logs "heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7f05d4c90700' had timed out after 60" (I increased it from 15 during testing) and slow requests can be observed.

Setting bluestore_throttle_bytes to 0 resolves issue.

Attaching gdb thread backtrace of one of OSDs.


Files

gdb.txt.gz (6.59 KB) gdb.txt.gz Henrik Korkuc, 08/30/2017 03:23 PM

Related issues 1 (0 open1 closed)

Related to RADOS - Bug #21171: bluestore: aio submission deadlockResolvedSage Weil08/29/2017

Actions
Actions

Also available in: Atom PDF