Project

General

Profile

Bug #46626

The bandwidth of bluestore was throttled

Added by xinwei wei 16 days ago. Updated 15 days ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
nautilus,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

We have a ceph cluster(v 14.2.9) with 3 hosts, and there are 6 HDD OSDs on each host.
An EC pool was created with the default erasure-code-profile(k=2,m=1), and nothing changes in ceph.conf

After we use 'rados bench 30 write -b 2M -t 256 -p ecpool clean-up' to test the bandwidth, and use iostat to monitor disk utilization. Got the results: 30MB/s per disk.
With osd perf dump we found lots throttle-bluestore_throttle_bytes, it seems that BlueStore.cc::queue_transactions() was blocked in throttle_bytes.get(txc->cost);

Then we open bluestore log and rerun the aboving test. In ceph-osd.log, appeared some logs "_txc_calc_cost 0x564c308b0b00 cost 173240518 (257 ios * 670000 + 1050518 bytes)". It seems the transactions packaged by ECBackend consist of many address discontinuity 4K IO, but these IO will be write once to kernel by pwritev.


Related issues

Copied to Ceph - Backport #46628: nautilus: The bandwidth of bluestore was throttled New
Copied to Ceph - Backport #46629: octopus: The bandwidth of bluestore was throttled New

History

#1 Updated by xinwei wei 16 days ago

We have a ceph cluster(v 14.2.9) with 3 hosts, and there are 6 HDD OSDs on each host.
An EC pool was created with the default erasure-code-profile(k=2,m=1), and nothing changes in ceph.conf

After we use 'rados bench 30 write -b 2M -t 256 -p ecpool clean-up' to test the bandwidth, and use iostat to monitor disk utilization. Got the results: 30MB/s per disk.
With osd perf dump we found lots throttle-bluestore_throttle_bytes, it seems that BlueStore.cc::queue_transactions() was blocked in throttle_bytes.get(txc->cost);

Then we open bluestore log and rerun the aboving test. In ceph-osd.log, appeared some logs "_txc_calc_cost 0x564c308b0b00 cost 173240518 (257 ios * 670000 + 1050518 bytes)". It seems the transactions packaged by ECBackend consist of many address discontinuity 4K IO, but these IO will be write once to kernel by pwritev.

#2 Updated by xinwei wei 16 days ago

The right "Affected Versions" should be "v14.2.10"

#3 Updated by Kefu Chai 16 days ago

  • Status changed from New to Pending Backport

#4 Updated by Nathan Cutler 15 days ago

  • Copied to Backport #46628: nautilus: The bandwidth of bluestore was throttled added

#5 Updated by Nathan Cutler 15 days ago

  • Copied to Backport #46629: octopus: The bandwidth of bluestore was throttled added

#7 Updated by Nathan Cutler 15 days ago

  • Backport changed from luminous,nautilus,octopus to nautilus,octopus

luminous is EOL

Also available in: Atom PDF