Project

General

Profile

Bug #46626

The bandwidth of bluestore was throttled

Added by xinwei wei over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
nautilus,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We have a ceph cluster(v 14.2.9) with 3 hosts, and there are 6 HDD OSDs on each host.
An EC pool was created with the default erasure-code-profile(k=2,m=1), and nothing changes in ceph.conf

After we use 'rados bench 30 write -b 2M -t 256 -p ecpool clean-up' to test the bandwidth, and use iostat to monitor disk utilization. Got the results: 30MB/s per disk.
With osd perf dump we found lots throttle-bluestore_throttle_bytes, it seems that BlueStore.cc::queue_transactions() was blocked in throttle_bytes.get(txc->cost);

Then we open bluestore log and rerun the aboving test. In ceph-osd.log, appeared some logs "_txc_calc_cost 0x564c308b0b00 cost 173240518 (257 ios * 670000 + 1050518 bytes)". It seems the transactions packaged by ECBackend consist of many address discontinuity 4K IO, but these IO will be write once to kernel by pwritev.


Related issues

Copied to Ceph - Backport #46628: nautilus: The bandwidth of bluestore was throttled Resolved
Copied to Ceph - Backport #46629: octopus: The bandwidth of bluestore was throttled Resolved

History

#1 Updated by xinwei wei over 3 years ago

We have a ceph cluster(v 14.2.9) with 3 hosts, and there are 6 HDD OSDs on each host.
An EC pool was created with the default erasure-code-profile(k=2,m=1), and nothing changes in ceph.conf

After we use 'rados bench 30 write -b 2M -t 256 -p ecpool clean-up' to test the bandwidth, and use iostat to monitor disk utilization. Got the results: 30MB/s per disk.
With osd perf dump we found lots throttle-bluestore_throttle_bytes, it seems that BlueStore.cc::queue_transactions() was blocked in throttle_bytes.get(txc->cost);

Then we open bluestore log and rerun the aboving test. In ceph-osd.log, appeared some logs "_txc_calc_cost 0x564c308b0b00 cost 173240518 (257 ios * 670000 + 1050518 bytes)". It seems the transactions packaged by ECBackend consist of many address discontinuity 4K IO, but these IO will be write once to kernel by pwritev.

#2 Updated by xinwei wei over 3 years ago

The right "Affected Versions" should be "v14.2.10"

#3 Updated by Kefu Chai over 3 years ago

  • Status changed from New to Pending Backport

#4 Updated by Nathan Cutler over 3 years ago

  • Copied to Backport #46628: nautilus: The bandwidth of bluestore was throttled added

#5 Updated by Nathan Cutler over 3 years ago

  • Copied to Backport #46629: octopus: The bandwidth of bluestore was throttled added

#7 Updated by Nathan Cutler over 3 years ago

  • Backport changed from luminous,nautilus,octopus to nautilus,octopus

luminous is EOL

#8 Updated by Nathan Cutler over 3 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF