Bug #46626
closedThe bandwidth of bluestore was throttled
0%
Description
We have a ceph cluster(v 14.2.9) with 3 hosts, and there are 6 HDD OSDs on each host.
An EC pool was created with the default erasure-code-profile(k=2,m=1), and nothing changes in ceph.conf
After we use 'rados bench 30 write b 2M -t 256 -p ecpool clean-up' to test the bandwidth, and use iostat to monitor disk utilization. Got the results: 30MB/s per disk.>cost);
With osd perf dump we found lots throttle-bluestore_throttle_bytes, it seems that BlueStore.cc::queue_transactions() was blocked in throttle_bytes.get(txc
Then we open bluestore log and rerun the aboving test. In ceph-osd.log, appeared some logs "_txc_calc_cost 0x564c308b0b00 cost 173240518 (257 ios * 670000 + 1050518 bytes)". It seems the transactions packaged by ECBackend consist of many address discontinuity 4K IO, but these IO will be write once to kernel by pwritev.
Updated by xinwei wei almost 4 years ago
We have a ceph cluster(v 14.2.9) with 3 hosts, and there are 6 HDD OSDs on each host.
An EC pool was created with the default erasure-code-profile(k=2,m=1), and nothing changes in ceph.conf
After we use 'rados bench 30 write b 2M -t 256 -p ecpool clean-up' to test the bandwidth, and use iostat to monitor disk utilization. Got the results: 30MB/s per disk.>cost);
With osd perf dump we found lots throttle-bluestore_throttle_bytes, it seems that BlueStore.cc::queue_transactions() was blocked in throttle_bytes.get(txc
Then we open bluestore log and rerun the aboving test. In ceph-osd.log, appeared some logs "_txc_calc_cost 0x564c308b0b00 cost 173240518 (257 ios * 670000 + 1050518 bytes)". It seems the transactions packaged by ECBackend consist of many address discontinuity 4K IO, but these IO will be write once to kernel by pwritev.
Updated by xinwei wei almost 4 years ago
The right "Affected Versions" should be "v14.2.10"
Updated by Kefu Chai almost 4 years ago
- Status changed from New to Pending Backport
Updated by Nathan Cutler almost 4 years ago
- Copied to Backport #46628: nautilus: The bandwidth of bluestore was throttled added
Updated by Nathan Cutler almost 4 years ago
- Copied to Backport #46629: octopus: The bandwidth of bluestore was throttled added
Updated by Nathan Cutler almost 4 years ago
- Backport changed from luminous,nautilus,octopus to nautilus,octopus
luminous is EOL
Updated by Nathan Cutler over 3 years ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".