Bug #46626
The bandwidth of bluestore was throttled
0%
Description
We have a ceph cluster(v 14.2.9) with 3 hosts, and there are 6 HDD OSDs on each host.
An EC pool was created with the default erasure-code-profile(k=2,m=1), and nothing changes in ceph.conf
After we use 'rados bench 30 write -b 2M -t 256 -p ecpool clean-up' to test the bandwidth, and use iostat to monitor disk utilization. Got the results: 30MB/s per disk.
With osd perf dump we found lots throttle-bluestore_throttle_bytes, it seems that BlueStore.cc::queue_transactions() was blocked in throttle_bytes.get(txc->cost);
Then we open bluestore log and rerun the aboving test. In ceph-osd.log, appeared some logs "_txc_calc_cost 0x564c308b0b00 cost 173240518 (257 ios * 670000 + 1050518 bytes)". It seems the transactions packaged by ECBackend consist of many address discontinuity 4K IO, but these IO will be write once to kernel by pwritev.
Related issues
History
#1 Updated by xinwei wei over 3 years ago
We have a ceph cluster(v 14.2.9) with 3 hosts, and there are 6 HDD OSDs on each host.
An EC pool was created with the default erasure-code-profile(k=2,m=1), and nothing changes in ceph.conf
After we use 'rados bench 30 write -b 2M -t 256 -p ecpool clean-up' to test the bandwidth, and use iostat to monitor disk utilization. Got the results: 30MB/s per disk.
With osd perf dump we found lots throttle-bluestore_throttle_bytes, it seems that BlueStore.cc::queue_transactions() was blocked in throttle_bytes.get(txc->cost);
Then we open bluestore log and rerun the aboving test. In ceph-osd.log, appeared some logs "_txc_calc_cost 0x564c308b0b00 cost 173240518 (257 ios * 670000 + 1050518 bytes)". It seems the transactions packaged by ECBackend consist of many address discontinuity 4K IO, but these IO will be write once to kernel by pwritev.
#2 Updated by xinwei wei over 3 years ago
The right "Affected Versions" should be "v14.2.10"
#3 Updated by Kefu Chai over 3 years ago
- Status changed from New to Pending Backport
#4 Updated by Nathan Cutler over 3 years ago
- Copied to Backport #46628: nautilus: The bandwidth of bluestore was throttled added
#5 Updated by Nathan Cutler over 3 years ago
- Copied to Backport #46629: octopus: The bandwidth of bluestore was throttled added
#7 Updated by Nathan Cutler over 3 years ago
- Backport changed from luminous,nautilus,octopus to nautilus,octopus
luminous is EOL
#8 Updated by Nathan Cutler over 3 years ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".