Bug #46411
mimic: Disks associated to osds have small write io even on an idle ceph cluster
0%
Description
- 1.anomalies
When the ceph cluster is idle,you can found that the disks associated to osds have small write io every bluefs_balance_interval seconds. just like below:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 4.00 0.00 0.01 4.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 4.00 0.00 0.01 4.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 4.00 0.00 0.01 4.00 0.00 0.00 0.00 0.00 0.00 0.00
sdk 0.00 0.00 0.00 4.00 0.00 0.01 4.00 0.00 0.00 0.00 0.00 0.00 0.00
- 2.reason
This abnormal phenomenon is caused by the pr(https://github.com/ceph/ceph/pull/26866 ),so ceph version of v12.2.12~~v12.2.13, v13.2.5 ~~ v13.2.9 has the problem.
The reason is that it will lead to an empty RocksDB transaction submite to RocksDB even on an idle system,the empty transaction lead to the small write.
More details can be seen in the discussion in that pr.
- 3.solution
check if the transaction is empty,then it's will not be submitted to the rocksdb.
History
#1 Updated by hongsong wu 8 months ago
Affected Versions: v12.2.12~~v12.2.13, v13.2.5 ~~ v13.2.9
#2 Updated by Igor Fedotov 8 months ago
- Project changed from Ceph to bluestore
#4 Updated by hongsong wu 7 months ago
hongsong wu wrote:
Affected Versions: v12.2.12~~v12.2.13, v13.2.5 ~~ v13.2.10
#5 Updated by hongsong wu 7 months ago
I am sorry, There is a problem with the pr link given above( This abnormal phenomenon is caused by the pr(https://github.com/ceph/ceph/pull/26866 ) ).
#6 Updated by hongsong wu 7 months ago
I have fixed this bug in pr: https://github.com/ceph/ceph/pull/36108 , can you help to have a review ? @Igor Fedotov @Adam Kupczyk