Bug #23251

ceph daemon osd.NNN slow_used_bytes and slow_total_bytes wrong?

Added by Ben England almost 2 years ago. Updated almost 2 years ago.

Target version:
% Done:


3 - minor
Affected Versions:
Pull request ID:
Crash signature:


version: ceph-osd-12.2.1-34.el7cp.x86_64 = RHCS 3.0z1

In trying to understand ceph daemon osd.NNN perf dump counters, I came across this riddle. Bluestore is supposed to return the total amount of space available on the "slow" (data) device and the amount in use with:

[root@b10-h01-r620 bene]# ssh c05-h21-6048r ceph daemon osd.240 perf dump | grep slow
"slow_total_bytes": 79989571584, (~80 GB)
"slow_used_bytes": 3394240512, (~3 GB)

However, if I look at OSD-level counters:

[root@b10-h01-r620 bene]# ssh c05-h21-6048r ceph daemon osd.240 perf dump | grep stat_bytes
"stat_bytes": 2000811954176, (~2 TB)
"stat_bytes_used": 1252726378496, (~1.25 TB)
"stat_bytes_avail": 748085575680,

stat_bytes matches what parted says about the partition size, so I believe this one and not the bluestore one. But how does the allocator know if it has free space when slow_total_bytes and slow_used_bytes are wrong?

Anyone else see this on newer versions?


#1 Updated by Igor Fedotov almost 2 years ago

"slow_total_bytes" and "slow_used_bytes" are under "BlueFS" section and denotes just a fraction of BlueStore block device space given/used by BlueFS (i.e. DB and/or WAL).
Hence that's not a bug IMO. Suggest to close.

#2 Updated by Ben England almost 2 years ago

Thanks for responding, I didn't realize that, thought from looking at code that it was used for data as well. You can close this tracker, but could someone tell me what it means if you have non-zero slow_total_bytes and slow_used_bytes when you have a dedicated partition/volume for the WAL and RocksDB? Does it mean that the OSD ran out of space in those dedicated partitions and had to resort to using the "slow" device space? This is important because it means that the partitions were sized wrong, am trying to learn how to size them. Maybe this is really a documentation bug? Because I didn't see this covered in Bluestore documentation.

#3 Updated by Igor Fedotov almost 2 years ago

  • Status changed from New to Rejected

BlueStore has a BlueFS rebalance feature that dynamically reserves some amount of space for BlueFS at 'slow' device - current value is reported as slow_total_bytes. And yes - DB/WAL can use it when they lacks space at their major device(s) - slow_used_bytes tracks amount of data spilled over. So you'll have non-zero slow_total_bytes and zero slow_used_bytes in the normal state and non-zero slow_used_bytes in case of spillover. Don't know what documentation is saying on this topic...

Also available in: Atom PDF