Ceph » rbd

This issue has been addressed in v15.2.16 of Octupus.

From the change log notes there's this patch: "librbd/object_map: rbd diff between two snapshots lists entire image content (pr#43806, Sunny Kumar)". This patch adds checking for "m_object_diff_state_valid" which will exclude invalid values that were previously counted.

Full change log:
https://docs.ceph.com/en/latest/releases/octopus/#v15-2-16-octopus

Hi Alex,

"rbd diff CephRBD_NVMe/vm-101-disk-0" reports the allocated areas of the image without taking snapshots into account, whereas "rbd du CephRBD_NVMe/vm-101-disk-0" attempts to report the amount of space used by the image including all its snapshots. Neither "rbd diff" nor "rbd du" are exact -- extent boundaries are rounded to the RBD object boundaries in most cases.

For example, here is a 1G image with 100M written at the beginning of the image:

$ rbd du testimage
NAME       PROVISIONED  USED   
testimage        1 GiB  100 MiB
$ rbd diff testimage | awk '{ SUM += $2 } END { print SUM/1024/1024 }'
100

Now, I take a snapshot:

$ rbd du testimage
NAME             PROVISIONED  USED   
testimage@snap1        1 GiB  100 MiB
testimage              1 GiB      0 B
<TOTAL>                1 GiB  100 MiB
$ rbd diff testimage |  awk '{ SUM += $2 } END { print SUM/1024/1024 }'
100

... and write 50M at the beginning of the image (thus overwriting 50M of those 100M in the snapshot):

$ rbd du testimage
NAME             PROVISIONED  USED   
testimage@snap1        1 GiB  100 MiB
testimage              1 GiB   52 MiB
<TOTAL>                1 GiB  152 MiB
$ rbd diff testimage | awk '{ SUM += $2 } END { print SUM/1024/1024 }'
100

The total amount of space used by the image is now 150M, "rbd du" estimates it to be 152M. Note that "rbd diff"-based summation still reports 100M because 0..100M is the only allocated area. To change that, let's write 10M at the end of the image:

$ rbd du testimage
NAME             PROVISIONED  USED   
testimage@snap1        1 GiB  100 MiB
testimage              1 GiB   64 MiB
<TOTAL>                1 GiB  164 MiB
$ rbd diff testimage | awk '{ SUM += $2 } END { print SUM/1024/1024
}'
112

The total amount of space used by the image is now 160M (110M + 50M for the snapshot delta), "rbd du" estimates it to be 164M. "rbd diff"-based summation went from 100M to 112M -- it can't "see" into the RBD object in this case so it can't report the exact 110M.

Hopefully this helps to understand that "rbd du" total for the image can be orders of magnitude bigger than "rbd diff"-based total. These are two different values, there is almost no connection between them.

Going back to CephRBD_NVMe/vm-101-disk-0 image, your "rbd du" output makes perfect sense to me based on what you said:

- 8 GiB in writes per day
- total of 87 GiB given by rbd diff is 100% accurate

root@virtual331:~# rbd du CephRBD_NVMe/vm-101-disk-0
NAME PROVISIONED USED
vm-101-disk-0@AutoSnap_10_17_2022_05_00_48 100 GiB 90 GiB

This is the initial snapshot with the entire filesystem (~87G) in it.

vm-101-disk-0@AutoSnap_10_17_2022_19_01_23 100 GiB 9.8 GiB
vm-101-disk-0@AutoSnap_10_18_2022_05_00_47 100 GiB 9.3 GiB
vm-101-disk-0@AutoSnap_10_18_2022_19_01_09 100 GiB 9.2 GiB
vm-101-disk-0@AutoSnap_10_19_2022_05_00_49 100 GiB 9.2 GiB
vm-101-disk-0@AutoSnap_10_19_2022_19_01_07 100 GiB 9.5 GiB
vm-101-disk-0@barcdaily20221020020029 100 GiB 9.3 GiB
vm-101-disk-0@AutoSnap_10_20_2022_05_00_53 100 GiB 7.6 GiB

These are the snapshot deltas, given that snapshots are taken on a schedule and the change rate is constant they are roughly equal in size.

vm-101-disk-0 100 GiB 9.0 GiB

This is the most recent delta which is about to be snapshotted. The size matches.

<TOTAL> 100 GiB 163 GiB

And this is the total amount of space used by the image. Again, keep in mind that all USED values are estimates -- in the worst case a 512-byte write can be accounted as a 4M write (default RBD object size).

Ilya, first thank you for the time you put into your messages. I am aware of the issue regarding RBD object size versus guest file system object size and its effects on usage reporting.

FYI the AutoSnap runs twice per day, and the actual 8 GiB per day in writes is known empirically via the guest OS. Going by the values given by du, it suggests nearly 20 GiB per day in writes which I find to be too high, and is what originally caught my attention, but you say the snaps likely contain mostly small writes, inflating the du value. That is easy to understand.

If I rbd du all of the ~200 images that I have, and sum the result, my total is about 24 TiB (or 72 raw TiB after 3x replication).

If I rbd diff all of my images, and sum that result, the total is only 14 TiB (or 42 raw TiB after 3x replication).

The usage on the pool reported by ceph df is 44 TiB, which is reasonably close to the replication-adjusted rbd diff total of 42 TiB.

PS /root> ceph df
--- RAW STORAGE ---
CLASS    SIZE   AVAIL    USED  RAW USED  %RAW USED
ssd    95 TiB  50 TiB  44 TiB    44 TiB      46.70
TOTAL  95 TiB  50 TiB  44 TiB    44 TiB      46.70

--- POOLS ---
POOL             ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
CephRBD_NVMe      3  128   24 TiB    6.80M   44 TiB  59.31     10 TiB

If I could ask my question in a different way, since rbd diff seems to be the more accurate and more valuable number and has a stronger correlation to ceph df, and real-world usage, why is rbd du the preferred value on the dashboard?

Thank you again

Alex Yarbrough wrote:

If I rbd du all of the ~200 images that I have, and sum the result, my total is about 24 TiB (or 72 raw TiB after 3x replication).

There should be no need to do that manually, just run "rbd du" with no arguments and grab <TOTAL> values at the bottom.

If I rbd diff all of my images, and sum that result, the total is only 14 TiB (or 42 raw TiB after 3x replication).

The usage on the pool reported by ceph df is 44 TiB, which is reasonably close to the replication-adjusted rbd diff total of 42 TiB.

I think this is just a coincidence.

> --- POOLS ---
> POOL             ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
> CephRBD_NVMe      3  128   24 TiB    6.80M   44 TiB  59.31     10 TiB

... 24T in STORED seems to align with your "rbd du" total?

If I could ask my question in a different way, since rbd diff seems to be the more accurate and more valuable number and has a stronger correlation to ceph df, and real-world usage, why is rbd du the preferred value on the dashboard?

Again, "rbd diff"-based summation isn't more accurate -- it's a completely different metric. Consider a workload that overwrites the entire RBD image in the course of a day and a daily snapshot schedule. Assuming all snapshots are retained, after 30 days, the total amount of space used by the RBD image would be 30 * image_size (without taking replication into account). However, "rbd diff"-based summation would still report 1 * `image_size' -- 30 times far from accurate...