Actions
Feature #41417
closedrgw: store small object's data part into xattr to avoid disk space wasting
% Done:
0%
Source:
Tags:
Backport:
Description
In the following test, I store 100000 1K/2K/4K/8K/16K/32K/64K object into newly created ceph cluster.
And record their space usage info. As we can see, a lot of disk space is wasted. In the 1K object case,
100MB small object consume at least 6.1GB disk space.
>>>>> 1K:
[root@um14 scripts]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
.rgw.root 256 KiB 4 0 4 0 0 0 36 36 KiB 4 4 KiB 0 B 0 B
default.rgw.buckets.data 6.1 GiB 100317 0 100317 0 0 0 1740 1.3 MiB 907203 98 MiB 0 B 0 B
default.rgw.buckets.index 0 B 1 0 1 0 0 0 604577 591 MiB 302257 197 MiB 0 B 0 B
default.rgw.control 0 B 8 0 8 0 0 0 0 0 B 0 0 B 0 B 0 B
default.rgw.log 0 B 175 0 175 0 0 0 4770 4.5 MiB 3814 634 KiB 0 B 0 B
default.rgw.meta 256 KiB 5 0 5 0 0 0 94 77 KiB 29 14 KiB 0 B 0 B
defaults.rgw.buckets.data 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B
defaults.rgw.buckets.index 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B
total_objects 100510
total_used 7.1 GiB
total_avail 551 GiB
total_space 558 GiB
[root@um14 scripts]# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
0 hdd 0.54489 1.00000 558 GiB 7.1 GiB 6.1 GiB 21 MiB 1003 MiB 551 GiB 1.28 1.00 432 up
TOTAL 558 GiB 7.1 GiB 6.1 GiB 21 MiB 1003 MiB 551 GiB 1.28
MIN/MAX VAR: 1.00/1.00 STDDEV: 0
...
>>>>> 4K:
[root@um14 scripts]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
.rgw.root 384 KiB 6 0 6 0 0 0 30 30 KiB 6 6 KiB 0 B 0 B
default.rgw.buckets.data 6.1 GiB 100001 0 100001 0 0 0 2 1 KiB 900012 391 MiB 0 B 0 B
default.rgw.buckets.index 0 B 1 0 1 0 0 0 600049 587 MiB 300006 195 MiB 0 B 0 B
default.rgw.control 0 B 8 0 8 0 0 0 0 0 B 0 0 B 0 B 0 B
default.rgw.log 0 B 207 0 207 0 0 0 2388 2.1 MiB 1530 2 KiB 0 B 0 B
default.rgw.meta 256 KiB 5 0 5 0 0 0 44 38 KiB 26 13 KiB 0 B 0 B
total_objects 100228
total_used 7.1 GiB
total_avail 551 GiB
total_space 558 GiB
>>>>> 8K:
[root@um14 scripts]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
.rgw.root 256 KiB 4 0 4 0 0 0 13 13 KiB 4 4 KiB 0 B 0 B
default.rgw.buckets.data 6.1 GiB 100000 0 100000 0 0 0 0 0 B 900000 781 MiB 0 B 0 B
default.rgw.buckets.index 0 B 1 0 1 0 0 0 600030 586 MiB 300001 195 MiB 0 B 0 B
default.rgw.control 0 B 8 0 8 0 0 0 0 0 B 0 0 B 0 B 0 B
default.rgw.log 0 B 207 0 207 0 0 0 2369 2.1 MiB 1558 0 B 0 B 0 B
default.rgw.meta 256 KiB 5 0 5 0 0 0 46 40 KiB 28 14 KiB 0 B 0 B
total_objects 100225
total_used 7.1 GiB
total_avail 551 GiB
total_space 558 GiB
>>>>>> 16K:
+ rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
.rgw.root 256 KiB 4 0 4 0 0 0 13 13 KiB 4 4 KiB 0 B 0 B
default.rgw.buckets.data 6.1 GiB 99803 0 99803 0 0 0 0 0 B 898227 1.5 GiB 0 B 0 B
default.rgw.buckets.index 0 B 1 0 1 0 0 0 598853 585 MiB 299411 195 MiB 0 B 0 B
default.rgw.control 0 B 8 0 8 0 0 0 0 0 B 0 0 B 0 B 0 B
default.rgw.log 0 B 175 0 175 0 0 0 1383 1.2 MiB 922 0 B 0 B 0 B
default.rgw.meta 256 KiB 5 0 5 0 0 0 46 40 KiB 28 14 KiB 0 B 0 B
total_objects 99996
total_used 7.1 GiB
total_avail 551 GiB
total_space 558 GiB
>>>>> 32K:
+ rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
.rgw.root 256 KiB 4 0 4 0 0 0 13 13 KiB 4 4 KiB 0 B 0 B
default.rgw.buckets.data 6.1 GiB 99700 0 99700 0 0 0 0 0 B 897300 3.0 GiB 0 B 0 B
default.rgw.buckets.index 0 B 1 0 1 0 0 0 598229 584 MiB 299099 195 MiB 0 B 0 B
default.rgw.control 0 B 8 0 8 0 0 0 0 0 B 0 0 B 0 B 0 B
default.rgw.log 0 B 175 0 175 0 0 0 1383 1.2 MiB 922 0 B 0 B 0 B
default.rgw.meta 256 KiB 5 0 5 0 0 0 46 40 KiB 28 14 KiB 0 B 0 B
total_objects 99893
total_used 7.1 GiB
total_avail 551 GiB
total_space 558 GiB
>>>>> 64K:
+ rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
.rgw.root 256 KiB 4 0 4 0 0 0 13 13 KiB 4 4 KiB 0 B 0 B
default.rgw.buckets.data 6.1 GiB 99845 0 99845 0 0 0 0 0 B 898605 6.1 GiB 0 B 0 B
default.rgw.buckets.index 0 B 1 0 1 0 0 0 599120 585 MiB 299536 195 MiB 0 B 0 B
default.rgw.control 0 B 8 0 8 0 0 0 0 0 B 0 0 B 0 B 0 B
default.rgw.log 0 B 175 0 175 0 0 0 2241 2.0 MiB 1494 0 B 0 B 0 B
default.rgw.meta 256 KiB 5 0 5 0 0 0 87 74 KiB 40 20 KiB 0 B 0 B
total_objects 100038
total_used 7.1 GiB
total_avail 551 GiB
total_space 558 GiB
In order to resolve this problem, we can store small object into rados object's xattr part,
the db of osd will convert these small kv into continuous large blocks, thus reducing the
waste of disk space.
Actions