Ceph » rgw

Need to wait for gc to run its course. Not all entries are going to show in the gc listing immediately. You need to run 'radosgw-admin gc list --include-all'.

Tested again: reduced rgw_gc settings to 300s.

On 0.80.7, the shadow object were purged.

On 0.87, 0.89 the garbage collection does not happen and remain behind.

1. ceph --admin-daemon /var/run/ceph/ceph-mon.b.asok config show | grep gc
   "rgw_gc_max_objs": "521",
   "rgw_gc_obj_min_wait": "1200",
   "rgw_gc_processor_max_time": "600",
   "rgw_gc_processor_period": "600",

On 0.89:

[root@radosgw ~]# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
596G 588G 8402M 1.38

.rgw.buckets           16     3962M      0.65          294G       14885

3G is correct "current" blobs.
1G of test data was created and rm'ed.

The "extra" 962M is not purged.
2472 default.95874.2__shadow_* objects

Reproduced this on a fresh 0.89.

Created 400 objects, deleted 300.

Found 3 objects in gc list --include-all.

After running gc process I have 697 objects in .rgw.buckets.
1. 100 objects are "real objects"
2. 597 objects are "shadow objects" (600-3; not sure why only 3 objects appeared in gc list --include-all)
3. Now gc list --include-all is empty
4. This pool has size = 2; on the 3 OSD /var/lib/ceph/osd/ceph-XX there are 1194 files with names shadow

find /var/lib/ceph | grep shadow | wc -l across all OSDs show
1194 files.

Do you by any chance have non default zone configuration?

It is a civetweb bug.

I added 100 objects and rm'ed - 10M ea, 300 shadow objects created.
All 300 shadow objects appear in gc list --include-all
and are purged.

Should have written:

It is a civetweb bug.

With apache/fastcgi: I added 100 objects and rm'ed - 10M ea,
300 shadow objects created.
All 300 shadow objects appear in gc list --include-all and are purged.

Summary of bug:

civetweb 0.89 is not correctly preparing the garbage collection list:
1. create 100 rgw objects (10M size) so 400 rados objects are created in .rgw.buckets
2. rm 100 rgw objects, 300 rados objects left in .rgw.bucket — all shadow objects
3. gc list --include-all only shows 3 oids

With apache/fastcgi
gc list --include-all will show 300 oids

Not sure how civetweb could affect gc work. I do see that your rgw_gc_max_objs is non standard. Did you put this config on the global section, or only under the section for the specific rgw user?

civetweb:

when you delete a second object, it does not appear in gc list --incude-all

How to reproduce:
1. add two objects (ea 10M)
2. rm one object, observe 3 oids in gc list --include-all
2. rm 2nd object, observe only 3 oids in gc list --include-all

1. empty output

1. only 3 oids instead of 6 expected
2. radosgw-admin gc list --include-all
   [ { "tag": "default.4292.0",
   "time": "2014-12-16 10:58:47.729479",
   "objs": [ { "pool": ".rgw.buckets",
   "oid": "default.4292.1__shadow_.0ZO019kstnvzHUTrGW0zz1sqdl08Efm_1",
   "key": ""}, { "pool": ".rgw.buckets",
   "oid": "default.4292.1__shadow_.0ZO019kstnvzHUTrGW0zz1sqdl08Efm_2",
   "key": ""}, { "pool": ".rgw.buckets",
   "oid": "default.4292.1__shadow_.0ZO019kstnvzHUTrGW0zz1sqdl08Efm_3",
   "key": ""}]}]

Revert rgw_gc_max_objs to 32, problem remains.

civetweb seems to "corrupt" the gc list.

1. rm one object (10M size) with civetweb, 3 oids in gc list
2. rm 2nd object with apache/fastcgi, no new oids appear

When the gc list is clear, delete two 10M objects with apache/fastcgi:
6 oids appear

1. The issue is reproducible only on objects created by civetweb.
The bucket can be created by either civetweb or fastcgi.
Only oids of the first deleted civetweb object deleted appear
in the gc list.
All oids of deleted fastcgi objects appear in the gc list.

2. Add 100 objects by fastcgi, delete by fastcgi/civetweb.
No issues here — 300 oids added to gc list

3. Add 100 objects by civetweb, delete by fastcgi/civetweb.
Only 3 oids added to gc list.

I'm facing with the same issue and I realised when I restart the radosgw the gc became clean but the shadow objects still the ceph pool, see below:

root@cephgw0001:~# radosgw-admin gc list
[ { "tag": "default.1552327.7",
"time": "2015-01-26 18:42:39.532908",
"objs": [ { "pool": ".rgw.buckets.replicated,",
"oid": "default.1311201.2__shadow_.RsL9Sl1vRYHdCkjgPC4Bgse7Pq9I7fM_1",
"key": ""}]}, { "tag": "_Paybwvun_sU_KcEFK5ov1ceCuSaP1ZM",
"time": "2015-01-26 18:42:48.900681",
"objs": [ { "pool": ".rgw.buckets.replicated",
"oid": "default.1507466.1__shadow_.dEOjMr96qgBh2SKDEbQ77CaRcPJ0LCJ_1",
"key": ""}]}]root@cephgw0001:~# ^C

root@cephgw0001:~# /etc/init.d/radosgw restart
Starting client.radosgw.gateway...
2015-01-26 19:10:48.573344 7fb6c1024780 -1 WARNING: libcurl doesn't support curl_multi_wait()
2015-01-26 19:10:48.573394 7fb6c1024780 -1 WARNING: cross zone / region transfer performance may be affected
/usr/bin/radosgw is running.

root@cephgw0001:~# radosgw-admin gc list
[]root@cephgw0001:~#

root@cephgw0001:~# rados -p .rgw.buckets.replicated ls
default.1554751.1__shadow_.aoCI7rTnBbJKfb5jogxvLStSsbJnL27_1
default.1554751.1__shadow_.lyntzWkaFPPgGfY5XhZBuClZI4Vd0TT_1

Also, I'm using Apache + FastCGI... exclude a possible issue with civetweb.

merged, commit:c26225980c2fd018e70033a453d635533fcdefec

hammer: 4e84f318de97e592e16493c67491ba0d7f8103a8 matching pull request https://github.com/ceph/ceph/pull/4515

How we can remove such objects from storage? We were running 0.94 with civetweb frontend for a couple of month and now it's about 70TiB of garbage in our buckets :(

I think this recipe must be included in release notes, since many of us running same configuration.

I've been working on a tool to identify these orphan objects. It's still in the works, but I'll try to get it ready in the following days.

Yehuda Sadeh wrote:

I've been working on a tool to identify these orphan objects. It's still in the works, but I'll try to get it ready in the following days.

FWIW we were never using civetweb, always fastcgi, and i show very large difference between ceph df output vs bucket stats size. Looking forward to what the tool produces, though.

Hi Yehuda,

Do you have an update on this bug?

I'm running Ceph 0.94.3 with civetweb and I've run some tests with cosbench that clean up the data they generate, but a week later I see the data is still there.

I have only 20 MB of valid data in the .rgw.buckets pool, but "ceph df" reports 26 GB.

The garbage collector doesn't know of any objects that need to be deleted:
$ radosgw-admin gc list --include-all
[]

The settings used by garbage collector:
$ ceph --admin-daemon /var/run/ceph/ceph-mon.controller1.asok config show | grep gc
"xio_trace_msgcnt": "false",
"osd_pool_default_flag_nopgchange": "false",
"rgw_enable_gc_threads": "true",
"rgw_gc_max_objs": "32",
"rgw_gc_obj_min_wait": "7200",
"rgw_gc_processor_max_time": "3600",
"rgw_gc_processor_period": "3600",

$ ceph df | grep .rgw.buckets
.rgw.buckets.index 12 0 0 520T 14
.rgw.buckets.extra 15 0 0 520T 1
.rgw.buckets 25 26037M 0 520T 428

There are 417 objects that should be removed
$ rados ls -p .rgw.buckets | grep -i -c shadow
417

I even changed rgw_gc_max_objs = 200 thinking that the default 32 would take too long to clean up maybe, then I restarted radosgw and ran "radosgw-admin gc process", but no difference.

The short term solution would be to manually delete all these "shadow" objects.

I have three radosgw instances running behind haproxy and use 64M stripe size, if it makes any difference.

Thank you,
George

Is the data that you see new, or is it the same old data that leaked before running 0.94.3? What do you see when you run 'radosgw-admin gc list --include-all' right after running cosbench? Try to avoid running 'gc process' using radosgw-admin when you test this (as you modified the number of shards, you might have an issue with mismatching shard numbers).

We are still facing this issue.