Ceph » rgw

Hello!

I have a Ceph Jewel 10.2.0 cluster consisting of 10 nodes, 29 OSD each, one serving as RGW (apache/fcgi).
I also have the following pools configuration, last column is pool type (erasure/replicated).
Erasure code profile is ISA k+m=6+3

GLOBAL:
SIZE AVAIL RAW USED %RAW USED
1582T 1581T 300G 0.02
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS pool_type
.rgw.root 3 5661 0 1053T 16 erasure
ed-1.rgw.control 5 0 0 1053T 8 erasure
ed-1.rgw.data.root 6 3558 0 1053T 12 erasure
ed-1.rgw.gc 7 0 0 1053T 32 erasure
ed-1.rgw.intent-log 9 0 0 1053T 0 erasure
ed-1.rgw.usage 10 0 0 1053T 0 erasure
ed-1.rgw.buckets.data 16 89275M 0 1053T 22375 erasure
ed-1.rgw.meta 17 5235 0 1053T 17 erasure
ed-1.rgw.users.keys 18 40 0 790T 3 replicated
ed-1.rgw.users.email 19 0 0 790T 0 replicated
ed-1.rgw.users.uid 20 997 0 790T 6 replicated
ed-1.rgw.users.swift 21 0 0 790T 0 replicated
ed-1.rgw.users 22 0 0 790T 0 replicated
ed-1.rgw.log 24 0 0 790T 136 replicated
ed-1.rgw.buckets.index 26 0 0 790T 6 replicated
ed-1.rgw.buckets.non-ec 27 0 0 790T 0 replicated

Problem is that pool ed-1.rgw.buckets.data is overloaded with objects named "__shadow*" and "__multipart*" and GC is not coming for them at all regardless of bucket.
Cluster was set up a week ago, lots of objects were uploaded and deleted, but GC is not deleting these types of objects.

$ rados ls -p ed-1.rgw.buckets.data | grep -c __shadow
22057
$ rados ls -p ed-1.rgw.buckets.data | grep -c __multipart
308
$ rados ls -p ed-1.rgw.buckets.data | grep -v __shadow | grep -vc __multipart
10
$ s3cmd la --recursive s3://test1 | grep s3 | wc -l
10

You may see here that have only 10 useful objects in cluster. Other 22365 are of "__shadow" and "__multipart" kinds.
I have no running multipart uploads ATM.

$ s3cmd multipart s3://test1
s3://test1/
Initiated Path Id
$

"radosgw-admin gc list --include-all" always returns empty list.
"radosgw-admin gc process" always dose nothing.

My ceph.conf:

[global]
fsid = 5ddb8aab-49b4-4a63-918e-33c569e3101e
mon initial members = ed-ds-c171, ed-ds-c172, ed-ds-c173
mon host = 10.144.66.171, 10.144.66.172, 10.144.66.173
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
public network = 10.144.66.0/24
cluster network = 10.144.126.0/24
osd pool default size = 2
osd pool default min size = 1
mon pg warn max per osd = 1000

[client.rgw.ed-1]
host = ed-ds-c180
keyring = /etc/ceph/ceph.client.rgw.ed-1.keyring
admin socket = /var/run/ceph/client.rgw.ed-1.asok
rgw frontends = fastcgi socket_port=9000 socket_host=10.144.66.180
rgw zone = ed-1
rgw dns name = dnsname_here
rgw ops log rados = false
rgw enable ops log = false
rgw print continue = false
debug rgw = 20

How can I make GC clean those objects?