Ceph » rgw

If we have orphan data in the bucket because of partial Multipart upload. The rados objects corresponding to the Multipart upload gets leaked when we delete the bucket.

• Steps to reproduce *
  1) take a Snapshot of "rados df" command
  2) Take a snapshot of items in "radosgw-admin gc list --include-all"
  3) Create a bucket
  3) Do a multipart upload without completing or cancelling the upload, hence creating orphan data
  4) Remove the bucket
  5) Check for orphan data
  6) Check the output of "rados df" and "radosgw-admin gc list --include-all"

Logs for the simulation

• Step 1 [ Output of rados df ] *
  pool name KB objects clones degraded unfound rd rd KB wr wr KB
  .in-chennai-1.intent-log 0 0 0 0 0 0 0 0 0
  .in-chennai-1.log 0 0 0 0 0 0 0 0 0
  .in-chennai-1.rgw 0 0 0 0 0 0 0 0 0
  .in-chennai-1.rgw.buckets 0 0 0 0 0 0 0 0 0
  .in-chennai-1.rgw.buckets.extra 0 0 0 0 0 0 0 0 0
  .in-chennai-1.rgw.buckets.index 0 0 0 0 0 0 0 0 0
  .in-chennai-1.rgw.control 0 8 0 0 0 0 0 0 0
  .in-chennai-1.rgw.gc 0 320 0 0 0 1382451 1382131 921634 0
  .in-chennai-1.usage 0 0 0 0 0 0 0 0 0
  .in-chennai-1.users 0 0 0 0 0 0 0 0 0
  .in-chennai-1.users.email 0 0 0 0 0 0 0 0 0
  .in-chennai-1.users.swift 0 0 0 0 0 0 0 0 0
  .in-chennai-1.users.uid 0 0 0 0 0 0 0 0 0
  .in-chennai.rgw.root 2 2 0 0 0 15 12 3 4
  .in.rgw.root 1 2 0 0 0 21 14 2 2
  total used 11018124 332
  total avail 23378484804
  total space 23389502928

There are zero items in rgw buckets

• Step 2 [ Output of gc list ] *
  [dev--mon~/] sudo radosgw-admin gc list --include-all
  []

There are zero items in GC list

• Step 3 [ Create the bucket ] *
  [deb~/python] s3cmd ls

[deb~/python] s3cmd mb s3://praveen
Bucket 's3://praveen/' created

[deb~/python] s3cmd ls
2016-08-29 11:03 s3://praveen

[deb~/python] python orphan_find.c.py
main
<Bucket: praveen>
0

No orphans in the bucket

• Step 4 [ Multipart upload done without complete, Hence we have orphan data ] *

[deb~/python] python orphan_find.c.py
main
<Bucket: praveen>
[<Part 1>, <Part 2>, <Part 3>, <Part 4>]
<MultiPartUpload 10:25:09.084077>
21000000

• Step 5 [ Remove the bucket ] *

[deb~/python] s3cmd rb s3://praveen
Bucket 's3://praveen/' removed

• Step 6 *

No orphan data

[deb~/python] python orphan_find.c.py
main
[deb~/python]

No items in GC list

[dev--mon~/] sudo radosgw-admin gc list --include-all
[]

Items still exist in rgw.bucket pool

[dev--mon~/] ./radosdf
pool name KB objects clones degraded unfound rd rd KB wr wr KB
.in-chennai-1.intent-log 0 0 0 0 0 0 0 0 0
.in-chennai-1.log 0 3 0 0 0 18 15 36 0
.in-chennai-1.rgw 0 0 0 0 0 18 14 14 3
.in-chennai-1.rgw.buckets 20508 7 0 0 0 8 4 57 20510
.in-chennai-1.rgw.buckets.extra 0 1 0 0 0 1 1 6 0
.in-chennai-1.rgw.buckets.index 0 32 0 0 0 148 84 126 0
.in-chennai-1.rgw.control 0 8 0 0 0 0 0 0 0
.in-chennai-1.rgw.gc 0 320 0 0 0 1387520 1387200 924800 0
.in-chennai-1.usage 0 1 0 0 0 9 9 18 0
.in-chennai-1.users 1 1 0 0 0 2 1 1 1
.in-chennai-1.users.email 0 0 0 0 0 0 0 0 0
.in-chennai-1.users.swift 0 0 0 0 0 0 0 0 0
.in-chennai-1.users.uid 1 2 0 0 0 15 13 9 1
.in-chennai.rgw.root 2 2 0 0 0 27 22 3 4
.in.rgw.root 1 2 0 0 0 33 22 2 2
total used 11093112 379
total avail 23378409816
total space 23389502928

We have 7 objects in rgw.buckets pool, because I uploaded 21 MB with chunk size 6MB. Rados object stripe size is 4 MB. hence we have 2 objects per 6MB chunk and 1 rados object for the last 3MB chunk. Totally 7 MB