Ceph » rgw

Using 0.94.1, the gc doesn't delete "__shadow_" objects while s3 object has been deleted.

Upload a file (10M) using s3cmd.

• s3cmd put 10M.file s3://test_bkt0/10M.file
  

  10M.file -> s3://test_bkt0/10M.file  [1 of 1]
   10485760 of 10485760   100% in    0s    27.32 MB/s  done
  

Check the object info

• Observe that .rgw.buckets occupies 10M data. That's as expected.
  

  $ ceph df
      .rgw.buckets           21     10240k         0        44641G           4
  

• There are 4 objects in .rgw.buckets, 3 of them are "__shadow_" objects.
  

  $ rados ls -p .rgw.buckets
  default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_3
  default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_1
  default.454156.12_10M.file
  default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_2
  

• Get object stat FYI.
  

  $ radosgw-admin object stat --bucket=test_bkt0 --object=10M.file
  {
      "name": "10M.file",
      "size": 10485760,
      "policy": {
          "acl": {
              "acl_user_map": [
                  {
                      "user": "albert_test",
                      "acl": 15
                  }
              ],
              "acl_group_map": [],
              "grant_map": [
                  {
                      "id": "albert_test",
                      "grant": {
                          "type": {
                              "type": 0
                          },
                          "id": "albert_test",
                          "email": "",
                          "permission": {
                              "flags": 15
                          },
                          "name": "0.01",
                          "group": 0
                      }
                  }
              ]
          },
          "owner": {
              "id": "albert_test",
              "display_name": "0.01" 
          }
      },
      "etag": "f1c9645dbc14efddc7d8a322685f26eb\u0000",
      "tag": "default.864240.52\u0000",
      "manifest": {
          "objs": [],
          "obj_size": 10485760,
          "explicit_objs": "false",
          "head_obj": {
              "bucket": {
                  "name": "test_bkt0",
                  "pool": ".rgw.buckets",
                  "data_extra_pool": ".rgw.buckets.extra",
                  "index_pool": ".rgw.buckets.index",
                  "marker": "default.454156.12",
                  "bucket_id": "default.454156.12" 
              },
              "key": "",
              "ns": "",
              "object": "10M.file",
              "instance": "" 
          },
          "head_size": 524288,
          "max_head_size": 524288,
          "prefix": ".UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_",
          "tail_bucket": {
              "name": "test_bkt0",
              "pool": ".rgw.buckets",
              "data_extra_pool": ".rgw.buckets.extra",
              "index_pool": ".rgw.buckets.index",
              "marker": "default.454156.12",
              "bucket_id": "default.454156.12" 
          },
          "rules": [
              {
                  "key": 0,
                  "val": {
                      "start_part_num": 0,
                      "start_ofs": 524288,
                      "part_size": 0,
                      "stripe_max_size": 4194304,
                      "override_prefix": "" 
                  }
              }
          ]
      },
      "attrs": {
          "user.rgw.content_type": "binary\/octet-stream\u0000",
          "user.rgw.x-amz-date": "Wed, 22 Apr 2015 07:04:03 +0000\u0000" 
      }
  }
  

• The garbage collection is empty
  

  $ radosgw-admin gc list --include-all
  []
  

Delete the s3 object "10M.file" via s3cmd

$ s3cmd del s3://test_bkt0/10M.file
File s3://test_bkt0/10M.file deleted

Check the deletion result

• Observe that .rgw.buckets occupies less 10M data and it leaves only 3 "__shadow_" objects.
  

  $ ceph df
      .rgw.buckets           21     9728k         0        44641G           3
  
  $ rados ls -p .rgw.buckets
  default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_3
  default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_1
  default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_2
  

• Show the gc info FYI
  

  $ radosgw-admin gc list --include-all
  [
      {
          "tag": "default.864240.52",
          "time": "2015-04-22 15:43:39.730086",
          "objs": [
              {
                  "pool": ".rgw.buckets",
                  "oid": "10M.file",
                  "key": "",
                  "instance": "" 
              },
              {
                  "pool": ".rgw.buckets",
                  "oid": "10M.file",
                  "key": "",
                  "instance": "" 
              },
              {
                  "pool": ".rgw.buckets",
                  "oid": "10M.file",
                  "key": "",
                  "instance": "" 
              }
          ]
      }
  ]
  

As expected, the object data would be deleted by running gc process, but things are not like that.

• Show the configs of gc FYI. for more config, please refer to the attachement.
  

  $ ceph daemon mon.node1-jovceph config show | grep "rgw_gc" 
      "rgw_gc_max_objs": "7877",
      "rgw_gc_obj_min_wait": "600",
      "rgw_gc_processor_max_time": "300",
      "rgw_gc_processor_period": "300",
  

• After half an hour, I find the gc is empty, but the 3 "__shadow_" objects are still occupy space in pool .rgw.buckets.
  

  $ radosgw-admin gc list --include-all
  []
  
  $ ceph df
      .rgw.buckets           21     9728k         0        44641G           3 
  
  $ rados ls -p .rgw.buckets
  default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_3
  default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_1
  default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_2
  

I ran into this issue in 0.94.1, but the same operation on 0.87.1 works well.
I found the gc list infos are different between 0.94.1 and 0.87.1. The field "oid" is same as s3 object name on 0.94.1, but it's "__shadow_" object name on 0.87.1
The gc list info on 0.87.1 as following:

[
    { "tag": "default.5220.7",
      "time": "2015-04-22 13:33:11.556569",
      "objs": [
            { "pool": ".rgw.buckets",
              "oid": "default.5220.1__shadow_.THpYR0Ny3eqzRvk-pYZmiPbI4Jdv15j_1",
              "key": ""},
            { "pool": ".rgw.buckets",
              "oid": "default.5220.1__shadow_.THpYR0Ny3eqzRvk-pYZmiPbI4Jdv15j_2",
              "key": ""},
            { "pool": ".rgw.buckets",
              "oid": "default.5220.1__shadow_.THpYR0Ny3eqzRvk-pYZmiPbI4Jdv15j_3",
              "key": ""}]}]

I think this issue is similar as Issue #10295, but not exactly the same.