Project

General

Profile

Bug #11447

gc deleting S3 objects leaves __shadow_ objects

Added by Albert AN over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
hammer
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
rgw
Pull request ID:
Crash signature:

Description

Using 0.94.1, the gc doesn't delete "__shadow_" objects while s3 object has been deleted.

Upload a file (10M) using s3cmd.
  • s3cmd put 10M.file s3://test_bkt0/10M.file
    10M.file -> s3://test_bkt0/10M.file  [1 of 1]
     10485760 of 10485760   100% in    0s    27.32 MB/s  done
    
Check the object info
  • Observe that .rgw.buckets occupies 10M data. That's as expected.
    $ ceph df
        .rgw.buckets           21     10240k         0        44641G           4
    
  • There are 4 objects in .rgw.buckets, 3 of them are "__shadow_" objects.
    $ rados ls -p .rgw.buckets
    default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_3
    default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_1
    default.454156.12_10M.file
    default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_2
    
  • Get object stat FYI.
    $ radosgw-admin object stat --bucket=test_bkt0 --object=10M.file
    {
        "name": "10M.file",
        "size": 10485760,
        "policy": {
            "acl": {
                "acl_user_map": [
                    {
                        "user": "albert_test",
                        "acl": 15
                    }
                ],
                "acl_group_map": [],
                "grant_map": [
                    {
                        "id": "albert_test",
                        "grant": {
                            "type": {
                                "type": 0
                            },
                            "id": "albert_test",
                            "email": "",
                            "permission": {
                                "flags": 15
                            },
                            "name": "0.01",
                            "group": 0
                        }
                    }
                ]
            },
            "owner": {
                "id": "albert_test",
                "display_name": "0.01" 
            }
        },
        "etag": "f1c9645dbc14efddc7d8a322685f26eb\u0000",
        "tag": "default.864240.52\u0000",
        "manifest": {
            "objs": [],
            "obj_size": 10485760,
            "explicit_objs": "false",
            "head_obj": {
                "bucket": {
                    "name": "test_bkt0",
                    "pool": ".rgw.buckets",
                    "data_extra_pool": ".rgw.buckets.extra",
                    "index_pool": ".rgw.buckets.index",
                    "marker": "default.454156.12",
                    "bucket_id": "default.454156.12" 
                },
                "key": "",
                "ns": "",
                "object": "10M.file",
                "instance": "" 
            },
            "head_size": 524288,
            "max_head_size": 524288,
            "prefix": ".UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_",
            "tail_bucket": {
                "name": "test_bkt0",
                "pool": ".rgw.buckets",
                "data_extra_pool": ".rgw.buckets.extra",
                "index_pool": ".rgw.buckets.index",
                "marker": "default.454156.12",
                "bucket_id": "default.454156.12" 
            },
            "rules": [
                {
                    "key": 0,
                    "val": {
                        "start_part_num": 0,
                        "start_ofs": 524288,
                        "part_size": 0,
                        "stripe_max_size": 4194304,
                        "override_prefix": "" 
                    }
                }
            ]
        },
        "attrs": {
            "user.rgw.content_type": "binary\/octet-stream\u0000",
            "user.rgw.x-amz-date": "Wed, 22 Apr 2015 07:04:03 +0000\u0000" 
        }
    }
    
  • The garbage collection is empty
    $ radosgw-admin gc list --include-all
    []
    

Delete the s3 object "10M.file" via s3cmd

$ s3cmd del s3://test_bkt0/10M.file
File s3://test_bkt0/10M.file deleted

Check the deletion result
  • Observe that .rgw.buckets occupies less 10M data and it leaves only 3 "__shadow_" objects.
    $ ceph df
        .rgw.buckets           21     9728k         0        44641G           3
    
    $ rados ls -p .rgw.buckets
    default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_3
    default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_1
    default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_2
    
  • Show the gc info FYI
    $ radosgw-admin gc list --include-all
    [
        {
            "tag": "default.864240.52",
            "time": "2015-04-22 15:43:39.730086",
            "objs": [
                {
                    "pool": ".rgw.buckets",
                    "oid": "10M.file",
                    "key": "",
                    "instance": "" 
                },
                {
                    "pool": ".rgw.buckets",
                    "oid": "10M.file",
                    "key": "",
                    "instance": "" 
                },
                {
                    "pool": ".rgw.buckets",
                    "oid": "10M.file",
                    "key": "",
                    "instance": "" 
                }
            ]
        }
    ]
    
As expected, the object data would be deleted by running gc process, but things are not like that.
  • Show the configs of gc FYI. for more config, please refer to the attachement.
    $ ceph daemon mon.node1-jovceph config show | grep "rgw_gc" 
        "rgw_gc_max_objs": "7877",
        "rgw_gc_obj_min_wait": "600",
        "rgw_gc_processor_max_time": "300",
        "rgw_gc_processor_period": "300",
    
  • After half an hour, I find the gc is empty, but the 3 "__shadow_" objects are still occupy space in pool .rgw.buckets.
    $ radosgw-admin gc list --include-all
    []
    
    $ ceph df
        .rgw.buckets           21     9728k         0        44641G           3 
    
    $ rados ls -p .rgw.buckets
    default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_3
    default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_1
    default.454156.12__shadow_.UaXnc9iIVNXnLzEY0qUzglOiPtIemfA_2
    

I ran into this issue in 0.94.1, but the same operation on 0.87.1 works well.
I found the gc list infos are different between 0.94.1 and 0.87.1. The field "oid" is same as s3 object name on 0.94.1, but it's "__shadow_" object name on 0.87.1
The gc list info on 0.87.1 as following:

[
    { "tag": "default.5220.7",
      "time": "2015-04-22 13:33:11.556569",
      "objs": [
            { "pool": ".rgw.buckets",
              "oid": "default.5220.1__shadow_.THpYR0Ny3eqzRvk-pYZmiPbI4Jdv15j_1",
              "key": ""},
            { "pool": ".rgw.buckets",
              "oid": "default.5220.1__shadow_.THpYR0Ny3eqzRvk-pYZmiPbI4Jdv15j_2",
              "key": ""},
            { "pool": ".rgw.buckets",
              "oid": "default.5220.1__shadow_.THpYR0Ny3eqzRvk-pYZmiPbI4Jdv15j_3",
              "key": ""}]}]

I think this issue is similar as Issue #10295, but not exactly the same.

conf_show.txt View - CONFIG BROWSER on calamari web (24.4 KB) Albert AN, 04/22/2015 08:37 AM

apache2_access.log View - Apache access log (215 Bytes) Albert AN, 04/22/2015 08:37 AM

client.radosgw.objgw.log View - rgw log (3.11 KB) Albert AN, 04/22/2015 08:37 AM

Associated revisions

Revision cc5d3467 (diff)
Added by Yehuda Sadeh over 5 years ago

rgw: use correct oid when creating gc chains

Fixes: #11447
Backport: hammer

When creating gc chain, use the appropriate oid, otherwise objects will
leak.

Signed-off-by: Yehuda Sadeh <>

Revision da4d2274 (diff)
Added by Yehuda Sadeh over 5 years ago

rgw: use correct oid when creating gc chains

Fixes: #11447
Backport: hammer

When creating gc chain, use the appropriate oid, otherwise objects will
leak.

Signed-off-by: Yehuda Sadeh <>
(cherry picked from commit cc5d34678c6d4bdcd552e6334a383c4df9797f46)

History

#1 Updated by Yehuda Sadeh over 5 years ago

  • Assignee changed from Yehuda Sadeh to Orit Wasserman

Yeah, it's not the same as #10295. The gc entries store the wrong info.

#2 Updated by Yehuda Sadeh over 5 years ago

  • Assignee changed from Orit Wasserman to Yehuda Sadeh

#3 Updated by Yehuda Sadeh over 5 years ago

  • Status changed from New to Fix Under Review

#4 Updated by Albert AN over 5 years ago

Yehuda Sadeh wrote:

Yeah, it's not the same as #10295. The gc entries store the wrong info.

How long will the review take?

#5 Updated by Yehuda Sadeh over 5 years ago

  • Status changed from Fix Under Review to Resolved
  • Backport set to hammer

Fixed, commit:cc5d34678c6d4bdcd552e6334a383c4df9797f46
Hammer, commit:da4d2274b5d83a116e767f3063752624d1719c32

Also available in: Atom PDF