Project

General

Profile

Bug #24505

radosgw-admin user stats are incorrect when dynamic re-sharding is enabled

Added by Mark Kogan 3 months ago. Updated 6 days ago.

Status:
Triaged
Priority:
High
Target version:
-
Start date:
06/12/2018
Due date:
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:

Description

On a Ceph cluster started with vstart:

MON=1 OSD=1 MDS=0 MGR=1 RGW=1 ../src/vstart.sh -n -o "bluestore_block_size = 5000000000000" -o "rgw_enable_usage_log = true" 

Using COSBench S3 workload - complete writing 1 million objects of 1KB --

<work type="prepare" workers="4" interval="5" rampup="5" config="content=zero;cprefix=s3cosbench;containers=r(1,1);objects=r(1,1000000);sizes=c(1)KB" />

although "ceph df" shows 999999 OBJECTS

POOLS: 
    NAME                          ID     USED        %USED     MAX AVAIL     OBJECTS  
    default.rgw.buckets.data      6      954 MiB      0.02       4.4 TiB      999999 

radosgw-admin user stats shows only 272721 entries :

$ ./bin/radosgw-admin user stats --sync-stats --uid=testid
2018-06-12 08:06:24.588 7fa2f6410800  0 check_bucket_shards: resharding needed: stats.num_objects=272721 shard max_objects=100000
{
    "stats": {
        "total_entries": 272721,
        "total_bytes": 272721000,
        "total_bytes_rounded": 1117065216
    },
    "last_stats_sync": "2018-06-12 12:06:24.590311Z",
    "last_stats_update": "2018-06-12 12:06:24.586702Z" 
}

(Note the line above - "2018-06-12 08:06:24.588 7fa2f6410800 0 check_bucket_shards: resharding needed: stats.num_objects=272721 shard max_objects=100000")

Trying with adding the "--reset-stats" parameter did not remedy:

$ ./bin/radosgw-admin user stats --reset-stats --uid=testid
{
    "stats": {
        "total_entries": 272721,
        "total_bytes": 272721000,
        "total_bytes_rounded": 1117065216
    },
    "last_stats_sync": "0.000000",
    "last_stats_update": "2018-06-12 12:08:15.044129Z" 
}

History

#1 Updated by Mark Kogan 3 months ago

  • Description updated (diff)

#2 Updated by Orit Wasserman 3 months ago

  • Priority changed from Normal to High

#3 Updated by Yehuda Sadeh 3 months ago

  • Status changed from New to Triaged

#4 Updated by Mark Kogan 3 months ago

Adding another scenario -
where the cluster was installed with resharding and then resharding was disabled and radosgw restarted in the middle of cosbench write operation
resulting in the following condition:
"marker": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8",
"bucket_id": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.18",

radosgw-admin metadata get bucket:s3testqwer1
{
    "key": "bucket:s3testqwer1",
    "ver": {
        "tag": "_BFTL0_ISwpWsXaGHkyI02j0",
        "ver": 15
    },
    "mtime": "2018-06-28 13:31:43.578495Z",
    "data": {
        "bucket": {
            "name": "s3testqwer1",
            "marker": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8",
            "bucket_id": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.18",
            "tenant": "",
            "explicit_placement": {
                "data_pool": "",
                "data_extra_pool": "",
                "index_pool": "" 
            }
        },
        "owner": "testid",
        "creation_time": "2018-06-14 13:55:37.844197Z",
        "linked": "true",
        "has_bucket_info": "false" 
    }

radosgw-admin metadata get bucket.instance:s3testqwer1:a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8                                                [10/42]
{
    "key": "bucket.instance:s3testqwer1:a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8",
    "ver": {
        "tag": "_4unVrCsMSAyG7XH5M0G7Ptj",
        "ver": 3
    },
    "mtime": "2018-06-14 14:41:03.598964Z",
    "data": {
        "bucket_info": {
            "bucket": {
                "name": "s3testqwer1",
                "marker": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8",
                "bucket_id": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8",
                "tenant": "",
                "explicit_placement": {
                    "data_pool": "",
                    "data_extra_pool": "",
                    "index_pool": "" 
                }
            },
            "creation_time": "2018-06-14 13:55:37.844197Z",
            "owner": "testid",
            "flags": 0,
            "zonegroup": "d6f2327a-9aad-45a9-b31f-9880a4d792af",
            "placement_rule": "default-placement",
            "has_instance_obj": "true",
            "quota": {
                "enabled": false,
                "check_on_raw": false,
                "max_size": -1,
                "max_size_kb": 0,
                "max_objects": -1
            },
            "num_shards": 0,
            "bi_shard_hash_type": 0,
            "requester_pays": "false",
            "has_website": "false",
            "swift_versioning": "false",
            "swift_ver_location": "",
            "index_type": 0,
            "mdsearch_config": [],
            "reshard_status": 2,
            "new_bucket_instance_id": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.11" 
        },
        "attrs": [
            {
                "key": "user.rgw.acl",
                "val": "AgKJAAAAAwIVAAAABgAAAHRlc3RpZAcAAABUZXN0IElkBANoAAAAAQEAAAAGAAAAdGVzdGlkDwAAAAEAAAAGAAAAdGVzdGlkBQM5AAAAAgIEAAAAAAAAAAYAAAB0ZXN0aWQAAAAAAAAAAAICBAAAAA8AAAAHAAAAVGVzdCBJZAAAAAAAAAAAAAAAAAAAAA=" 
            },
            {
                "key": "user.rgw.idtag",
                "val": "" 
            }
        ]
    }
}

radosgw-admin metadata get bucket.instance:s3testqwer1:a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.18
{
    "key": "bucket.instance:s3testqwer1:a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.18",
    "ver": {
        "tag": "8EKJdLtqy1iBUoVTH6JzkYZ2",
        "ver": 4
    },
    "mtime": "2018-06-17 08:44:12.670789Z",
    "data": {
        "bucket_info": {
            "bucket": {
                "name": "s3testqwer1",
                "marker": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8",
                "bucket_id": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.18",
                "tenant": "",
                "explicit_placement": {
                    "data_pool": "",
                    "data_extra_pool": "",
                    "index_pool": "" 
                }
            },
            "creation_time": "2018-06-14 13:55:37.844197Z",
            "owner": "testid",
            "flags": 0,
            "zonegroup": "d6f2327a-9aad-45a9-b31f-9880a4d792af",
            "placement_rule": "default-placement",
            "has_instance_obj": "true",
            "quota": {
                "enabled": false,
                "check_on_raw": false,
                "max_size": -1,
                "max_size_kb": 0,
                "max_objects": -1
            },
            "num_shards": 2,
            "bi_shard_hash_type": 0,
            "requester_pays": "false",
            "has_website": "false",
            "swift_versioning": "false",
            "swift_ver_location": "",
            "index_type": 0,
            "mdsearch_config": [],
            "reshard_status": 1,
            "new_bucket_instance_id": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.724274.1" 
        },
        "attrs": [
            {
                "key": "user.rgw.acl",
                "val": "AgKJAAAAAwIVAAAABgAAAHRlc3RpZAcAAABUZXN0IElkBANoAAAAAQEAAAAGAAAAdGVzdGlkDwAAAAEAAAAGAAAAdGVzdGlkBQM5AAAAAgIEAAAAAAAAAAYAAAB0ZXN0aWQAAAAAAAAAAAICBAAAAA8AAAAHAAAAVGVzdCBJZAAAAAAAAAAAAAAAAAAAAAA=" 
            },
            {
                "key": "user.rgw.idtag",
                "val": "" 
            }
        ]
    }
}

#5 Updated by Matt Benjamin 24 days ago

Mark, have we addressed any of these issues in recent stats work?

Matt

#6 Updated by Lei Liu 24 days ago

#7 Updated by Mark Kogan 6 days ago

Thanks for the quick reproduction procedure in #27205
Those bugs seem related, I will debug.

Also available in: Atom PDF