Project

General

Profile

Actions
Copy link

Bug #24505

closed

radosgw-admin user stats are incorrect when dynamic re-sharding is enabled

Added by Mark Kogan almost 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
High
Assignee:
Casey Bodley
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
luminous mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v13.0.0
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

On a Ceph cluster started with vstart:

MON=1 OSD=1 MDS=0 MGR=1 RGW=1 ../src/vstart.sh -n -o "bluestore_block_size = 5000000000000" -o "rgw_enable_usage_log = true"

Using COSBench S3 workload - complete writing 1 million objects of 1KB --

<work type="prepare" workers="4" interval="5" rampup="5" config="content=zero;cprefix=s3cosbench;containers=r(1,1);objects=r(1,1000000);sizes=c(1)KB" />

although "ceph df" shows 999999 OBJECTS

POOLS: 
    NAME                          ID     USED        %USED     MAX AVAIL     OBJECTS  
    default.rgw.buckets.data      6      954 MiB      0.02       4.4 TiB      999999

radosgw-admin user stats shows only 272721 entries :

$ ./bin/radosgw-admin user stats --sync-stats --uid=testid
2018-06-12 08:06:24.588 7fa2f6410800  0 check_bucket_shards: resharding needed: stats.num_objects=272721 shard max_objects=100000
{
    "stats": {
        "total_entries": 272721,
        "total_bytes": 272721000,
        "total_bytes_rounded": 1117065216
    },
    "last_stats_sync": "2018-06-12 12:06:24.590311Z",
    "last_stats_update": "2018-06-12 12:06:24.586702Z" 
}

(Note the line above - "2018-06-12 08:06:24.588 7fa2f6410800 0 check_bucket_shards: resharding needed: stats.num_objects=272721 shard max_objects=100000")

Trying with adding the "--reset-stats" parameter did not remedy:

$ ./bin/radosgw-admin user stats --reset-stats --uid=testid
{
    "stats": {
        "total_entries": 272721,
        "total_bytes": 272721000,
        "total_bytes_rounded": 1117065216
    },
    "last_stats_sync": "0.000000",
    "last_stats_update": "2018-06-12 12:08:15.044129Z" 
}

Related issues 3 (0 open — 3 closed)

Related to rgw - Bug #27205: Incorrect stats by "radosgw-admin user stats" after bucket reshardingDuplicateMark Kogan08/23/2018

Actions
Copied to rgw - Backport #36535: mimic: radosgw-admin user stats are incorrect when dynamic re-sharding is enabled ResolvedNathan CutlerActions
Copied to rgw - Backport #36536: luminous: radosgw-admin user stats are incorrect when dynamic re-sharding is enabled ResolvedCasey BodleyActions
Actions
Copy link
 #1

Updated by Mark Kogan almost 6 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Orit Wasserman almost 6 years ago

  • Priority changed from Normal to High
Actions
Copy link
 #3

Updated by Yehuda Sadeh almost 6 years ago

  • Status changed from New to Triaged
Actions
Copy link
 #4

Updated by Mark Kogan almost 6 years ago

Adding another scenario -
where the cluster was installed with resharding and then resharding was disabled and radosgw restarted in the middle of cosbench write operation
resulting in the following condition:
"marker": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8",
"bucket_id": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.18",

radosgw-admin metadata get bucket:s3testqwer1
{
    "key": "bucket:s3testqwer1",
    "ver": {
        "tag": "_BFTL0_ISwpWsXaGHkyI02j0",
        "ver": 15
    },
    "mtime": "2018-06-28 13:31:43.578495Z",
    "data": {
        "bucket": {
            "name": "s3testqwer1",
            "marker": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8",
            "bucket_id": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.18",
            "tenant": "",
            "explicit_placement": {
                "data_pool": "",
                "data_extra_pool": "",
                "index_pool": "" 
            }
        },
        "owner": "testid",
        "creation_time": "2018-06-14 13:55:37.844197Z",
        "linked": "true",
        "has_bucket_info": "false" 
    }

radosgw-admin metadata get bucket.instance:s3testqwer1:a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8                                                [10/42]
{
    "key": "bucket.instance:s3testqwer1:a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8",
    "ver": {
        "tag": "_4unVrCsMSAyG7XH5M0G7Ptj",
        "ver": 3
    },
    "mtime": "2018-06-14 14:41:03.598964Z",
    "data": {
        "bucket_info": {
            "bucket": {
                "name": "s3testqwer1",
                "marker": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8",
                "bucket_id": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8",
                "tenant": "",
                "explicit_placement": {
                    "data_pool": "",
                    "data_extra_pool": "",
                    "index_pool": "" 
                }
            },
            "creation_time": "2018-06-14 13:55:37.844197Z",
            "owner": "testid",
            "flags": 0,
            "zonegroup": "d6f2327a-9aad-45a9-b31f-9880a4d792af",
            "placement_rule": "default-placement",
            "has_instance_obj": "true",
            "quota": {
                "enabled": false,
                "check_on_raw": false,
                "max_size": -1,
                "max_size_kb": 0,
                "max_objects": -1
            },
            "num_shards": 0,
            "bi_shard_hash_type": 0,
            "requester_pays": "false",
            "has_website": "false",
            "swift_versioning": "false",
            "swift_ver_location": "",
            "index_type": 0,
            "mdsearch_config": [],
            "reshard_status": 2,
            "new_bucket_instance_id": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.11" 
        },
        "attrs": [
            {
                "key": "user.rgw.acl",
                "val": "AgKJAAAAAwIVAAAABgAAAHRlc3RpZAcAAABUZXN0IElkBANoAAAAAQEAAAAGAAAAdGVzdGlkDwAAAAEAAAAGAAAAdGVzdGlkBQM5AAAAAgIEAAAAAAAAAAYAAAB0ZXN0aWQAAAAAAAAAAAICBAAAAA8AAAAHAAAAVGVzdCBJZAAAAAAAAAAAAAAAAAAAAA=" 
            },
            {
                "key": "user.rgw.idtag",
                "val": "" 
            }
        ]
    }
}

radosgw-admin metadata get bucket.instance:s3testqwer1:a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.18
{
    "key": "bucket.instance:s3testqwer1:a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.18",
    "ver": {
        "tag": "8EKJdLtqy1iBUoVTH6JzkYZ2",
        "ver": 4
    },
    "mtime": "2018-06-17 08:44:12.670789Z",
    "data": {
        "bucket_info": {
            "bucket": {
                "name": "s3testqwer1",
                "marker": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.8",
                "bucket_id": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.4241.18",
                "tenant": "",
                "explicit_placement": {
                    "data_pool": "",
                    "data_extra_pool": "",
                    "index_pool": "" 
                }
            },
            "creation_time": "2018-06-14 13:55:37.844197Z",
            "owner": "testid",
            "flags": 0,
            "zonegroup": "d6f2327a-9aad-45a9-b31f-9880a4d792af",
            "placement_rule": "default-placement",
            "has_instance_obj": "true",
            "quota": {
                "enabled": false,
                "check_on_raw": false,
                "max_size": -1,
                "max_size_kb": 0,
                "max_objects": -1
            },
            "num_shards": 2,
            "bi_shard_hash_type": 0,
            "requester_pays": "false",
            "has_website": "false",
            "swift_versioning": "false",
            "swift_ver_location": "",
            "index_type": 0,
            "mdsearch_config": [],
            "reshard_status": 1,
            "new_bucket_instance_id": "a4bddfac-6b8e-49f9-b222-25fc2cb2c118.724274.1" 
        },
        "attrs": [
            {
                "key": "user.rgw.acl",
                "val": "AgKJAAAAAwIVAAAABgAAAHRlc3RpZAcAAABUZXN0IElkBANoAAAAAQEAAAAGAAAAdGVzdGlkDwAAAAEAAAAGAAAAdGVzdGlkBQM5AAAAAgIEAAAAAAAAAAYAAAB0ZXN0aWQAAAAAAAAAAAICBAAAAA8AAAAHAAAAVGVzdCBJZAAAAAAAAAAAAAAAAAAAAAA=" 
            },
            {
                "key": "user.rgw.idtag",
                "val": "" 
            }
        ]
    }
}
Actions
Copy link
 #5

Updated by Matt Benjamin over 5 years ago

Mark, have we addressed any of these issues in recent stats work?

Matt

Actions
Copy link
 #6

Updated by Lei Liu over 5 years ago

Actions
Copy link
 #7

Updated by Mark Kogan over 5 years ago

Thanks for the quick reproduction procedure in #27205
Those bugs seem related, I will debug.

Actions
Copy link
 #8

Updated by Casey Bodley over 5 years ago

  • Status changed from Triaged to Fix Under Review
  • Assignee changed from Orit Wasserman to Casey Bodley
Actions
Copy link
 #9

Updated by Casey Bodley over 5 years ago

  • Related to Bug #27205: Incorrect stats by "radosgw-admin user stats" after bucket resharding added
Actions
Copy link
 #10

Updated by Casey Bodley over 5 years ago

  • Backport set to luminous mimic
Actions
Copy link
 #11

Updated by Casey Bodley over 5 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #12

Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #36535: mimic: radosgw-admin user stats are incorrect when dynamic re-sharding is enabled added
Actions
Copy link
 #13

Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #36536: luminous: radosgw-admin user stats are incorrect when dynamic re-sharding is enabled added
Actions
Copy link
 #14

Updated by Nathan Cutler over 5 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF