Project

General

Profile

Bug #40794

[RGW] Active bucket marker in stale instances list

Added by Aleksandr Rudenko 12 months ago. Updated 5 months ago.

Status:
New
Priority:
Normal
Target version:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

Hi,

I have luminous 12.2.12.

At 12.2.5 auto-resharding was enabled and turned off after few problems with it.
After update to 12.2.12 auto-resharding was enabled again. Now it works well as I can see.

Now I'm worried about stale instances.

For example:

I have bucket which was successfully sharded in past:

radosgw-admin bucket stats --bucket clx | grep marker

    "marker": "default.422998.196",
    "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#",

And I can see this marker in stale instances list:

radosgw-admin reshard stale-instances list | grep clx

    "clx:default.422998.196",

As I know, stale-instances list must contain only previous marker ids.

If I run:

radosgw-admin reshard stale-instances rm

can it destroy my bucket?

I have a few buckets with this problem.

History

#1 Updated by Casey Bodley 12 months ago

  • Assignee set to Abhishek Lekshmanan

#2 Updated by Aleksandr Rudenko 11 months ago

Hi Abhishek, can you help me with this problem?

I have permanent WARNING on production cluster about large OMAP for about 2 month. I need clean this warning. Please, help me.

#3 Updated by Abhishek Lekshmanan 9 months ago

What does bucket stats on the said bucket show as ID? The ID is the current index, everything else is a stale bucket index that is no longer referenced, can you do the bucket stats, and do the stale-instances list and make sure none of the ids in stale-instances match the bucket id (this should be the behaviour, there shouldn't really be any case where a current ID will show up in stale-instances, but if it does then that is a bug), if they are not the same, these are safe to remove with stale-instances rm

#4 Updated by Aleksandr Rudenko 9 months ago

radosgw-admin bucket stats --bucket clx
{
    "bucket": "clx",
    "zonegroup": "default",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": ".rgw.buckets",
        "data_extra_pool": ".rgw.buckets.extra",
        "index_pool": ".rgw.buckets.index" 
    },
    "id": "default.38502374.3",
    "marker": "default.422998.196",
    "index_type": "Normal",
    "owner": "7d5ca7d1-f8f5-42e5-817b-36a763dd780d",
    "ver": "0#116960,1#117529,2#117103,3#117299,4#117605,5#117426,6#118100,7#117055,8#116604,9#118760,10#117098,11#117673,12#116991,13#118056,14#117906,15#116064,16#121320,17#118134,18#117806,19#116608,20#117314,21#116342,22#117026,23#116993,24#116867,25#117854,26#117965,27#118668,28#117437,29#117589,30#117118,31#117955,32#118023,33#116286,34#116158,35#116300,36#115914,37#116130,38#115292,39#116587,40#115499,41#115079,42#115259,43#115152,44#115237,45#115006,46#116536,47#115161,48#114608,49#115552,50#115843,51#116788,52#115508,53#116022,54#115543,55#116285,56#114885,57#116143,58#116155,59#116720,60#116122,61#114632,62#116969,63#116018,64#114780,65#117079,66#115714,67#118016,68#115641,69#115420,70#115218,71#115027,72#114635,73#116455,74#115550,75#114995,76#115352,77#114795,78#116326,79#115716,80#115811,81#116635,82#115669,83#116605,84#115937,85#116557,86#116213,87#115013,88#116555,89#117005,90#115202,91#116683,92#116013,93#116216,94#115481,95#114302,96#115405,97#115399,98#114843,99#115507,100#115807,101#115738,102#115399,103#116151,104#116627,105#115804",
    "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0",
    "mtime": "2019-10-18 11:04:48.899146",
    "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#",
    "usage": {
        "rgw.none": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 3462163
        },
        "rgw.main": {
            "size": 26008987553891,
            "size_actual": 26016839872512,
            "size_utilized": 26008991748380,
            "size_kb": 25399401909,
            "size_kb_actual": 25407070188,
            "size_kb_utilized": 25399406005,
            "num_objects": 3800908
        },
        "rgw.multimeta": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 13471743
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1024,
        "max_size_kb": 0,
        "max_objects": -1
    }
}
radosgw-admin reshard stale-instances list | grep clx
    "clx:default.422998.196",

I can't see current bucket ID in in stale instances list. But how about marker?. It is correct if marker in stale instances list?
Are you sure that i can run "reshard stale-instances rm"?

#5 Updated by Abhishek Lekshmanan 6 months ago

Marker is the first bucket instance that was created, usually for non resharded buckets marker and ID will be similar, however after a reshard marker is no longer related to the bucket ID, and the instance with the older ID (ie. marker) can be removed. What you need to make sure is that bucket stale-instance list no longer lists anything that has a ID field in bucket stats as these are the active instances.

#6 Updated by Aleksandr Rudenko 5 months ago

Thank you Abhishek!

All stale indexes were successfully removed.

You can close this issue.

Also available in: Atom PDF