Bug #40794
open[RGW] Active bucket marker in stale instances list
0%
Description
Hi,
I have luminous 12.2.12.
At 12.2.5 auto-resharding was enabled and turned off after few problems with it.
After update to 12.2.12 auto-resharding was enabled again. Now it works well as I can see.
Now I'm worried about stale instances.
For example:
I have bucket which was successfully sharded in past:
radosgw-admin bucket stats --bucket clx | grep marker
"marker": "default.422998.196",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#",
And I can see this marker in stale instances list:
radosgw-admin reshard stale-instances list | grep clx
"clx:default.422998.196",
As I know, stale-instances list must contain only previous marker ids.
If I run:
radosgw-admin reshard stale-instances rm
can it destroy my bucket?
I have a few buckets with this problem.
Updated by Aleksandr Rudenko over 4 years ago
Hi Abhishek, can you help me with this problem?
I have permanent WARNING on production cluster about large OMAP for about 2 month. I need clean this warning. Please, help me.
Updated by Abhishek Lekshmanan over 4 years ago
What does bucket stats on the said bucket show as ID? The ID is the current index, everything else is a stale bucket index that is no longer referenced, can you do the bucket stats, and do the stale-instances list and make sure none of the ids in stale-instances match the bucket id (this should be the behaviour, there shouldn't really be any case where a current ID will show up in stale-instances, but if it does then that is a bug), if they are not the same, these are safe to remove with stale-instances rm
Updated by Aleksandr Rudenko over 4 years ago
radosgw-admin bucket stats --bucket clx
{
"bucket": "clx",
"zonegroup": "default",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": ".rgw.buckets",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index"
},
"id": "default.38502374.3",
"marker": "default.422998.196",
"index_type": "Normal",
"owner": "7d5ca7d1-f8f5-42e5-817b-36a763dd780d",
"ver": "0#116960,1#117529,2#117103,3#117299,4#117605,5#117426,6#118100,7#117055,8#116604,9#118760,10#117098,11#117673,12#116991,13#118056,14#117906,15#116064,16#121320,17#118134,18#117806,19#116608,20#117314,21#116342,22#117026,23#116993,24#116867,25#117854,26#117965,27#118668,28#117437,29#117589,30#117118,31#117955,32#118023,33#116286,34#116158,35#116300,36#115914,37#116130,38#115292,39#116587,40#115499,41#115079,42#115259,43#115152,44#115237,45#115006,46#116536,47#115161,48#114608,49#115552,50#115843,51#116788,52#115508,53#116022,54#115543,55#116285,56#114885,57#116143,58#116155,59#116720,60#116122,61#114632,62#116969,63#116018,64#114780,65#117079,66#115714,67#118016,68#115641,69#115420,70#115218,71#115027,72#114635,73#116455,74#115550,75#114995,76#115352,77#114795,78#116326,79#115716,80#115811,81#116635,82#115669,83#116605,84#115937,85#116557,86#116213,87#115013,88#116555,89#117005,90#115202,91#116683,92#116013,93#116216,94#115481,95#114302,96#115405,97#115399,98#114843,99#115507,100#115807,101#115738,102#115399,103#116151,104#116627,105#115804",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0",
"mtime": "2019-10-18 11:04:48.899146",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#",
"usage": {
"rgw.none": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 3462163
},
"rgw.main": {
"size": 26008987553891,
"size_actual": 26016839872512,
"size_utilized": 26008991748380,
"size_kb": 25399401909,
"size_kb_actual": 25407070188,
"size_kb_utilized": 25399406005,
"num_objects": 3800908
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 13471743
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1024,
"max_size_kb": 0,
"max_objects": -1
}
}
radosgw-admin reshard stale-instances list | grep clx
"clx:default.422998.196",
I can't see current bucket ID in in stale instances list. But how about marker?. It is correct if marker in stale instances list?
Are you sure that i can run "reshard stale-instances rm"?
Updated by Abhishek Lekshmanan over 4 years ago
Marker is the first bucket instance that was created, usually for non resharded buckets marker and ID will be similar, however after a reshard marker is no longer related to the bucket ID, and the instance with the older ID (ie. marker) can be removed. What you need to make sure is that bucket stale-instance list no longer lists anything that has a ID field in bucket stats as these are the active instances.
Updated by Aleksandr Rudenko about 4 years ago
Thank you Abhishek!
All stale indexes were successfully removed.
You can close this issue.