Project

General

Profile

Bug #55546

rgw: trigger dynamic reshard on index entry count rather than object count

Added by J. Eric Ivancich 7 months ago. Updated 6 months ago.

Status:
Triaged
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Tags:
reshard low-hanging-fruit
Backport:
quincy,pacific,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

With versioned buckets, there can be as many as 4 bucket index entries per object (when there is one version). That can allow ceph to complain about large omaps. Consider using number of bucket index entries rather than object count to determine when to issue a dynamic reshard.

History

#1 Updated by Casey Bodley 7 months ago

  • Status changed from New to Triaged
  • Tags set to reshard

#2 Updated by J. Eric Ivancich 7 months ago

Currently reshard is triggered on bucket stats as there's a field for number of objects. Currently there's not a field for number of bucket index entries. That could be added, but attention needs to be given to backwards compatability.

#3 Updated by Casey Bodley 7 months ago

that would require 'bucket stats' to track individual omap entries. i think it would be a lot easier for rgw to just scale rgw_max_objs_per_shard based on whether or not the bucket is versioned

#4 Updated by Benoît Knecht 7 months ago

I just want to note that if you want to scale `rgw_max_objs_per_shard` for versioned buckets, there's a special case where you have 6 index entries per object: deleted versioned objects.

```
$ radosgw-admin bi list --bucket=version | jq '.[]|select(.idx|contains("deleted.txt"))|{type,idx}' {
"type": "plain",
"idx": "deleted.txt"
} {
"type": "plain",
"idx": "deleted.txt\u0000v912\u0000iBbqNwu1mPL81fzD.s0aw1f3KzoRp-01"
} {
"type": "plain",
"idx": "deleted.txt\u0000v913\u0000ig3Ow6494L-Adys21ghi4hPkmyqjG8SH"
} {
"type": "instance",
"idx": "�1000_deleted.txt\u0000iBbqNwu1mPL81fzD.s0aw1f3KzoRp-01"
} {
"type": "instance",
"idx": "�1000_deleted.txt\u0000ig3Ow6494L-Adys21ghi4hPkmyqjG8SH"
} {
"type": "olh",
"idx": "�1001_deleted.txt"
}
```

So if each object has exactly one version, you get a factor of 4 between the number of index entries and the number of objects, but it tends to a factor of 2 when you increase the number of versions per object. However, it can get as bad as a factor of 6 if every versioned object is deleted.

So I don't know what the ideal scaling factor would be. 6 would ensure that we never underestimate the real value, but it seems very unlikely to get that large in practice. And for users who keep many versions of every object, even a factor of 4 would be overkill.

#5 Updated by Casey Bodley 6 months ago

  • Tags changed from reshard to reshard low-hanging-fruit

Also available in: Atom PDF