Ceph » RADOS

This is Luminous, 12.2.12 by now.

Balancing on bytes (reweight-by-utilization) was unable to resolve the issue previously. When the clients were confirmed to be recent enough, we were going for the upmap balancer in the hope of this helping. And we got perfectly balanced pg counts, unfortunately the PGs themselves are really quite uneven.

Greg Farnum wrote:

It sure looks like the PG count isn't a power of two, so some of them are simply half size compared to the others. (Since they cluster around ~17200 and 33500 objects — although I'm surprised it's not exactly doubled).

My last info is that the pool in question (id 2) has 4096 PGs.

Jan Fajerski wrote:

Greg Farnum wrote:

It sure looks like the PG count isn't a power of two, so some of them are simply half size compared to the others. (Since they cluster around ~17200 and 33500 objects — although I'm surprised it's not exactly doubled).

My last info is that the pool in question (id 2) has 4096 PGs.

Correction: The pool in question has 6144 PGs.

I'm however surprised how large an imbalance this cluster shows. Again, PGs are perfectly balanced through the upmap balancer. Regardless the least used OSD uses 1.61TiB or 45.96% of its capacity, whereas the most heavily used OSD occupies 2.60TiB or 74.31% of its capacity.
Is the impact of not using a power of two for the PG count really that high?

Lars Marowsky-Brée wrote:

This is Luminous, 12.2.12 by now.

Balancing on bytes (reweight-by-utilization) was unable to resolve the issue previously. When the clients were confirmed to be recent enough, we were going for the upmap balancer in the hope of this helping. And we got perfectly balanced pg counts, unfortunately the PGs themselves are really quite uneven.

The reweight-by-utilization command has always been pretty weak and is not what I meant: the upmap balancer (in some releases) has a mode that balances for bytes rather than PG counts.

Jan Fajerski wrote:

Jan Fajerski wrote:

Greg Farnum wrote:

It sure looks like the PG count isn't a power of two, so some of them are simply half size compared to the others. (Since they cluster around ~17200 and 33500 objects — although I'm surprised it's not exactly doubled).

My last info is that the pool in question (id 2) has 4096 PGs.

Correction: The pool in question has 6144 PGs.

I'm however surprised how large an imbalance this cluster shows. Again, PGs are perfectly balanced through the upmap balancer. Regardless the least used OSD uses 1.61TiB or 45.96% of its capacity, whereas the most heavily used OSD occupies 2.60TiB or 74.31% of its capacity.
Is the impact of not using a power of two for the PG count really that high?

PGs split by splitting their hash range in half. So if you have not-a-power-of-two, some of the PGs are twice the size of the other. Our expectation in the past is that this shouldn't be an issue in normal use, and it mostly isn't, but it does make some difference. It's possible the balancer has some behavior that exacerbates things though.

Greg Farnum wrote:

PGs split by splitting their hash range in half. So if you have not-a-power-of-two, some of the PGs are twice the size of the other. Our expectation in the past is that this shouldn't be an issue in normal use, and it mostly isn't, but it does make some difference. It's possible the balancer has some behavior that exacerbates things though.

I was wondering if this could also be exasperated by the erasure coding setup. They run 10+2 with failure domain host and have 17 hosts. Could this create a situation where the larger and smaller PGs are not (almost) randomly distributed to all OSDs but the distributions ends up being somewhat regular? Basically does a ruleset like this reduce the potential for randomness of crushs distribution?

In any case, given the significant imbalance this can produce, should we warn more explicitly in the docs against using pg counts that are not a power of two?

Jan Fajerski wrote:

Greg Farnum wrote:

PGs split by splitting their hash range in half. So if you have not-a-power-of-two, some of the PGs are twice the size of the other. Our expectation in the past is that this shouldn't be an issue in normal use, and it mostly isn't, but it does make some difference. It's possible the balancer has some behavior that exacerbates things though.

I was wondering if this could also be exasperated by the erasure coding setup. They run 10+2 with failure domain host and have 17 hosts. Could this create a situation where the larger and smaller PGs are not (almost) randomly distributed to all OSDs but the distributions ends up being somewhat regular? Basically does a ruleset like this reduce the potential for randomness of crushs distribution?

I don't think anybody's looked at this in depth. A lot of things look very different when the "branch factor" is not small compared to the number of items in a CRUSH bucket, so it's definitely possible

In any case, given the significant imbalance this can produce, should we warn more explicitly in the docs against using pg counts that are not a power of two?

I haven't looked at it in a while but I was comfortable with the statements we made back when last looking at this. Mark Nelson has looked at this a lot and is probably your ally if you want to look at it more though. :)