Ceph » rgw

We recently began our upgrade testing for going from Jewel (10.2.10) to
Luminous (12.2.5) on our clusters. The first part of the upgrade went
pretty smoothly (upgrading the mon nodes, adding the mgr nodes, upgrading
the OSD nodes), however, when we got to the RGWs we started seeing internal
server errors (500s) on the Jewel RGWs once the first RGW was upgraded to
Luminous. Further testing found two different problems:

The first problem (internal server error) was seen when the container and
object were created by a Luminous RGW, but then a Jewel RGW attempted to
list the container.

The second problem (container appears to be empty) was seen when the
container was created by a Luminous RGW, an object was added using a Jewel
RGW, and then the container was listed by a Luminous RGW.

Here were all the tests I performed:

Test #1: Create container (Jewel), Add object (Jewel), List container (Jewel), Result: Success
Test #2: Create container (Jewel), Add object (Jewel), List container (Luminous), Result: Success
Test #3: Create container (Jewel), Add object (Luminous), List container (Jewel), Result: Success
Test #4: Create container (Jewel), Add object (Luminous), List container (Luminous), Result: Success
Test #5: Create container (Luminous), Add object (Jewel), List container (Jewel), Result: Success
Test #6: Create container (Luminous), Add object (Jewel), List container (Luminous), Result: Failure (Container appears empty)
Test #7: Create container (Luminous), Add object (Luminous), List container (Jewel), Result: Failure (Internal Server Error)
Test #8: Create container (Luminous), Add object (Luminous), List container (Luminous), Result: Success

It appears that we ran into these bugs because our load balancer was
alternating between the RGWs while they were running a mixture of the two
versions (like you would expect during an upgrade).