Ceph » rgw

Seen on a containerised cluster running across 3 VMs with 3x MON, 3x OSD, 3x MDS, 6x RGW (3 regular, 3 pubsub).

ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)

During testing we had a pubsub queue grow very large (2.5M events) which were all subsequently acknowledged.

Some more details on this context of this large queue can be found below:
- https://tracker.ceph.com/issues/52963
- https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/DPPEPYPAWLQIRPRZAEJAWJ72S2W6INNN/

Following this I saw some odd behaviour:

Expected outcome ================
The pubsub queue reduces to 0. When calling the API for more events I expect a response of:

{"next_marker":"","is_truncated":"false","events":[]}

Actual outcome ==============
When calling the API for more events I observe the response:

{"next_marker":"albansstacksub/1636029768.704860.1ee5a51c","is_truncated":"true","events":[]}

Passing the "next marker" into the next query, this repeats 3 times before eventually responding (on the 3rd call):

{"next_marker":"","is_truncated":"false","events":[]}

I don't think it should be possible to return both "is_truncated":"true" and an empty list of events.

Might also be worth noting that there claims to be 269 objects in the pubsub data pool:

POOL                           ID  PGS  STORED   OBJECTS  USED     %USED  MAX AVAIL
siteBpubsub.rgw.buckets.data   19    4  100 KiB      269  3.2 MiB      0    192 GiB

This interestingly correlates to the actual outcome above. The API call goes 3 loops (of 100), almost as if there are 269 real events in the pubsub queue that are not being reported in the events field of the API response, and hence going unacknowledged.