Bug #53173
closedpubsub API reports is_truncated: "true" but provides no events
0%
Description
Seen on a containerised cluster running across 3 VMs with 3x MON, 3x OSD, 3x MDS, 6x RGW (3 regular, 3 pubsub).
ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)
During testing we had a pubsub queue grow very large (2.5M events) which were all subsequently acknowledged.
Some more details on this context of this large queue can be found below:
- https://tracker.ceph.com/issues/52963
- https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/DPPEPYPAWLQIRPRZAEJAWJ72S2W6INNN/
Following this I saw some odd behaviour:
Expected outcome ================
The pubsub queue reduces to 0. When calling the API for more events I expect a response of:
{"next_marker":"","is_truncated":"false","events":[]}
Actual outcome ==============
When calling the API for more events I observe the response:
{"next_marker":"albansstacksub/1636029768.704860.1ee5a51c","is_truncated":"true","events":[]}
Passing the "next marker" into the next query, this repeats 3 times before eventually responding (on the 3rd call):
{"next_marker":"","is_truncated":"false","events":[]}
I don't think it should be possible to return both "is_truncated":"true" and an empty list of events.
Might also be worth noting that there claims to be 269 objects in the pubsub data pool:
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
siteBpubsub.rgw.buckets.data 19 4 100 KiB 269 3.2 MiB 0 192 GiB
This interestingly correlates to the actual outcome above. The API call goes 3 loops (of 100), almost as if there are 269 real events in the pubsub queue that are not being reported in the events field of the API response, and hence going unacknowledged.
Updated by Neha Ojha over 2 years ago
- Project changed from Ceph to rgw
- Category deleted (
rest-api)
Updated by Casey Bodley over 2 years ago
- Assignee set to Yuval Lifshitz
- Tags set to pubsub
Updated by Yuval Lifshitz over 1 year ago
- Status changed from New to Won't Fix
pubsub functionality was removed from ceph in: https://github.com/ceph/ceph/pull/48996