Stale bucket index entry remains after object deletion
We have a race condition:
RGW client 1: requests an object be deleted. RGW client 1: sends a prepare op to bucket index OSD 1. OSD 1: prepares the op, adding pending ops to the bucket dir entry RGW client 2: sends a list bucket to OSD 1 RGW client 2: sees that there are pending operations on bucket dir entry, and calls check_disk_state RGW client 2: check_disk_state sees that the object still exists, so it sends CEPH_RGW_UPDATE to bucket index OSD (1) RGW client 1: sends a delete object to object OSD (2) OSD 2: deletes the object RGW client 2: sends a complete op to bucket index OSD (1) OSD 1: completes the op OSD 1: receives the CEPH_RGW_UPDATE and updates the bucket index entry, thereby RECREATING it
At step 5 the object's dir entry exists. If we get to beginning of step 11 and the object's dir entry no longer exists, we know that the dir entry was just actively being modified, and ignore the CEPH_RGW_UPDATE operation, thereby NOT recreating it.