Ceph » rbd

I fetch the rbd_object_map for the image and the snapshot, the rbd_object_map(s) of them are different.

The command I used is as follows:

rados -p reppool get rbd_object_map.<id> image_map
rados -p reppool get rbd_object_map.<id>.<snapseq> snap_map

hexdump -Cv image_map
hexdump -Cv snap_map

On the OSD side, the snapshot is marked as "removed" (in the removed_snaps set). So the find_object_context function returns "-ENOENT" and librbd returns all zero block to the client.

However, the snapshot is NOT removed by us, because the snapshot is protected all the time. And the rbd_header.<id> block in the "reppool" pool still records the snapshot all the time.

Are you using the RBD data-pool feature on this image? The initial version of Luminous had a bug where the snapshots were recorded in the image header pool instead of the data pool.

How to check whether the data-pool feature is enabled?

I use the command "rbd info reppool/<image-id>" on the image, and the output contains the following information about features：

features: layering, exclusive-lock, object-map, fast-diff, deep-flatten

Jason Dillaman wrote:

Are you using the RBD data-pool feature on this image? The initial version of Luminous had a bug where the snapshots were recorded in the image header pool instead of the data pool.

OK, then it's not enabled.

At this point, I don't have enough information to assist trying to determine how your snapshot was deleted. The only way that librbd could do it would be if you had an unprotected snapshot, started the snap remove process and aborted it after it deleted the RADOS snapshot but before it removed the snapshot record from the image header, and then protected the snapshot. If you can discover how to repeat this issue, we will look into it.

This issue can be triggered by deleting a RBD image from a cache tiering pool. But the probability of the occurrence is not 100%.

When I delete a RBD image, other images in the same cache tiering pool might be destroyed.

So you are saying that if you remove image X via 'rbd rm <base-tier-pool>/<image-name>', running "ceph osd pool ls detail --format json-pretty | grep removed" for your pool will show new, unrelated snapshots added to the collection?

Yes, this is it.

Jason Dillaman wrote:

So you are saying that if you remove image X via 'rbd rm <base-tier-pool>/<image-name>', running "ceph osd pool ls detail --format json-pretty | grep removed" for your pool will show new, unrelated snapshots added to the collection?

I cannot reproduce what you are seeing. Can you please provide exact CLI commands and CLI output?