Ceph » CephFS

I've got a bunch of krbd test failures which I tracked down to a recent change in "rados ls" behavior.

Before:

$ rbd create --size 16 --image-feature layering a
<map and fill>
$ rados -p rbd ls
rbd_directory
rbd_data.10116b8b4567.0000000000000003
rbd_data.10116b8b4567.0000000000000001
rbd_info
rbd_data.10116b8b4567.0000000000000002
rbd_data.10116b8b4567.0000000000000000
rbd_header.10116b8b4567
rbd_id.a
$ rbd snap create a@snap
<discard -- delete HEADs>
$ rados -p rbd ls
rbd_directory
rbd_info
rbd_header.10116b8b4567
rbd_id.a

After, deleted HEADs are listed:

$ rbd create --size 16 --image-feature layering a
<map and fill>
$ rados -p rbd ls
rbd_data.100f6b8b4567.0000000000000002
rbd_data.100f6b8b4567.0000000000000003
rbd_directory
rbd_info
rbd_data.100f6b8b4567.0000000000000001
rbd_data.100f6b8b4567.0000000000000000
rbd_id.a
rbd_header.100f6b8b4567
$ rbd snap create a@snap
<discard -- delete HEADs>
$ rados -p rbd ls
rbd_data.100f6b8b4567.0000000000000002
rbd_data.100f6b8b4567.0000000000000003
rbd_directory
rbd_info
rbd_data.100f6b8b4567.0000000000000001
rbd_data.100f6b8b4567.0000000000000000
rbd_id.a
rbd_header.100f6b8b4567

This is both on filestore and bluestore, in 6152fd9f01fffbb1a2a96c46bb12f30a31da3257..404cee744fb6891fe7bf48cfe012cecdc1c2eba1 range. Josh confirms that this wasn't intended -- I suspect the snapset refactoring.

Hmm, listing and snapshots have never gotten along well, but that makes me think we should list stuff even if it doesn't have a HEAD object — we don't support listing of non-HEAD so without this we can't see clones at all.

It is annoying for the FS and rbd tests, but I'd rather change those than have "invisible" objects.

I don't think we want a plain "rados ls" with no flags (--all?) output something that can't be "rados stat"ed. Support for listing of non-HEAD needs to be thought through: how is non-HEAD distinguished from HEAD in the output, how do we expose unique versions, which commands accept non-HEAD as input and which don't, etc.

Agree with Greg. Also: making pgnls skip whiteout objects means we need to load the object_info_t, which is significantly more expensive.

We do have the list-snaps operation to list snaps on an object--that should work on a whiteout. It isn't immediately clear that that's what you have to do, yes, but you can infer it from an ENOENT.

Also, we've yet to come across a real user of the pgnls interface that needs to do this. The only users right now are cephfs repair and all of the tests.

Just to check I understand -- the claim is that we are seeing the object in pgnls output because it has a snapshot that hasn't been removed yet?

If so, we should be able to just update the cephfs test to do a wait_until_true while it waits for the removed_snaps in the osdmap to propagate to the OSDs.

Yes, that's correct.

FWIW that is basicaly what we did with the rados api test cleanup failures (loop waiting for snaptrimmer to do its thing).