Ceph » bluestore

for the missing namespace, its key was looking like that :

key (31 bytes):
00000000  00 00 00 00 00 00 00 01  00 00 00 00 01 65 0a de  |.............e..|
00000010  2e 6e 61 6d 65 5f 69 6e  73 74 61 6e 63 65 73     |.name_instances|
0000001f

value (0 bytes) :

Actually this behavior looks expected for a partially removed image.

Some time ago the behavior of rbd remove command was changed to move the image to the trash as its first step and the proceed with removing objects (truncating) and then removing metadata. If the removal is interrupted the image is still in the trash but it is not shown by "normal" `rbd trash ls` command.

The `rbd ls` was also updated to list images from both `rbd_directory` and `rbd trash` (only those in "removing state" here). It was supposed to be useful for users to detect those partially removed images and complete the removal. For this you just need to run `rbd rm {image}` again.

The simplest way to check if it is the case I suppose is to run:

rados -p rbd listomapvals  rbd_trash

and see if you can find the image in the list here. You can provide the output here for review if unsure.

It looks like the current rbd behavior is rather confusing and could be actually improved. I remember we had some improvements in dashboard already, so if you use dashboard you may also want to try listing images there. In newer version the dashboard should show such images in "removing" state.

After checking, there are some images in the trash, but the one we have issues with in these cluster were not deleted, and are not present in the trash

Florian Florensa wrote:

After checking, there are some images in the trash, but the one we have issues with in these cluster were not deleted, and are not present in the trash

Just to make sure, you used `rados -p rbd listomapvals rbd_trash` command, right? (not `rbd trash ls`)

Ah, sorry, when reading the first time I was too fast and somehow missed your description about the non-binary chars in key names, and that fixing it fixed the problem. Yes, this looks strange indeed.

I was basing my assumption about partially removed image just on the `rbd ls` and `rbd info` behavior you experienced and that looked the same as if you had a partially removed image. Now I see that this is not the case.

Is the problem observed for images created before upgrade or after?

All the one that encountered this issue so far have all been created before the upgrade to ceph 16

Florian, this 17-byte prefix in the keys makes me think you're bitten by https://tracker.ceph.com/issues/53062

I still don't fully understand why this doesn't cause some failures at earlier stages, e.g. on OSD startup, but it looks like the keys are broken in the similar manner.
Could you please stop any arbitrary OSD and run:
ceph-kvstore-tool bluestore-kv <path-to-osd> list p

Please share the output to stdout if any

Florian Florensa wrote:

Here is the output, and the keys prefix in there looks familiar with what i was seeing in the omapvals

Well, there is some mess in OMAP keys indeed, compare these two records:
p %00%00%00%00%00%00%00%03%9d%1cj%e0%00%00%00%00%00%02%fd%b8.%00%00%00%00%00%00%00%03%00%00%00%00%00%00M%3b.20211028-001726
p %00%00%00%00%00%00%00%03%9d%1cj%e0%00%00%00%00%00%02%fd%b8.20211029-001108

The first ne has got a redundant substring in the middle:
%00%00%00%00%00%00%00%03%00%00%00%00%00%00M%3b.

And apparently something happened to cluster on Oct 28/29 2021 - every invalid record group ends with 20211028 in the key and good one starts with 20211029.

To a great extent this is similar to #53062 but there are still some unclear differences which prevents from exact matching to that ticket. And unfortunately our recovery tools are not applicable for now.
Working on both root cause analysis and recovery procedure...

I upgraded the cluster in the end of october, and by crawling a chat that i had with a coworkers the correct record group that starts from the 28th are the one i reinjected myself manually to repair basic functionnality on some images (IE, prior to that a volume that was in use was still usable, but detaching it made it impossible to reattach, i guess it is a matter of most of those metadata are not used after the connection is established)
We got another volume that started acting up today, but the reason i guess, is that the VM using it got restarted, and it was unable to "re-attach" to this volume because everything that tries to "stat(2)" it does not work.

Here is a summary for another similar case shared by the affected cluster's admin. I presume it provides good enough explanation how current state in this ticket has been achieved as well:
"
well, one of my osds didn't crash after my upgrade like the other ones did.
After repairing the crashing osds (and not repairing the one that didn't
crash) all rbd images were visible in rbd info (and usable as disk) but after
a few hours of resyncing the cluster, some rbd disks weren't accessible
anymore.

My guess is, that the osd that did not crash still had wrong omap keys and the
rebuilding spread them over the cluster or somehow activated these wrong omap
keys.
"
So we can probably mark this ticket as a duplicate of #53062 and apparently close. Fabian, any objecttions?