Ceph » rbd

Jacobus Erasmus wrote:

When a rbd_plugins is set incorrectly (no plugin available) it will accept the change without any error.

Hi Jacobus,

I think this first part is intended because one could run "rbd config image set" (or any other centralized configuration) command on one node and then attempt to open the image on another node. It's not required that the same set of plugins in installed on every node so little can be done in terms of meaningful validation.

If you later try and change the plugin (legitimate or not) rbd gives and error.ix it will fail.

Eg.
rbd config image set xppool/test rbd_plugins test
works (No Error)
rbd config image set xppool/test rbd_plugins pwl_cache
gives

2024-03-07T11:20:21.559+0200 7fe793ff76c0 -1 load failed dlopen(): "/usr/lib/ceph/librbd/libceph_librbd_test.so: cannot open shared object file: No such file or directory" or "/usr/lib/ceph/libceph_librbd_test.so: cannot open shared object file: No such file or directory"
2024-03-07T11:20:21.559+0200 7fe793ff76c0 -1 librbd::PluginRegistry: 0x55c06160d560 init: failed to load plugin: test
2024-03-07T11:20:21.559+0200 7fe793ff76c0 -1 librbd::image::OpenRequest: failed to initialize plugin registry: (38) Function not implemented
2024-03-07T11:20:21.559+0200 7fe790ff16c0 -1 librbd::ImageState: 0x55c06160db80 failed to open image: (38) Function not implemented
rbd: error opening image test: (38) Function not implemented

Although not intended, this second part kind of falls out from the first and the use of "rbd config image set" command (i.e. a centralized configuration override) instead of a local ceph.conf file. Because image-level configuration overrides are stored in the image itself, they can be manipulated only after opening the image. After rbd_plugins is set to a bad value, the image can't be opened because the plugin(s) can't be located, so undoing becomes a catch-22 problem. To get out, one can remove the configuration override from the image header by using the rados tool but it requires some familiarity with RBD internals.

There doesn't appear to be an easy fix here as rbd_plugins is really one-of-a-kind special option -- no other option is supposed to be able to cause something like this. We could disallow centralized configuration overrides for rbd_plugins but some might consider that too restrictive.