Generic support for plugins installation and upgrade¶

Summary¶

Ceph daemons rely on plugins (cls, erasure code or key/value store backends in the future) that are dynamically loaded. At present the plugins are considered an integral part of the daemon and their lifecycle is the same. It would be convenient to have generic support to install a new plugin in an existing cluster or upgrade a plugin without requiring an upgrade of the daemons.
Integration tests should demonstrate that all border cases are taken care of by a given plugin version. That not only includes testing what happens when a plugin is used in the ideal situation, but also checks what happens during upgrades, when a cluster runs various Ceph versions and / or plugin versions, when a plugin cannot be installed etc. It is essentially what https://github.com/ceph/ceph-qa-suite do for Ceph but applied at a smaller scale to each plugin.
Dependency management is seemingly trivial to implement but easy to get wrong. Instead of inventing something, plugins can be packaged using whatever packaging system the host uses (apt / yum). Although it raises the requirements for plugin authors (they must know packaging in addition to development), there is very little difference in terms of packaging from one plugin to the other. Most of the time, the developer can copy/paste an existing plugin package and focus on changing the version requirements according to what the plugin needs.

Plugin support requires:

• A documented plugin API to help plugin authors (for instance ErasureCodeInterface)
• An ABI for the Ceph daemons to consume (for instance ErasureCodePlugin::load)
• Plugin versions (similar to shared libraries versions)
• Distributing the plugin on the hosts where the daemons loading them are running (packages)

Owners¶

• Loic Dachary (Red Hat)
• Name (Affiliation)
• Name

Interested Parties¶

• Miyamae, Takeshi (Fujitsu)
• Varada Kari (SanDisk)
• Danny Al-Gaaf (Deutsche Telekom)
• Name

Current Status¶

Detailed Description¶

Adding to the plugin whitelist

• The shec erasure code plugin is packaged for CentOS, for i386 and x86_64 and the RPM contains the libec_shec.so.1.0.0 plugin
• ceph plugin-whitelist add shec schedule loading the shec plugin on all MON and OSDs
• when all MON and OSDs have the shec plugin, it is added to the whiltelist

Remove a plugin from the whitelist

• ceph plugin-whitelist remove shec fails if the plugin is in use
• it is removed from the whitelist and not necessarily unloaded from MON / OSD but it won't be loaded if the MON or OSD is restarted

Checking the whitelist before allowing a plugin to be used

• The ceph erasure-code-profile set myprofile plugin=shec ... command succeeds if the shec package is installed on all OSDs and MONs, otherwise it fails
• When all monitors and all OSDs have the plugin, it is added to the whitelist of plugins
• When a monitor tries to join the quorum, the missing plugins are distributed to it before it can be accepted. The same is true for the OSDs. If there does not exist a package for the OSD or Monitor architecture and operating system, it cannot join the cluster.

Part of this logic applies to cls plugins as well and can be used for OSD objectstore backends.

Work items¶

Coding tasks¶

1. create a C ABI for erasure code plugins
2. package erasure code plugins (ceph-erasure-code-jerasure, ceph-erasure-code-isa etc.)
3. add a whitelist of erasure code plugin in osdmap (monmap ?) with version constraints (jerasure >= 0.87 etc)
4. each osd in osdmap has a list of loaded plugins and their version
5. when a mon / osd joins the cluster, it goes over the whitelist of erasure code plugin and apt-get/yum install it and exits on error if it can't
6. ceph plugin-whitelist add shec
   1. for each osd from the osdmap send a message to load the shec plugin
   2. the osd runs yum / apt-get install the shec plugin (version check & al happen there)
   3. load the plugin and send the list of loaded plugins back to the MON
   4. the same is done for each mon in the monmap
   5. when the list of loaded plugins has been received for each OSD and mon, update the whitelist to add shec

Build / release tasks¶

1. Task 1
2. Task 2
3. Task 3

Documentation tasks¶

1. Task 1
2. Task 2
3. Task 3

Deprecation tasks¶

1. Task 1
2. Task 2
3. Task 3