Rgw - plugin architecture


There is a growing amount of functionality in RGW like the S3 API, Swift API, and Keystone Auth. Right now this is not really segregated in the code; we even have a "bool do_swift" in rgw_main::main(). Besides that, we have a cluttered namespace (for instance, we have a suffix '_S3' to all S3 classes and '_SWIFT' to all Swift classes, but Keystone functionality is only available in RGWSwift)
A plugin architecture might solve this.


Interested Parties

Current Status

I'm currently implementing the proposed solution described below here; This is specific for a "AUTH" type plugin for keystone support. For this, I'm also implementing an authentication pipeline mechanism/internal API. This falls a bit out of the scope of this blueprint however.

Detailed Description

I propose to introduce a simple internal API for the plugin system which is build upon a newly RGWPluginManager. Other components ("owners") in the RGW can request to "register" plugins of a specific type. For instance, RGWREST might want to register plugins of type RGW_PLUGIN_TYPE_REST_API. The API between the RGWPluginManager and the plugin can be very simple and flexible, because the plugin can count on an "owner" of a specific type (e.g. a RGW_PLUGIN_TYPE_REST_API type of plugin will always have an instance of RGWREST as its "owner") and can access it directly.


  • Plugin manager: instance of RGWPluginManager
  • Owner: another component within RGW that requests access to plugins through the plugin manager
  • Type: predefined type of a plugin, for instance RGW_PLUGIN_TYPE_REST_API or RGW_PLUGIN_TYPE_AUTH
  • Loading: dynamic load the plugin, making it known to the plugin manager
  • Registering: happens when an owner requests access to plugins; this actually initializes a new instance of the plugin
  • Plugin: dynamic loadable .so, exporting rgw_plugin_init which is used by the plugin manager

Layout of a plugin

Plugins are in src/rgw/plugins/[pluginname] and are compiled into a .so. They export at least one symbol:

extern "C" rgw_plugin* rgw_plugin_init(); 

This function returns a new instance of rgw_plugin, a struct defined in rgw_plugin.h as follows;

struct rgw_plugin {
  const uint32_t type;
  const bool is_singular;
  int (* init)(rgw_plugin *);
  int (* exit)(rgw_plugin *);
  RGWPluginManager * plugin_manager;
  void * owner;
  CephContext *context;

It needs to set at least the type of plugin, whether it can be registered multiple times by multiple owners (is_singular), and the init and exit functions that the plugin manager can use for each time this plugin gets registered. The plugin manager sets the plugin_manager pointer to itself, so the plugin knows where to find the plugin manager when it needs to. This is useful for instance when a plugin needs/wants access to another plugin.

Step 1 - Loading a plugin
First, the plugin manager needs to be initialized (this maybe should be a global?). For instance, in main() in;

 RGWPluginManager *pm = new RGWPluginManager(g_ceph_context);
  if (pm->load_plugins() < 0)
    return 1;

The load_plugins() function will automatically load the plugins that are configured in OPTION(rgw_load_plugins). From this moment on, these plugins can be registered by owners when they want to.
This is a separate call, because a client that will implement librgw in the future might want to load only specific plugins using a load_plugin() method.

Step 2 - Registering a plugin
For instance, the RGWREST might want to access plugins that supply new APIs (type RGW_PLUGIN_TYPE_REST_API). In main();

RGWREST *rest = new RGWREST(pm);

And in the RGWREST constructor;

RGWREST::RGWREST(RGWPluginManager *_pm)
: plugin_manager(_pm) {
  plugin_manager->register_plugins(RGW_PLUGIN_TYPE_REST_API, (void *) this);

It also supplies a void pointer to itself, so the plugin manager can register this instance of RGWREST as the owner. The plugin manager will make a copy of the rgw_plugin definition, and link it to the correct owner (e.g. set the void pointer). Then, it runs the init() function of the plugin to notify the plugin that an owner wants to use this plugin and it needs to initialize.
Within the plugin manager, a list of registered plugins is maintained;

map<string, list<rgw_plugin*> > plugin_map;

Where the string is the "name" of the plugin and the list is the list of copies of the rgw_plugin struct that the plugin supplied during loading. Of course, when the plugin has is_singular set to true, and this plugin is already registered to another owner, it fails.
Important is that the owner "unregisters" plugins also;

  plugin_manager->unregister_plugins(RGW_PLUGIN_TYPE_REST_API, (void *) this);

This will cause the plugin manager to call the exit() function of the plugin, remove the effective copies of the rgw_plugin plugin definitions from the plugin_map, and clean up nicely.

Work items

Coding tasks

  1. Creating the RGWPluginManager
  2. Modify the so that the plugin manager gets initialized.
  3. For future RGW_PLUGIN_TYPE_REST_API type API support:
    1. Modify the RGWREST constructor/destructor so that it can be passed an instance of RGWPluginManager
  4. For future RGW_PLUGIN_TYPE_AUTH type API support:
    1. Modify the RGWProcess constructor/destructor so that it can be passed an instance of RGWPluginManager

Documentation tasks

  1. Document the RGWPluginManager <-> Plugin API and requirements