Ceph » rgw

Well, this needs a description, so here goes.

When rgw makes outbound http requests, it uses libcurl. For keystone, this results in setting up a CURL* structure, using it once, then tearing it down. A direct consequence of this, on all platforms, is setting up and tearing down an SSL connection to keystone each and every keystone validation call. This is somewhat expensive for swift (depending on how efficiently the client reuses tokens), and very expensive for s3 auth (because each aws signature requires validating the hash in keystone).

Our current platform specific distribution choices for libcurl make this worse. On ubuntu, we use libcurl-gnutls. gnutls turns out to be very innefficient about memory usage, starts up and tears down a thread per request to do dns lookups, and is in general very inefficient. On rhel, we use the stock libcurl which is linked against nss. nss keeps memory around until it is unloaded, which in rgw happens only when rgw is shut down. There are various ways to reduce this, but not eliminate it. Also nss is not terribly efficient about its memory allocations.

Openssl is the default choice for libcurl in ubuntu, and also in recent versions of fedora. Openssl is significantly better than nss in terms of connection overhead, and it is way better than gnutls. libcurl+openssl performs nearly as well as plain http for rgw keystone traffic. Reusing CURL* structures is also a big win. Reusing CURL* structures avoids opening a new connection to keystone, which dramatically reduces load on keystone. Eyeballing it, looks like 1/4 the cpu load there.

I have an experiment build against ubuntu for this, which so far in my testing, has been much better. I'll make a PR with this available shortly.

I've made a PR against jewel: https://github.com/ceph/ceph/pull/20623
I will have another PR against master later tonight.

This gets CURL* reuse on all platforms, and it gets libcurl + openssl on debian/ubuntu.

Another PR that may also matter,
https://github.com/ceph/ceph/pull/20390
contains openssl locking change. I don't have that and didn't see a problem, but I may not have pushed it hard enough, or it may matter in other environments.

I've made a PR for master with the same changes for CURL* and openssl/ubuntu. https://github.com/ceph/ceph/pull/20635