Project

General

Profile

Feature #23162

Curl+OpenSSL support in RGW

Added by Vikhyat Umrao 11 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
02/27/2018
Due date:
% Done:

0%

Source:
Support
Tags:
Backport:
jewel,luminous
Reviewed:
Affected Versions:
Pull request ID:


Related issues

Copied to rgw - Backport #23239: luminous: Curl+OpenSSL support in RGW Resolved
Copied to rgw - Backport #23240: jewel: Curl+OpenSSL support in RGW Resolved

History

#1 Updated by Marcus Watts 11 months ago

Well, this needs a description, so here goes.

When rgw makes outbound http requests, it uses libcurl. For keystone, this results in setting up a CURL* structure, using it once, then tearing it down. A direct consequence of this, on all platforms, is setting up and tearing down an SSL connection to keystone each and every keystone validation call. This is somewhat expensive for swift (depending on how efficiently the client reuses tokens), and very expensive for s3 auth (because each aws signature requires validating the hash in keystone).

Our current platform specific distribution choices for libcurl make this worse. On ubuntu, we use libcurl-gnutls. gnutls turns out to be very innefficient about memory usage, starts up and tears down a thread per request to do dns lookups, and is in general very inefficient. On rhel, we use the stock libcurl which is linked against nss. nss keeps memory around until it is unloaded, which in rgw happens only when rgw is shut down. There are various ways to reduce this, but not eliminate it. Also nss is not terribly efficient about its memory allocations.

Openssl is the default choice for libcurl in ubuntu, and also in recent versions of fedora. Openssl is significantly better than nss in terms of connection overhead, and it is way better than gnutls. libcurl+openssl performs nearly as well as plain http for rgw keystone traffic. Reusing CURL* structures is also a big win. Reusing CURL* structures avoids opening a new connection to keystone, which dramatically reduces load on keystone. Eyeballing it, looks like 1/4 the cpu load there.

I have an experiment build against ubuntu for this, which so far in my testing, has been much better. I'll make a PR with this available shortly.

#2 Updated by Marcus Watts 11 months ago

I've made a PR against jewel: https://github.com/ceph/ceph/pull/20623
I will have another PR against master later tonight.

This gets CURL* reuse on all platforms, and it gets libcurl + openssl on debian/ubuntu.

Another PR that may also matter,
https://github.com/ceph/ceph/pull/20390
contains openssl locking change. I don't have that and didn't see a problem, but I may not have pushed it hard enough, or it may matter in other environments.

#3 Updated by Marcus Watts 11 months ago

I've made a PR for master with the same changes for CURL* and openssl/ubuntu. https://github.com/ceph/ceph/pull/20635

#4 Updated by Vikhyat Umrao 11 months ago

  • Assignee set to Marcus Watts
  • Backport set to jewel,luminous

Thanks Marcus for detail explanation.

#5 Updated by Nathan Cutler 11 months ago

  • Status changed from New to Pending Backport

#6 Updated by Nathan Cutler 11 months ago

#7 Updated by Nathan Cutler 11 months ago

#8 Updated by Nathan Cutler 10 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF