Project

General

Profile

Bug #23375

Memory leak in RGW when libcurl is configured with --with-nss and performing https requests to keystone

Added by Mark Kogan about 6 years ago. Updated about 6 years ago.

Status:
In Progress
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Support
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When libcurl is configured with --with-nss
There is a memory leak in the PK11_CreateGenericObject() function
when keystone users authority authentication performed using ssl
ceph.conf:
rgw_s3_auth_use_keystone = true
rgw_keystone_verify_ssl = true

   ¦ MB
 8.480^                                                                       #
   ¦ ¦|                                                                   @@@@#
   ¦ ¦|                                                                @@@@@@@#
   ¦ ¦|                                                           :::::@@@@@@@#
   ¦ ¦|                                                      ::@@:: :: @@@@@@@#
   ¦ ¦|                                                  :::@: @ :: :: @@@@@@@#
   ¦ ¦|                                             @@@:::: @: @ :: :: @@@@@@@#
   ¦ ¦|                                           ::@@ : :: @: @ :: :: @@@@@@@#
   ¦ ¦|                                   @@@::@::::@@ : :: @: @ :: :: @@@@@@@#
   ¦ ¦|                               @@::@ @::@: ::@@ : :: @: @ :: :: @@@@@@@#
   ¦ ¦|                            :::@@::@ @::@: ::@@ : :: @: @ :: :: @@@@@@@#
   ¦ ¦|                       @:::::: @@::@ @::@: ::@@ : :: @: @ :: :: @@@@@@@#
   ¦ ¦|                   @@::@: : :: @@::@ @::@: ::@@ : :: @: @ :: :: @@@@@@@#
   ¦ ¦|               ::::@@: @: : :: @@::@ @::@: ::@@ : :: @: @ :: :: @@@@@@@#
   ¦ ¦|           :@::: ::@@: @: : :: @@::@ @::@: ::@@ : :: @: @ :: :: @@@@@@@#
   ¦ ¦|    @  :::::@::: ::@@: @: : :: @@::@ @::@: ::@@ : :: @: @ :: :: @@@@@@@#
   ¦ ¦| :::@::::: :@::: ::@@: @: : :: @@::@ @::@: ::@@ : :: @: @ :: :: @@@@@@@#
   ¦ ¦| : :@: ::: :@::: ::@@: @: : :: @@::@ @::@: ::@@ : :: @: @ :: :: @@@@@@@#
   ¦ ¦| : :@: ::: :@::: ::@@: @: : :: @@::@ @::@: ::@@ : :: @: @ :: :: @@@@@@@#
   ¦ ¦| : :@: ::: :@::: ::@@: @: : :: @@::@ @::@: ::@@ : :: @: @ :: :: @@@@@@@#
   ¦0 +----------------------------------------------------------------------->Gi
   ¦ ¦0                                                                   11.32

 --------------------------------------------------------------------------------
   n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
 --------------------------------------------------------------------------------
  66 12,126,113,059        8,719,784        7,271,908     1,447,876            0
  67 12,155,945,056        8,891,544        7,436,525     1,455,019            0
 83.64% (7,436,525B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 ->73.74% (6,556,871B) 0xB05B6F8: ???
 | ->28.52% (2,535,936B) 0xB04C362: ???
 | | ->28.52% (2,535,936B) 0xB0505E7: ???
 | |   ->28.52% (2,535,936B) 0xB0554B9: ???
 | |     ->28.52% (2,535,936B) 0x5D71860: PK11_CreateNewObject (pk11obj.c:405)
 | |       ->28.52% (2,535,936B) 0x5D73721: PK11_CreateGenericObject (pk11obj.c:1638)
 | |         ->28.52% (2,535,936B) 0x4E3766B: nss_create_object.isra.10 (nss.c:401)
 | |           ->28.52% (2,535,936B) 0x4E37727: nss_load_cert (nss.c:438)
 | |             ->28.52% (2,535,936B) 0x4E702E5: nss_connect_common (nss.c:1200)
 | |               ->28.52% (2,535,936B) 0x4E6695C: Curl_ssl_connect_nonblocking (sslgen.c:229)
 | |                 ->28.52% (2,535,936B) 0x4E3DE4B: Curl_http_connect (http.c:1345)
 | |                   ->28.52% (2,535,936B) 0x4E4D613: Curl_protocol_connect (url.c:3297)
 | |                     ->28.52% (2,535,936B) 0x4E60CC9: multi_runsingle (multi.c:1159)
 | |                       ->28.52% (2,535,936B) 0x4E6121F: curl_multi_perform (multi.c:1757)
 | |                         ->28.52% (2,535,936B) 0x4E58521: curl_easy_perform (easy.c:480)
 | |                           ->28.52% (2,535,936B) 0x400B26: doit (curl1.c:36)
 | |                             ->28.52% (2,535,936B) 0x400BB2: process (curl1.c:54)
 | |                               ->28.52% (2,535,936B) 0x400BEA: main (curl1.c:62)
 | |

I found a way to free the memory accumulated in Curl/NSS
by calling the curl_global_cleanup() function.


Related issues

Related to rgw - Bug #37388: rgw: memory leak with multisite sync In Progress 11/26/2018

History

#1 Updated by Mark Kogan about 6 years ago

  • Description updated (diff)

#2 Updated by Mark Kogan about 6 years ago

  • Description updated (diff)

#3 Updated by Mark Kogan about 6 years ago

  • Subject changed from Memory leak in RGW when libcurl is configured with --with-nss to Memory leak in RGW when libcurl is configured with --with-nss and performing https requests to keystone

#4 Updated by Mark Kogan about 6 years ago

  • Description updated (diff)

#6 Updated by Casey Bodley over 5 years ago

  • Related to Bug #37388: rgw: memory leak with multisite sync added

Also available in: Atom PDF