Project

General

Profile

Bug #22951

possible issue with ssl + libcurl

Added by Abhishek Lekshmanan over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
High
Target version:
-
Start date:
02/07/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
luminous, jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

Getting random segmentation faults when radosgw is trying to authenticate with keystone over ssl, while this issue is not reproducible consistently, the stacktrace looks like below. Libcurl mentions the need to set callbacks for openssl <= 1.02 at https://curl.haxx.se/libcurl/c/threadsafe.html and https://www.openssl.org/docs/man1.0.2/crypto/threads.html

2ea9db in raise (sig=11) at ../sysdeps/unix/sysv/linux/pt-raise.c:36
#1  0x0000557a5b5adb07 in reraise_fatal (signum=11)
    at /usr/src/debug/ceph-12.2.1/src/global/signal_handler.cc:74
#2  handle_fatal_signal (signum=11)
    at /usr/src/debug/ceph-12.2.1/src/global/signal_handler.cc:138
#3  <signal handler called>
#4  sha1_block_data_order () at sha1-x86_64.s:50
#5  0x7d1e023451022026 in ?? ()
#6  0x00000000a7a1f41c in ?? ()
#7  0xc542a82f7154296a in ?? ()
#8  0x402802d15aca6ccf in ?? ()
#9  0xfc542a82a5c402be in ?? ()
#10 0x1402802dad0fcbe1 in ?? ()
#11 0x2d140280f5c06a0a in ?? ()
#12 0x00000000cb4f6b10 in ?? ()
#13 0x00007f733153c0f8 in ?? ()
#14 0x00007f735f146148 in CRYPTO_malloc (num=1880103424,
    file=0x391682ad <error: Cannot access memory at address 0x391682ad>, line=271811570)
    at mem.c:346
#15 0x00007f735f14d58f in SHA1_Update (c=0x7f735f52ca3f <state+1023>, data_=<optimized out>,
    len=<optimized out>) at ../md32_common.h:350
#16 0x00007f735f1f952e in ssleay_rand_add (buf=0x7f733153c200, num=144, add=0) at md_rand.c:256
#17 0x00007f735f1f9cb1 in RAND_load_file (file=file@entry=0x7f735f2b79fb "/dev/urandom",
    bytes=bytes@entry=48) at randfile.c:177
#18 0x00007f735f1f9786 in ssleay_rand_bytes (buf=0x557a6205afc4 "", num=32, pseudo=0, lock=1)
    at md_rand.c:347
#19 0x00007f735f55cfc7 in ssl23_client_hello (s=0x557a64ed43c0) at s23_clnt.c:387
#20 ssl23_connect (s=0x557a64ed43c0) at s23_clnt.c:218
#21 0x00007f736ab1cc62 in ossl_connect_step2 (sockindex=0, conn=0x557a66340000)
    at vtls/openssl.c:1904
#22 ossl_connect_common (conn=conn@entry=0x557a66340000, sockindex=sockindex@entry=0,
    nonblocking=nonblocking@entry=true, done=done@entry=0x7f733153cad5) at vtls/openssl.c:2634
#23 0x00007f736ab1e5cd in Curl_ossl_connect_nonblocking (conn=conn@entry=0x557a66340000,
    sockindex=sockindex@entry=0, done=done@entry=0x7f733153cad5) at vtls/openssl.c:2670
#24 0x00007f736ab1ebd0 in Curl_ssl_connect_nonblocking (conn=conn@entry=0x557a66340000,
    sockindex=sockindex@entry=0, done=done@entry=0x7f733153cad5) at vtls/vtls.c:304
#25 0x00007f736aada27d in https_connecting (done=0x7f733153cad5, conn=0x557a66340000)
    at http.c:1383
#26 Curl_http_connect (conn=0x557a66340000, done=0x7f733153cad5) at http.c:1353
#27 0x00007f736aaea045 in Curl_protocol_connect (conn=0x557a66340000,
    protocol_done=protocol_done@entry=0x7f733153cad5) at url.c:3460
#28 0x00007f736aafd4de in multi_runsingle (multi=multi@entry=0x557a74f37380, now=...,
    data=data@entry=0x557a70290000) at multi.c:1166
#29 0x00007f736aafdd41 in curl_multi_perform (multi_handle=multi_handle@entry=0x557a74f37380,
    running_handles=running_handles@entry=0x7f733153cc24) at multi.c:1762
#30 0x00007f736aaf5067 in easy_transfer (multi=0x557a74f37380) at easy.c:709
#31 easy_perform (events=false, data=0x557a70290000) at easy.c:797
#32 curl_easy_perform (easy=0x557a70290000) at easy.c:816
#33 0x0000557a5b5f2346 in RGWHTTPClient::process (this=this@entry=0x7f733153d380,
    method=method@entry=0x557a5b8b63ff "POST", url=<optimized out>)
    at /usr/src/debug/ceph-12.2.1/src/rgw/rgw_http_client.cc:286
#34 0x0000557a5b5b4740 in rgw::auth::keystone::EC2Engine::get_from_keystone (
    this=this@entry=0x557a5d933bc8, access_key_id=..., string_to_sign=..., signature=...)
    at /usr/src/debug/ceph-12.2.1/src/rgw/rgw_auth_keystone.cc:350
#35 0x0000557a5b5b502e in rgw::auth::keystone::EC2Engine::authenticate(boost::basic_string_view<char, std::char_traits<char> > const&, boost::basic_string_view<char, std::char_traits<char> > const&, std::string const&, std::function<basic_sstring<char, unsigned short, (unsigned short)65> (CephContext*, std::string const&, std::string const&)> const&, std::function<std::shared_ptr<rgw::auth::Completer> (boost::optional<std::string> const&)> const&, req_state const*) const (
    this=0x557a5d933bc8, access_key_id=..., signature=..., string_to_sign=...,
    completer_factory=..., s=0x7f733153eba0)
    at /usr/src/debug/ceph-12.2.1/src/rgw/rgw_auth_keystone.cc:442
#36 0x0000557a5b744199 in rgw::auth::s3::AWSEngine::authenticate (this=0x557a5d933bc8,
    s=0x7f733153eba0)
    at /usr/src/debug/ceph-12.2.1/src/rgw/rgw_rest_s3.cc:4042
#37 0x0000557a5b5b0478 in rgw::auth::Strategy::authenticate (this=0x557a5d933b90,
    s=0x7f733153eba0)
    at /usr/src/debug/ceph-12.2.1/src/rgw/rgw_auth.cc:209
#38 0x0000557a5b5b0478 in rgw::auth::Strategy::authenticate (this=0x557a5d933b38,
    s=0x7f733153eba0)
    at /usr/src/debug/ceph-12.2.1/src/rgw/rgw_auth.cc:209
#39 0x0000557a5b5b0478 in rgw::auth::Strategy::authenticate (this=0x557a5d933b18,
    s=s@entry=0x7f733153eba0)
    at /usr/src/debug/ceph-12.2.1/src/rgw/rgw_auth.cc:209
#40 0x0000557a5b5b0f85 in rgw::auth::Strategy::apply (auth_strategy=...,
    s=s@entry=0x7f733153eba0)
    at /usr/src/debug/ceph-12.2.1/src/rgw/rgw_auth.cc:260
#41 0x0000557a5b743c59 in RGW_Auth_S3::authorize (store=<optimized out>, auth_registry=...,
    s=0x7f733153eba0)
    at /usr/src/debug/ceph-12.2.1/src/rgw/rgw_rest_s3.cc:3358
#42 0x0000557a5b682886 in process_request (store=0x557a5db02000, rest=0x7ffece6eec90,
    req=req@entry=0x7f733153f220, frontend_prefix=..., auth_registry=...,
    client_io=client_io@entry=0x7f733153f250, olog=0x0)
    at /usr/src/debug/ceph-12.2.1/src/rgw/rgw_process.cc:175
#43 0x0000557a5b569dc6 in RGWCivetWebFrontend::process (this=0x557a5d909040,
    conn=<optimized out>)
    at /usr/src/debug/ceph-12.2.1/src/rgw/rgw_civetweb_frontend.cc:35
#44 0x0000557a5b59edff in handle_request (conn=conn@entry=0x557a5f7ff000)
    at /usr/src/debug/ceph-12.2.1/src/civetweb/src/civetweb.c:9890
#45 0x0000557a5b5a07ab in process_new_connection (conn=<optimized out>)
    at /usr/src/debug/ceph-12.2.1/src/civetweb/src/civetweb.c:12327
#46 worker_thread_run (thread_func_param=0x557a5da42800)
    at /usr/src/debug/ceph-12.2.1/src/civetweb/src/civetweb.c:12504


Related issues

Copied to rgw - Backport #23221: luminous: possible issue with ssl + libcurl Resolved
Copied to rgw - Backport #23243: jewel: possible issue with ssl + libcurl Resolved

History

#1 Updated by Abhishek Lekshmanan over 1 year ago

I was able to reproduce this with libopenssl 1.0.2j-16.1 & curl 7.37 which are shipped with opensuse 42.3. There were 2 threads trying to read rand
  • 1 Thread 0x7f29b3074700 (LWP 2106) RAND_add (buf=0x7f29b306ae40, num=48, entropy=48) at rand_lib.c:154...
    115 Thread 0x7f29a9060700 (LWP 2126) RAND_bytes (buf=0x55cb36ae84f0 "", num=16) at rand_lib.c:161

#3 Updated by Abhishek Lekshmanan over 1 year ago

  • Status changed from New to In Progress

#4 Updated by Abhishek Lekshmanan over 1 year ago

  • Assignee set to Abhishek Lekshmanan

#5 Updated by Abhishek Lekshmanan over 1 year ago

  • Priority changed from Normal to High

#6 Updated by Abhishek Lekshmanan over 1 year ago

  • Backport set to luminous, jewel

#7 Updated by Abhishek Lekshmanan over 1 year ago

  • Copied to Backport #23221: luminous: possible issue with ssl + libcurl added

#8 Updated by Yehuda Sadeh over 1 year ago

  • Status changed from In Progress to Need Review

#9 Updated by Abhishek Lekshmanan over 1 year ago

  • Copied to Backport #23243: jewel: possible issue with ssl + libcurl added

#10 Updated by Abhishek Lekshmanan over 1 year ago

  • Status changed from Need Review to Pending Backport

#13 Updated by Nathan Cutler about 1 year ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF