Bug #50103: multisite: crash in RGWRESTStreamRWRequest::do_send_prepare() with empty url - rgw - Ceph

Actions

Copy link

Bug #50103

closed

multisite: crash in RGWRESTStreamRWRequest::do_send_prepare() with empty url

Added by Casey Bodley about 3 years ago. Updated over 1 year ago.

Status:

Resolved

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Tags:

multisite backport_processed

Backport:

pacific octopus nautilus

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v15.2.13, Ceph - v15.2.5, Ceph - v15.2.7, Ceph - v15.2.8, Ceph - v15.2.9

ceph-qa-suite:

Pull request ID:

40563

Crash signature (v1):

33117cad3ca92b33597050a6d13cef4c6e3110898b509a82211a859ed6333821
39a3d8cc71898dd0f486ea1cace46babf5c4cdbf18d64adaf1505e7d447ee44d

Crash signature (v2):

28fe0b5c7ff6ce6e891150246c8c6ac1bdba56bbddfd36b17123e7fe23ffd493

Description

ceph version 17.0.0-2661-g9561b4dde37 (9561b4dde37fe1ecdd471151bdca5997536feb53) quincy (dev)
 1: /root/projects/another-ceph/build/lib/libradosgw.so.2(+0x2799040) [0x7f76c00ff040]                                                                                                                                                        
 2: /lib64/libpthread.so.0(+0x141e0) [0x7f76ba9781e0]
 3: gsignal()
 4: abort()
 5: (std::exception::exception()+0) [0x5615b35e1ecc]
 6: (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::operator[](unsigned long)+0x46) [0x5615b35ee600]
 7: (RGWRESTStreamRWRequest::do_send_prepare(RGWAccessKey*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x83) [0x7f76bfc99d09]
 8: (RGWRESTStreamRWRequest::send_prepare(RGWAccessKey*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x6f) [0x7f76bfc99c3f]
 9: (RGWRESTStreamRWRequest::send_request(RGWAccessKey*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWHTTPManager*, ceph::buffer::v15_2_0::list*)+0x3f) [0x7f76bfc9a339]
 10: (RGWRESTReadResource::aio_read()+0x84) [0x7f76bfca162c]
 11: (RGWReadRawRESTResourceCR::send_request()+0xc7) [0x7f76bf99b4b5]
 12: (RGWSimpleCoroutine::state_send_request()+0x23) [0x7f76bfb3f1e5]
 13: (RGWSimpleCoroutine::operate()+0x21e) [0x7f76bfb3ee9a]
 14: (RGWCoroutine::operate_wrapper()+0x23) [0x7f76bf62969d]
 15: (RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x148) [0x7f76bfb399f6]
 16: (RGWCoroutinesManager::run(std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x2ff) [0x7f76bfb3bcbf]
 17: (RGWSyncLogTrimThread::process()+0x292) [0x7f76bfc4e472]
 18: (RGWRadosThread::Worker::entry()+0x91) [0x7f76bfbf9061]
 19: (Thread::entry_wrapper()+0x83) [0x7f76bc2b3dd9]
 20: (Thread::_entry_func(void*)+0x18) [0x7f76bc2b3d4c]
 21: /lib64/libpthread.so.0(+0x93f9) [0x7f76ba96d3f9]
 22: clone()

when the given url is empty, this code tries to access new_url[-1]:

int RGWRESTStreamRWRequest::do_send_prepare(RGWAccessKey *key, map<string, string>& extra_headers, const string& resource,
                                         bufferlist *send_data)
{
  string new_url = url;
  if (new_url[new_url.size() - 1] != '/')
    new_url.append("/");

Related issues 4 (0 open — 4 closed)

Actions

Copy link

Updated by lei cao about 3 years ago

https://github.com/ceph/ceph/pull/40563

Actions

Copy link

Updated by Casey Bodley about 3 years ago

Status changed from New to In Progress
Pull request ID set to 40563

Actions

Copy link

Updated by Casey Bodley about 3 years ago

still needs investigation to find where we're sending to an empty url

Actions

Copy link

Updated by J. Eric Ivancich almost 3 years ago

Status changed from In Progress to Pending Backport

Actions

Copy link

Updated by Backport Bot almost 3 years ago

Copied to Backport #50727: octopus: multisite: crash in RGWRESTStreamRWRequest::do_send_prepare() with empty url added

Actions

Copy link

Updated by Backport Bot almost 3 years ago

Copied to Backport #50728: pacific: multisite: crash in RGWRESTStreamRWRequest::do_send_prepare() with empty url added

Actions

Copy link

Updated by Backport Bot almost 3 years ago

Copied to Backport #50729: nautilus: multisite: crash in RGWRESTStreamRWRequest::do_send_prepare() with empty url added

Actions

Copy link

Updated by Casey Bodley over 2 years ago

Has duplicate Bug #51910: crash: RGWRESTStreamRWRequest::do_send_prepare added

Actions

Copy link

Updated by Telemetry Bot over 2 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)
Affected Versions v15.2.13, v15.2.5, v15.2.7, v15.2.8, v15.2.9 added

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=33117cad3ca92b33597050a6d13cef4c6e3110898b509a82211a859ed6333821

Sanitized backtrace:

    RGWRESTStreamRWRequest::do_send_prepare(RGWAccessKey*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)
    RGWRESTStreamRWRequest::send_prepare(RGWAccessKey*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)
    RGWRESTStreamRWRequest::send_request(RGWAccessKey*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWHTTPManager*, ceph::buffer::v15_2_0::list*)
    RGWRESTReadResource::aio_read()
    RGWReadRawRESTResourceCR::send_request()
    RGWSimpleCoroutine::state_send_request()
    RGWSimpleCoroutine::operate()
    RGWCoroutinesStack::operate(RGWCoroutinesEnv*)
    RGWCoroutinesManager::run(std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)
    RGWSyncLogTrimThread::process()
    RGWRadosThread::Worker::entry()
    clone()

Crash dump sample:

{
    "backtrace": [
        "(()+0x12b20) [0x7f3cc9095b20]",
        "(gsignal()+0x10f) [0x7f3cc76da7ff]",
        "(abort()+0x127) [0x7f3cc76c4c35]",
        "(()+0x3eeea8) [0x7f3cd3cc6ea8]",
        "(RGWRESTStreamRWRequest::do_send_prepare(RGWAccessKey*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x6a5) [0x7f3cd4187865]",
        "(RGWRESTStreamRWRequest::send_prepare(RGWAccessKey*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x66) [0x7f3cd41879f6]",
        "(RGWRESTStreamRWRequest::send_request(RGWAccessKey*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWHTTPManager*, ceph::buffer::v15_2_0::list*)+0x18) [0x7f3cd4187a58]",
        "(RGWRESTReadResource::aio_read()+0x50) [0x7f3cd418fbd0]",
        "(RGWReadRawRESTResourceCR::send_request()+0x75) [0x7f3cd3fb6335]",
        "(RGWSimpleCoroutine::state_send_request()+0x13) [0x7f3cd4097c53]",
        "(RGWSimpleCoroutine::operate()+0xac) [0x7f3cd409d82c]",
        "(RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x67) [0x7f3cd409b117]",
        "(RGWCoroutinesManager::run(std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x271) [0x7f3cd409bf41]",
        "(RGWSyncLogTrimThread::process()+0x200) [0x7f3cd4165410]",
        "(RGWRadosThread::Worker::entry()+0x176) [0x7f3cd412db36]",
        "(()+0x814a) [0x7f3cc908b14a]",
        "(clone()+0x43) [0x7f3cc779ff23]" 
    ],
    "ceph_version": "15.2.8",
    "crash_id": "2021-07-26T07:58:52.468750Z_cd178b46-2239-4d58-9a04-c9d691a55858",
    "entity_name": "client.c5e62e84d2a35aaa270e0da36b42d29c3397929b",
    "os_id": "centos",
    "os_name": "CentOS Linux",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "radosgw",
    "stack_sig": "39a3d8cc71898dd0f486ea1cace46babf5c4cdbf18d64adaf1505e7d447ee44d",
    "timestamp": "2021-07-26T07:58:52.468750Z",
    "utsname_machine": "x86_64",
    "utsname_release": "4.18.0-240.10.1.el8_3.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP Mon Jan 18 17:05:51 UTC 2021" 
}

Actions

Copy link

#10