Ceph » rgw

Built ceph from master and started rgw as stand-alone process with:¶

radosgw --rgw-backend-store=dbstore -i foo --debug-rgw 15 --no-mon-config --rgw-data ./rgw --run-dir ./rgw -d

Using s3cmd as client, issued the following commands:¶

1. s3cmd mb s3://test
2. s3cmd put myfile s3://test
3. s3cmd rm s3://test/myfile

The 3. command crashes rgw process and produces the following stack-trace:

std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::capacity(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > * const this) (/usr/include/c++/9/bits/basic_string.h:997)
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > * const this, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > & __str) (/usr/include/c++/9/bits/basic_string.tcc:260)
libradosgw.so.2!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::assign(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > & __str, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > * const this) (/usr/include/c++/9/bits/basic_string.h:1368)
libradosgw.so.2!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::operator=(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > & __str, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > * const this) (/usr/include/c++/9/bits/basic_string.h:700)
libradosgw.so.2!rgw_bucket::operator=(rgw_bucket * const this) (/workspaces/ceph/src/rgw/rgw_basic_types.h:289)
libradosgw.so.2!rgw_obj::operator=(rgw_obj * const this) (/workspaces/ceph/src/rgw/rgw_common.h:1799)
libradosgw.so.2!RGWObjState::operator=(RGWObjState * const this) (/workspaces/ceph/src/rgw/rgw_sal.h:104)
libradosgw.so.2!rgw::store::DB::Object::get_obj_state(rgw::store::DB::Object * const this, const DoutPrefixProvider * dpp, const RGWBucketInfo & bucket_info, const rgw_obj & obj, bool follow_olh, RGWObjState ** state) (/workspaces/ceph/src/rgw/store/dbstore/common/dbstore.cc:1229)
libradosgw.so.2!rgw::sal::DBObject::get_obj_state(rgw::sal::DBObject * const this, const DoutPrefixProvider * dpp, RGWObjState ** pstate, optional_yield y, bool follow_olh) (/workspaces/ceph/src/rgw/rgw_sal.h:1094)
libradosgw.so.2!RGWDeleteObj::execute(RGWDeleteObj * const this, optional_yield y) (/workspaces/ceph/src/rgw/rgw_op.cc:4932)
...

A possible explanation for the crash:¶

In the rgw_op.cc file, in the member-function RGWDeleteObj::execute, there is call to get_obj_state in order to retrieve a pointer pointing to an RGWObjState object.
Please note that this code is abstract in regarding of the actual be implementation.

void RGWDeleteObj::execute(optional_yield y)
{
...
    {
      RGWObjState* astate = nullptr;
...
      op_ret = s->object->get_obj_state(this, &astate, s->yield, true);
...
}

get_obj_state() is an abstract method that depends on the actual be implementation.
For dbstore this will lead to execute dbstore.cc DB::Object::get_obj_state function.

int DB::Object::get_obj_state(const DoutPrefixProvider *dpp,
                              const RGWBucketInfo& bucket_info, const rgw_obj& obj,
                              bool follow_olh, RGWObjState **state)
{
...

  DBOpParams params = {};
  RGWObjState* s;

...

  s = &params.op.obj.state;

...

  **state = *s;

...

}

The statement that is actually causing the crash is **state = *s, this because *state points to an invalid memory location.

Looking at the rados-be implementation I would say that *state will be assigned with an instance retrieved from an object's context (the object being deleted).