Project

General

Profile

Actions

Bug #64156

open

rgw/multisite: radosgw crashes during add_datalog_entry()

Added by Shilpa MJ 3 months ago. Updated 3 months ago.

Status:
New
Priority:
Urgent
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
multisite
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

From a recently rebased build on Jan 23:

http://qa-proxy.ceph.com/teuthology/smanjara-2024-01-24_16:24:23-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/7531067/teuthology.log

2024-01-24T17:21:00.295 INFO:teuthology.orchestra.run.smithi144.stderr:*** Caught signal (Segmentation fault) *
2024-01-24T17:21:00.295 INFO:teuthology.orchestra.run.smithi144.stderr: in thread 7f82837e6640 thread_name:rados_async
2024-01-24T17:21:00.295 INFO:teuthology.orchestra.run.smithi144.stderr: ceph version 19.0.0-839-ge877ac4d (e877ac4de7bff7085e0ac032b0409b4b9faface4) squi
d (dev)
2024-01-24T17:21:00.295 INFO:teuthology.orchestra.run.smithi144.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f82c1454db0]
2024-01-24T17:21:00.295 INFO:teuthology.orchestra.run.smithi144.stderr: 2: /lib64/librados.so.2(+0x9edce) [0x7f82c4510dce]
2024-01-24T17:21:00.295 INFO:teuthology.orchestra.run.smithi144.stderr: 3: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string<char, std::char_
traits<char>, std::allocator<char> > const&, librados::v14_2_0::ObjectWriteOperation
, int)+0x76) [0x7f82c44f1f66]
2024-01-24T17:21:00.295 INFO:teuthology.orchestra.run.smithi144.stderr: 4: (rgw_rados_operate(DoutPrefixProvider const*, librados::v14_2_0::IoCtx&, std::
_cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::ObjectWriteOperation*, optional_yield, int)+0x553)
[0x55ce39c1f5f3]
2024-01-24T17:21:00.296 INFO:teuthology.orchestra.run.smithi144.stderr: 5: (rgw::cls::fifo::FIFO::push_entries(DoutPrefixProvider const*, std::deque<ceph
::buffer::v15_2_0::list, std::allocator<ceph::buffer::v15_2_0::list> > const&, unsigned long, optional_yield)+0x449) [0x55ce39d5edf9]
2024-01-24T17:21:00.296 INFO:teuthology.orchestra.run.smithi144.stderr: 6: (rgw::cls::fifo::FIFO::push(DoutPrefixProvider const*, std::vector<ceph::buffe
r::v15_2_0::list, std::allocator<ceph::buffer::v15_2_0::list> > const&, optional_yield)+0xd93) [0x55ce39d616b3]
2024-01-24T17:21:00.296 INFO:teuthology.orchestra.run.smithi144.stderr: 7: (rgw::cls::fifo::FIFO::push(DoutPrefixProvider const*, ceph::buffer::v15_2_0::
list const&, optional_yield)+0x13c) [0x55ce39d6209c]
2024-01-24T17:21:00.296 INFO:teuthology.orchestra.run.smithi144.stderr: 8: radosgw-admin(+0x68914b) [0x55ce39a8114b]
2024-01-24T17:21:00.296 INFO:teuthology.orchestra.run.smithi144.stderr: 9: (RGWDataChangesLog::add_entry(DoutPrefixProvider const*, RGWBucketInfo const&,
rgw::bucket_log_layout_generation const&, int, optional_yield)+0x8b7) [0x55ce39a8ab47]
2024-01-24T17:21:00.296 INFO:teuthology.orchestra.run.smithi144.stderr: 10: (add_datalog_entry(DoutPrefixProvider const*, RGWDataChangesLog*, RGWBucketIn
fo const&, unsigned int, optional_yield)+0x49) [0x55ce39b22ba9]
2024-01-24T17:21:00.296 INFO:teuthology.orchestra.run.smithi144.stderr: 11: (RGWRados::Bucket::UpdateIndex::complete(DoutPrefixProvider const*, long, uns
igned long, unsigned long, unsigned long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >
const&, std::
_cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*, RGWO
bjCategory, std::__cxx11::list<rgw_obj_index_key, std::allocator<rgw_obj_index_key> >, optional_yield, std::__cxx11::basic_string<char, std::char_traits
<char>, std::allocator<char> > const
, bool, bool)+0x3bf) [0x55ce39b5237f]
2024-01-24T17:21:00.297 INFO:teuthology.orchestra.run.smithi144.stderr: 12: (RGWRados::Object::Write::_do_write_meta(unsigned long, unsigned long, std::m
ap<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<cha
r, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<cha
r> > const, ceph::buffer::v15_2_0::list> > >&, bool, bool, void*, req_context const&, bool)+0xd79) [0x55ce39b48579]
2024-01-24T17:21:00.297 INFO:teuthology.orchestra.run.smithi144.stderr: 13: (RGWRados::Object::Write::write_meta(unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, req_context const&, bool)+0x55b) [0x55ce39b4960b]
2024-01-24T17:21:00.297 INFO:teuthology.orchestra.run.smithi144.stderr: 14: (rgw::putobj::AtomicObjectProcessor::complete(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, char const, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rgw_zone_set*, bool*, req_context const&, unsigned int)+0x2f7) [0x55ce39b24967]
2024-01-24T17:21:00.297 INFO:teuthology.orchestra.run.smithi144.stderr: 15: (RGWRados::fetch_remote_obj(RGWObjectCtx&, rgw_user const&, req_info*, rgw_zone_id const&, rgw_obj const&, rgw_obj const&, RGWBucketInfo&, RGWBucketInfo*, std::optional<rgw_placement_rule>, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const*, bool, char const*, char const*, RGWRados::AttrsMod, bool, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, RGWObjCategory, std::optional<unsigned long>, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, void ()(long, void), void*, req_context const&, RGWFetchObjFilter*, bool, rgw_obj const&, rgw_zone_set_entry const&, rgw_zone_set*, std::optional<unsigned long>*)+0x1875) [0x55ce39b437a5]
2024-01-24T17:21:00.297 INFO:teuthology.orchestra.run.smithi144.stderr: 16: (RGWAsyncFetchRemoteObj::_send_request(DoutPrefixProvider const*)+0x622) [0x55ce39a74f52]

Actions #1

Updated by Christian Rohmann 3 months ago

Could this be related to https://tracker.ceph.com/issues/63373 maybe?

Actions #2

Updated by Casey Bodley 3 months ago

  • Priority changed from High to Urgent
  • Tags set to multisite
Actions #3

Updated by Shilpa MJ 3 months ago

so far this crash has been reproducible on teuthology runs, particularly while running test_zap_init_bucket_sync_run() after the secondary radosgw restart.

Actions

Also available in: Atom PDF