Project

General

Profile

Actions
Copy link

Bug #49955

closed

upgrade: rgw multisite failures

Added by Sage Weil about 3 years ago. Updated over 2 years ago.

Status:
Won't Fix
Priority:
High
Assignee:
Casey Bodley
Target version:
-
% Done:

0%

Source:
Tags:
multisite
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
42869
Crash signature (v1):
Crash signature (v2):

Description

2021-03-24T12:32:50.156 INFO:tasks.rgw_multisite_tests:======================================================================
2021-03-24T12:32:50.157 INFO:tasks.rgw_multisite_tests:FAIL: test notification of deletion markers
2021-03-24T12:32:50.157 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2021-03-24T12:32:50.158 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2021-03-24T12:32:50.158 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2021-03-24T12:32:50.158 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2021-03-24T12:32:50.159 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 2764, in test_ps_versioned_deletion
2021-03-24T12:32:50.159 INFO:tasks.rgw_multisite_tests:    master_zone, ps_zone = init_env()
2021-03-24T12:32:50.159 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 573, in init_env
2021-03-24T12:32:50.160 INFO:tasks.rgw_multisite_tests:    zonegroup_meta_checkpoint(zonegroup)
2021-03-24T12:32:50.160 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 199, in zonegroup_meta_checkpoint
2021-03-24T12:32:50.160 INFO:tasks.rgw_multisite_tests:    zone_meta_checkpoint(zone, meta_master_zone, master_status)
2021-03-24T12:32:50.161 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 188, in zone_meta_checkpoint
2021-03-24T12:32:50.161 INFO:tasks.rgw_multisite_tests:    assert False, 'failed meta checkpoint for zone=%s' % zone.name
2021-03-24T12:32:50.161 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2
2021-03-24T12:32:50.162 INFO:tasks.rgw_multisite_tests:
2021-03-24T12:32:50.162 INFO:tasks.rgw_multisite_tests:======================================================================
2021-03-24T12:32:50.163 INFO:tasks.rgw_multisite_tests:FAIL: test notification status upon bucket deletion
2021-03-24T12:32:50.163 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2021-03-24T12:32:50.163 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2021-03-24T12:32:50.164 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2021-03-24T12:32:50.164 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2021-03-24T12:32:50.164 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 3470, in test_ps_delete_bucket
2021-03-24T12:32:50.165 INFO:tasks.rgw_multisite_tests:    master_zone, ps_zone = init_env()
2021-03-24T12:32:50.165 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 573, in init_env
2021-03-24T12:32:50.165 INFO:tasks.rgw_multisite_tests:    zonegroup_meta_checkpoint(zonegroup)
2021-03-24T12:32:50.166 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 199, in zonegroup_meta_checkpoint
2021-03-24T12:32:50.166 INFO:tasks.rgw_multisite_tests:    zone_meta_checkpoint(zone, meta_master_zone, master_status)
2021-03-24T12:32:50.166 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 188, in zone_meta_checkpoint
2021-03-24T12:32:50.167 INFO:tasks.rgw_multisite_tests:    assert False, 'failed meta checkpoint for zone=%s' % zone.name
2021-03-24T12:32:50.167 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2
2021-03-24T12:32:50.167 INFO:tasks.rgw_multisite_tests:
2021-03-24T12:32:50.168 INFO:tasks.rgw_multisite_tests:======================================================================
2021-03-24T12:32:50.168 INFO:tasks.rgw_multisite_tests:FAIL: test creating a subscription when no topic info exists
2021-03-24T12:32:50.168 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2021-03-24T12:32:50.169 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2021-03-24T12:32:50.169 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2021-03-24T12:32:50.169 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2021-03-24T12:32:50.170 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 3539, in test_ps_missing_topic
2021-03-24T12:32:50.170 INFO:tasks.rgw_multisite_tests:    master_zone, ps_zone = init_env()
2021-03-24T12:32:50.170 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 573, in init_env
2021-03-24T12:32:50.171 INFO:tasks.rgw_multisite_tests:    zonegroup_meta_checkpoint(zonegroup)
2021-03-24T12:32:50.171 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 199, in zonegroup_meta_checkpoint
2021-03-24T12:32:50.171 INFO:tasks.rgw_multisite_tests:    zone_meta_checkpoint(zone, meta_master_zone, master_status)
2021-03-24T12:32:50.172 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 188, in zone_meta_checkpoint
2021-03-24T12:32:50.172 INFO:tasks.rgw_multisite_tests:    assert False, 'failed meta checkpoint for zone=%s' % zone.name
2021-03-24T12:32:50.173 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2
2021-03-24T12:32:50.173 INFO:tasks.rgw_multisite_tests:
2021-03-24T12:32:50.174 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------

/a/sage-2021-03-24_06:13:24-upgrade:octopus-x-wip-sage-testing-2021-03-23-2309-distro-basic-smithi/5993430
Actions
Copy link
 #1

Updated by Sage Weil about 3 years ago

/a/sage-2021-03-23_02:00:05-upgrade:octopus-x-wip-sage-testing-2021-03-22-1729-distro-basic-gibba/5989341
/a/sage-2021-03-23_02:00:05-upgrade:octopus-x-wip-sage-testing-2021-03-22-1729-distro-basic-gibba/5989356

Actions
Copy link
 #2

Updated by Casey Bodley about 3 years ago

from http://qa-proxy.ceph.com/teuthology/sage-2021-03-24_06:13:24-upgrade:octopus-x-wip-sage-testing-2021-03-23-2309-distro-basic-smithi/5993430/teuthology.log:

2021-03-24T12:16:57.992 INFO:tasks.rgw.c2.client.0.smithi166.stdout:*** Caught signal (Segmentation fault) **
2021-03-24T12:16:57.992 INFO:tasks.rgw.c2.client.0.smithi166.stdout: in thread 7f5e85ff3700 thread_name:rados_async
2021-03-24T12:16:57.994 INFO:tasks.rgw.c2.client.0.smithi166.stdout: ceph version 15.2.10-72-g13f4488c (13f4488c965bc9695a78d13528fefbd589d3015b) octopus (stable)
2021-03-24T12:16:57.994 INFO:tasks.rgw.c2.client.0.smithi166.stdout: 1: (()+0x3f040) [0x7f5ec6be5040]
2021-03-24T12:16:57.995 INFO:tasks.rgw.c2.client.0.smithi166.stdout: 2: (rgw::putobj::ETagVerifier_MPU::process(ceph::buffer::v15_2_0::list&&, unsigned long)+0x3f) [0x7f5ec794090f]
2021-03-24T12:16:57.995 INFO:tasks.rgw.c2.client.0.smithi166.stdout: 3: (RGWRados::fetch_remote_obj(RGWObjectCtx&, rgw_user const&, req_info*, rgw_zone_id const&, rgw_obj const&, rgw_obj const&, RGWBucketInfo const&, RGWBucketInfo const*, std::optional<rgw_placement_rule>, std::chrono::time_point<ceph::time_detail::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::time_detail::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::time_detail::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<ceph::time_detail::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const*, bool, char const*, char const*, RGWRados::AttrsMod, bool, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, RGWObjCategory, std::optional<unsigned long>, std::chrono::time_point<ceph::time_detail::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, void (*)(long, void*), void*, DoutPrefixProvider const*, RGWFetchObjFilter*, rgw_zone_set*, std::optional<unsigned long>*)+0xd05) [0x7f5ec779d785]
2021-03-24T12:16:57.995 INFO:tasks.rgw.c2.client.0.smithi166.stdout: 4: (RGWAsyncFetchRemoteObj::_send_request()+0x3d6) [0x7f5ec76e78c6]
2021-03-24T12:16:57.996 INFO:tasks.rgw.c2.client.0.smithi166.stdout: 5: (RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f5ec76e16c0]
2021-03-24T12:16:57.996 INFO:tasks.rgw.c2.client.0.smithi166.stdout: 6: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f5ec76e9c0d]
2021-03-24T12:16:57.996 INFO:tasks.rgw.c2.client.0.smithi166.stdout: 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x9fa) [0x7f5ebd57c9ea]
2021-03-24T12:16:57.996 INFO:tasks.rgw.c2.client.0.smithi166.stdout: 8: (ThreadPool::WorkThread::entry()+0x11) [0x7f5ebd57d8d1]
2021-03-24T12:16:57.996 INFO:tasks.rgw.c2.client.0.smithi166.stdout: 9: (()+0x76db) [0x7f5ebc5376db]
2021-03-24T12:16:57.997 INFO:tasks.rgw.c2.client.0.smithi166.stdout: 10: (clone()+0x3f) [0x7f5ec6cc771f]

Actions
Copy link
 #3

Updated by Casey Bodley about 3 years ago

  • Assignee set to Casey Bodley
  • Backport set to pacific octopus nautilus
Actions
Copy link
 #4

Updated by Casey Bodley almost 3 years ago

  • Assignee deleted (Casey Bodley)
  • Priority changed from Immediate to High
  • Backport deleted (pacific octopus nautilus)

i verified that the ubuntu crashes went away in https://pulpito.ceph.com/cbodley-2021-05-13_16:10:36-rgw:multisite-upgrade-master-distro-basic-smithi/

we're still left with lots of multisite test failures

Actions
Copy link
 #5

Updated by Casey Bodley over 2 years ago

  • Assignee set to Casey Bodley
Actions
Copy link
 #6

Updated by Casey Bodley over 2 years ago

  • Tags set to multisite
  • Pull request ID set to 42869
Actions
Copy link
 #7

Updated by Casey Bodley over 2 years ago

  • Status changed from New to Fix Under Review
Actions
Copy link
 #8

Updated by Casey Bodley over 2 years ago

  • Status changed from Fix Under Review to Won't Fix

moved out of the upgrade suite, not a real fix

Actions
Copy link

Also available in: Atom PDF