Bug #56007: "AssertionError: rgw multisite test failures" in upgrade:octopus-x-quincy-release - Ceph - Ceph

Actions

Copy link

Bug #56007

closed

"AssertionError: rgw multisite test failures" in upgrade:octopus-x-quincy-release

Added by Yuri Weinstein almost 2 years ago. Updated almost 2 years ago.

Status:

Duplicate

Priority:

Urgent

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

v17.1.0

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

This is for 17.2.1
Run: http://pulpito.front.sepia.ceph.com/yuriw-2022-06-11_18:42:53-upgrade:octopus-x-quincy-release-distro-default-smithi/
Jobs: '6874762', '6874748'
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-06-11_18:42:53-upgrade:octopus-x-quincy-release-distro-default-smithi/6874748/teuthology.log

2022-06-11T21:00:16.248 INFO:tasks.rgw_multisite_tests:======================================================================
2022-06-11T21:00:16.248 INFO:tasks.rgw_multisite_tests:FAIL: test notification of deletion markers
2022-06-11T21:00:16.248 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-06-11T21:00:16.249 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-06-11T21:00:16.249 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-06-11T21:00:16.250 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-06-11T21:00:16.250 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 2774, in test_ps_versioned_deletion
2022-06-11T21:00:16.251 INFO:tasks.rgw_multisite_tests:    master_zone, ps_zone = init_env()
2022-06-11T21:00:16.251 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 573, in init_env
2022-06-11T21:00:16.252 INFO:tasks.rgw_multisite_tests:    zonegroup_meta_checkpoint(zonegroup)
2022-06-11T21:00:16.252 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint
2022-06-11T21:00:16.252 INFO:tasks.rgw_multisite_tests:    zone_meta_checkpoint(zone, meta_master_zone, master_status)
2022-06-11T21:00:16.253 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 194, in zone_meta_checkpoint
2022-06-11T21:00:16.253 INFO:tasks.rgw_multisite_tests:    assert False, 'failed meta checkpoint for zone=%s' % zone.name
2022-06-11T21:00:16.253 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2
2022-06-11T21:00:16.254 INFO:tasks.rgw_multisite_tests:
2022-06-11T21:00:16.254 INFO:tasks.rgw_multisite_tests:======================================================================
2022-06-11T21:00:16.255 INFO:tasks.rgw_multisite_tests:FAIL: test notification status upon bucket deletion
2022-06-11T21:00:16.255 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-06-11T21:00:16.256 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-06-11T21:00:16.256 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-06-11T21:00:16.257 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-06-11T21:00:16.257 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 3480, in test_ps_delete_bucket
2022-06-11T21:00:16.257 INFO:tasks.rgw_multisite_tests:    master_zone, ps_zone = init_env()
2022-06-11T21:00:16.258 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 573, in init_env
2022-06-11T21:00:16.258 INFO:tasks.rgw_multisite_tests:    zonegroup_meta_checkpoint(zonegroup)
2022-06-11T21:00:16.259 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint
2022-06-11T21:00:16.259 INFO:tasks.rgw_multisite_tests:    zone_meta_checkpoint(zone, meta_master_zone, master_status)
2022-06-11T21:00:16.259 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 182, in zone_meta_checkpoint
2022-06-11T21:00:16.260 INFO:tasks.rgw_multisite_tests:    period, realm_epoch, num_shards, sync_status = meta_sync_status(zone)
2022-06-11T21:00:16.260 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 139, in meta_sync_status
2022-06-11T21:00:16.260 INFO:tasks.rgw_multisite_tests:    assert(retcode == 2) # ENOENT
2022-06-11T21:00:16.261 INFO:tasks.rgw_multisite_tests:AssertionError
2022-06-11T21:00:16.261 INFO:tasks.rgw_multisite_tests:
2022-06-11T21:00:16.261 INFO:tasks.rgw_multisite_tests:======================================================================
2022-06-11T21:00:16.262 INFO:tasks.rgw_multisite_tests:FAIL: test creating a subscription when no topic info exists
2022-06-11T21:00:16.263 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-06-11T21:00:16.263 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-06-11T21:00:16.264 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-06-11T21:00:16.264 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-06-11T21:00:16.265 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 3549, in test_ps_missing_topic
2022-06-11T21:00:16.265 INFO:tasks.rgw_multisite_tests:    master_zone, ps_zone = init_env()
2022-06-11T21:00:16.266 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 573, in init_env
2022-06-11T21:00:16.266 INFO:tasks.rgw_multisite_tests:    zonegroup_meta_checkpoint(zonegroup)
2022-06-11T21:00:16.267 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint
2022-06-11T21:00:16.267 INFO:tasks.rgw_multisite_tests:    zone_meta_checkpoint(zone, meta_master_zone, master_status)
2022-06-11T21:00:16.268 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 182, in zone_meta_checkpoint
2022-06-11T21:00:16.268 INFO:tasks.rgw_multisite_tests:    period, realm_epoch, num_shards, sync_status = meta_sync_status(zone)
2022-06-11T21:00:16.269 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 139, in meta_sync_status
2022-06-11T21:00:16.269 INFO:tasks.rgw_multisite_tests:    assert(retcode == 2) # ENOENT
2022-06-11T21:00:16.270 INFO:tasks.rgw_multisite_tests:AssertionError
2022-06-11T21:00:16.271 INFO:tasks.rgw_multisite_tests:
2022-06-11T21:00:16.271 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-06-11T21:00:16.272 INFO:tasks.rgw_multisite_tests:Ran 74 tests in 7205.155s
2022-06-11T21:00:16.272 INFO:tasks.rgw_multisite_tests:
2022-06-11T21:00:16.273 INFO:tasks.rgw_multisite_tests:FAILED (failures=3)
2022-06-11T21:00:16.274 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/teuthology/run_tasks.py", line 106, in run_tasks
    manager.__enter__()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/teuthology/task/__init__.py", line 123, in __enter__
    self.begin()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_9c59f57df6b60922a0810152a42e8734514d99c7/qa/tasks/rgw_multisite_tests.py", line 93, in begin
    assert nose.run(argv=argv, config=conf), 'rgw multisite test failures'
AssertionError: rgw multisite test failures
2022-06-11T21:00:16.357 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c5eca2aced4a4a5d87c3998d5a111779
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/teuthology/run_tasks.py", line 106, in run_tasks
    manager.__enter__()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/teuthology/task/__init__.py", line 123, in __enter__
    self.begin()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_9c59f57df6b60922a0810152a42e8734514d99c7/qa/tasks/rgw_multisite_tests.py", line 93, in begin
    assert nose.run(argv=argv, config=conf), 'rgw multisite test failures'
AssertionError: rgw multisite test failures

Related issues 1 (0 open — 1 closed)