Project

General

Profile

Bug #55498

"AssertionError: rgw multisite test failures" in upgrade:octopus

Added by Yuri Weinstein 7 months ago. Updated 20 days ago.

Status:
Duplicate
Priority:
Urgent
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
pubsub
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.front.sepia.ceph.com/yuriw-2022-04-30_04:09:01-upgrade:octopus-x-wip-yuri3-testing-2022-04-29-1830-distro-default-smithi/
Jobs: 6815714 6815743
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-30_04:09:01-upgrade:octopus-x-wip-yuri3-testing-2022-04-29-1830-distro-default-smithi/6815714/teuthology.log

2022-04-30T09:46:41.924 INFO:tasks.rgw_multisite_tests:test creating a subscription when no topic info exists ... FAIL
2022-04-30T09:46:41.925 INFO:tasks.rgw_multisite_tests:test updating topic associated with a notification ... ok
2022-04-30T09:46:41.926 INFO:tasks.rgw_multisite_tests:test updating the topic of a notification ... ok
2022-04-30T09:46:41.926 INFO:tasks.rgw_multisite_tests:test notification creation with multiple topics ... ok
2022-04-30T09:46:41.927 INFO:tasks.rgw_multisite_tests:
2022-04-30T09:46:41.927 INFO:tasks.rgw_multisite_tests:======================================================================
2022-04-30T09:46:41.927 INFO:tasks.rgw_multisite_tests:FAIL: test notification of deletion markers
2022-04-30T09:46:41.928 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-04-30T09:46:41.928 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-04-30T09:46:41.928 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-04-30T09:46:41.928 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-04-30T09:46:41.929 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 2774, in test_ps_versioned_deletion
2022-04-30T09:46:41.929 INFO:tasks.rgw_multisite_tests:    master_zone, ps_zone = init_env()
2022-04-30T09:46:41.929 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 573, in init_env
2022-04-30T09:46:41.929 INFO:tasks.rgw_multisite_tests:    zonegroup_meta_checkpoint(zonegroup)
2022-04-30T09:46:41.929 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint
2022-04-30T09:46:41.930 INFO:tasks.rgw_multisite_tests:    zone_meta_checkpoint(zone, meta_master_zone, master_status)
2022-04-30T09:46:41.930 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 182, in zone_meta_checkpoint
2022-04-30T09:46:41.930 INFO:tasks.rgw_multisite_tests:    period, realm_epoch, num_shards, sync_status = meta_sync_status(zone)
2022-04-30T09:46:41.930 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 139, in meta_sync_status
2022-04-30T09:46:41.930 INFO:tasks.rgw_multisite_tests:    assert(retcode == 2) # ENOENT
2022-04-30T09:46:41.930 INFO:tasks.rgw_multisite_tests:AssertionError
2022-04-30T09:46:41.931 INFO:tasks.rgw_multisite_tests:
2022-04-30T09:46:41.931 INFO:tasks.rgw_multisite_tests:======================================================================
2022-04-30T09:46:41.931 INFO:tasks.rgw_multisite_tests:FAIL: test notification status upon bucket deletion
2022-04-30T09:46:41.931 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-04-30T09:46:41.931 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-04-30T09:46:41.932 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-04-30T09:46:41.932 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-04-30T09:46:41.932 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 3480, in test_ps_delete_bucket
2022-04-30T09:46:41.932 INFO:tasks.rgw_multisite_tests:    master_zone, ps_zone = init_env()
2022-04-30T09:46:41.932 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 573, in init_env
2022-04-30T09:46:41.933 INFO:tasks.rgw_multisite_tests:    zonegroup_meta_checkpoint(zonegroup)
2022-04-30T09:46:41.933 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint
2022-04-30T09:46:41.933 INFO:tasks.rgw_multisite_tests:    zone_meta_checkpoint(zone, meta_master_zone, master_status)
2022-04-30T09:46:41.933 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 182, in zone_meta_checkpoint
2022-04-30T09:46:41.933 INFO:tasks.rgw_multisite_tests:    period, realm_epoch, num_shards, sync_status = meta_sync_status(zone)
2022-04-30T09:46:41.934 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 139, in meta_sync_status
2022-04-30T09:46:41.934 INFO:tasks.rgw_multisite_tests:    assert(retcode == 2) # ENOENT
2022-04-30T09:46:41.934 INFO:tasks.rgw_multisite_tests:AssertionError
2022-04-30T09:46:41.934 INFO:tasks.rgw_multisite_tests:
2022-04-30T09:46:41.934 INFO:tasks.rgw_multisite_tests:======================================================================
2022-04-30T09:46:41.935 INFO:tasks.rgw_multisite_tests:FAIL: test creating a subscription when no topic info exists
2022-04-30T09:46:41.935 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-04-30T09:46:41.935 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-04-30T09:46:41.935 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-04-30T09:46:41.935 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-04-30T09:46:41.936 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 3549, in test_ps_missing_topic
2022-04-30T09:46:41.936 INFO:tasks.rgw_multisite_tests:    master_zone, ps_zone = init_env()
2022-04-30T09:46:41.936 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 573, in init_env
2022-04-30T09:46:41.936 INFO:tasks.rgw_multisite_tests:    zonegroup_meta_checkpoint(zonegroup)
2022-04-30T09:46:41.936 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint
2022-04-30T09:46:41.937 INFO:tasks.rgw_multisite_tests:    zone_meta_checkpoint(zone, meta_master_zone, master_status)
2022-04-30T09:46:41.937 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 182, in zone_meta_checkpoint
2022-04-30T09:46:41.937 INFO:tasks.rgw_multisite_tests:    period, realm_epoch, num_shards, sync_status = meta_sync_status(zone)
2022-04-30T09:46:41.937 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 139, in meta_sync_status
2022-04-30T09:46:41.937 INFO:tasks.rgw_multisite_tests:    assert(retcode == 2) # ENOENT
2022-04-30T09:46:41.938 INFO:tasks.rgw_multisite_tests:AssertionError
2022-04-30T09:46:41.938 INFO:tasks.rgw_multisite_tests:
2022-04-30T09:46:41.938 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-04-30T09:46:41.938 INFO:tasks.rgw_multisite_tests:Ran 74 tests in 10240.297s
2022-04-30T09:46:41.938 INFO:tasks.rgw_multisite_tests:
2022-04-30T09:46:41.939 INFO:tasks.rgw_multisite_tests:FAILED (failures=3)
2022-04-30T09:46:41.939 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/teuthology/run_tasks.py", line 106, in run_tasks
    manager.__enter__()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/teuthology/task/__init__.py", line 123, in __enter__
    self.begin()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_4b999a23c145e80320290b03ae8fcf3a81b1e329/qa/tasks/rgw_multisite_tests.py", line 93, in begin
    assert nose.run(argv=argv, config=conf), 'rgw multisite test failures'
AssertionError: rgw multisite test failures
2022-04-30T09:46:42.074 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=90ec645321a0495c899eef8ce406d637
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/teuthology/run_tasks.py", line 106, in run_tasks
    manager.__enter__()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/teuthology/task/__init__.py", line 123, in __enter__
    self.begin()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_4b999a23c145e80320290b03ae8fcf3a81b1e329/qa/tasks/rgw_multisite_tests.py", line 93, in begin
    assert nose.run(argv=argv, config=conf), 'rgw multisite test failures'
AssertionError: rgw multisite test failures

Related issues

Related to rgw-testing - Bug #55183: test: pubsub tests are failing in multisite suite New
Duplicated by Ceph - Bug #56007: "AssertionError: rgw multisite test failures" in upgrade:octopus-x-quincy-release Duplicate

History

#1 Updated by Casey Bodley 7 months ago

  • Status changed from New to Triaged
  • Assignee set to Yuval Lifshitz

#2 Updated by Casey Bodley 7 months ago

  • Related to Bug #55183: test: pubsub tests are failing in multisite suite added

#3 Updated by Casey Bodley 6 months ago

  • Duplicated by Bug #56007: "AssertionError: rgw multisite test failures" in upgrade:octopus-x-quincy-release added

#4 Updated by Yuri Weinstein 6 months ago

  • Priority changed from Normal to Urgent

Changed the priority as it's for 17.2.1

#5 Updated by Yuval Lifshitz 3 months ago

  • Tags set to pubsub

#6 Updated by Yuval Lifshitz 20 days ago

  • Status changed from Triaged to Duplicate

will be fixed as part of: https://tracker.ceph.com/issues/58036

Also available in: Atom PDF