Project

General

Profile

Bug #55183

test: pubsub tests are failing in multisite suite

Added by Yuval Lifshitz almost 2 years ago. Updated over 1 year ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

this is seen in the master baseline from 6-March: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-04_22:06:30-rgw-master-distro-default-smithi/6722135/teuthology.log

2022-03-05T14:31:50.120 INFO:tasks.rgw_multisite_tests:======================================================================
2022-03-05T14:31:50.121 INFO:tasks.rgw_multisite_tests:FAIL: test s3 records fetching
2022-03-05T14:31:50.121 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-03-05T14:31:50.122 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-03-05T14:31:50.122 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-03-05T14:31:50.122 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-03-05T14:31:50.123 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 773, in test_ps_s3_notification_records
2022-03-05T14:31:50.123 INFO:tasks.rgw_multisite_tests:    verify_s3_records_by_elements(records, keys, exact_match=False)
2022-03-05T14:31:50.123 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 358, in verify_s3_records_by_elements
2022-03-05T14:31:50.124 INFO:tasks.rgw_multisite_tests:    assert False, err
2022-03-05T14:31:50.124 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-72,0>
2022-03-05T14:31:50.125 INFO:tasks.rgw_multisite_tests:
2022-03-05T14:31:50.125 INFO:tasks.rgw_multisite_tests:======================================================================
2022-03-05T14:31:50.125 INFO:tasks.rgw_multisite_tests:FAIL: test set/get/delete of subscription
2022-03-05T14:31:50.126 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-03-05T14:31:50.126 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-03-05T14:31:50.126 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-03-05T14:31:50.127 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-03-05T14:31:50.127 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1291, in test_ps_subscription
2022-03-05T14:31:50.127 INFO:tasks.rgw_multisite_tests:    verify_events_by_elements(events, keys, exact_match=False)
2022-03-05T14:31:50.128 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements
2022-03-05T14:31:50.128 INFO:tasks.rgw_multisite_tests:    assert False, err
2022-03-05T14:31:50.128 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-78,0>
2022-03-05T14:31:50.129 INFO:tasks.rgw_multisite_tests:
2022-03-05T14:31:50.129 INFO:tasks.rgw_multisite_tests:======================================================================
2022-03-05T14:31:50.130 INFO:tasks.rgw_multisite_tests:FAIL: test radosgw-admin commands
2022-03-05T14:31:50.130 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-03-05T14:31:50.130 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-03-05T14:31:50.131 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-03-05T14:31:50.131 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-03-05T14:31:50.131 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1402, in test_ps_admin
2022-03-05T14:31:50.132 INFO:tasks.rgw_multisite_tests:    verify_events_by_elements({"events": events1+events2}, keys, exact_match=False)
2022-03-05T14:31:50.132 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements
2022-03-05T14:31:50.132 INFO:tasks.rgw_multisite_tests:    assert False, err
2022-03-05T14:31:50.133 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-79,0>
2022-03-05T14:31:50.133 INFO:tasks.rgw_multisite_tests:
2022-03-05T14:31:50.134 INFO:tasks.rgw_multisite_tests:======================================================================
2022-03-05T14:31:50.134 INFO:tasks.rgw_multisite_tests:FAIL: test that events are only sent on incremental sync
2022-03-05T14:31:50.134 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-03-05T14:31:50.135 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-03-05T14:31:50.135 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-03-05T14:31:50.135 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-03-05T14:31:50.136 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1474, in test_ps_incremental_sync
2022-03-05T14:31:50.136 INFO:tasks.rgw_multisite_tests:    assert_equal(count, number_of_objects)
2022-03-05T14:31:50.136 INFO:tasks.rgw_multisite_tests:AssertionError: 0 != 10
2022-03-05T14:31:50.137 INFO:tasks.rgw_multisite_tests:
2022-03-05T14:31:50.137 INFO:tasks.rgw_multisite_tests:======================================================================
2022-03-05T14:31:50.138 INFO:tasks.rgw_multisite_tests:FAIL: test subscriptions for different events
2022-03-05T14:31:50.138 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-03-05T14:31:50.138 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-03-05T14:31:50.139 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-03-05T14:31:50.139 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-03-05T14:31:50.139 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1552, in test_ps_event_type_subscription
2022-03-05T14:31:50.140 INFO:tasks.rgw_multisite_tests:    verify_events_by_elements(events, keys, exact_match=False)
2022-03-05T14:31:50.140 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements
2022-03-05T14:31:50.140 INFO:tasks.rgw_multisite_tests:    assert False, err
2022-03-05T14:31:50.141 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-81,0>
2022-03-05T14:31:50.141 INFO:tasks.rgw_multisite_tests:
2022-03-05T14:31:50.142 INFO:tasks.rgw_multisite_tests:======================================================================
2022-03-05T14:31:50.142 INFO:tasks.rgw_multisite_tests:FAIL: test incremental fetching of events from a subscription
2022-03-05T14:31:50.142 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-03-05T14:31:50.143 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-03-05T14:31:50.143 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-03-05T14:31:50.143 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-03-05T14:31:50.144 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1670, in test_ps_event_fetching
2022-03-05T14:31:50.144 INFO:tasks.rgw_multisite_tests:    verify_events_by_elements({'events': all_events}, keys, exact_match=False)
2022-03-05T14:31:50.145 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements
2022-03-05T14:31:50.145 INFO:tasks.rgw_multisite_tests:    assert False, err
2022-03-05T14:31:50.145 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-82,0>
2022-03-05T14:31:50.146 INFO:tasks.rgw_multisite_tests:
2022-03-05T14:31:50.146 INFO:tasks.rgw_multisite_tests:======================================================================
2022-03-05T14:31:50.146 INFO:tasks.rgw_multisite_tests:FAIL: test acking of some events in a subscription
2022-03-05T14:31:50.147 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-03-05T14:31:50.147 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-03-05T14:31:50.148 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-03-05T14:31:50.148 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-03-05T14:31:50.148 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1720, in test_ps_event_acking
2022-03-05T14:31:50.149 INFO:tasks.rgw_multisite_tests:    verify_events_by_elements(events, keys, exact_match=False)
2022-03-05T14:31:50.149 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements
2022-03-05T14:31:50.149 INFO:tasks.rgw_multisite_tests:    assert False, err
2022-03-05T14:31:50.150 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-83,0>
2022-03-05T14:31:50.150 INFO:tasks.rgw_multisite_tests:
2022-03-05T14:31:50.150 INFO:tasks.rgw_multisite_tests:======================================================================
2022-03-05T14:31:50.151 INFO:tasks.rgw_multisite_tests:FAIL: test object creation notifications in using put/copy/post
2022-03-05T14:31:50.151 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-03-05T14:31:50.152 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-03-05T14:31:50.152 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-03-05T14:31:50.152 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-03-05T14:31:50.153 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1797, in test_ps_creation_triggers
2022-03-05T14:31:50.153 INFO:tasks.rgw_multisite_tests:    assert len(events['events']) >= 3
2022-03-05T14:31:50.153 INFO:tasks.rgw_multisite_tests:AssertionError
2022-03-05T14:31:50.154 INFO:tasks.rgw_multisite_tests:
2022-03-05T14:31:50.154 INFO:tasks.rgw_multisite_tests:======================================================================
2022-03-05T14:31:50.154 INFO:tasks.rgw_multisite_tests:FAIL: test notification status upon bucket deletion
2022-03-05T14:31:50.155 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2022-03-05T14:31:50.155 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2022-03-05T14:31:50.155 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-03-05T14:31:50.156 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2022-03-05T14:31:50.156 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 2217, in test_ps_delete_bucket
2022-03-05T14:31:50.157 INFO:tasks.rgw_multisite_tests:    verify_s3_records_by_elements(records, keys, exact_match=False)
2022-03-05T14:31:50.157 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 358, in verify_s3_records_by_elements
2022-03-05T14:31:50.157 INFO:tasks.rgw_multisite_tests:    assert False, err
2022-03-05T14:31:50.158 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-86,0>
2022-03-05T14:31:50.158 INFO:tasks.rgw_multisite_tests:


Related issues

Related to rgw - Bug #55498: "AssertionError: rgw multisite test failures" in upgrade:octopus Duplicate

History

#1 Updated by Casey Bodley almost 2 years ago

  • Related to Bug #55498: "AssertionError: rgw multisite test failures" in upgrade:octopus added

#2 Updated by Yuval Lifshitz over 1 year ago

  • Status changed from New to Duplicate

Also available in: Atom PDF