Bug #55183
test: pubsub tests are failing in multisite suite
Status:
Duplicate
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
this is seen in the master baseline from 6-March: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-04_22:06:30-rgw-master-distro-default-smithi/6722135/teuthology.log
2022-03-05T14:31:50.120 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-03-05T14:31:50.121 INFO:tasks.rgw_multisite_tests:FAIL: test s3 records fetching 2022-03-05T14:31:50.121 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-03-05T14:31:50.122 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-03-05T14:31:50.122 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-03-05T14:31:50.122 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-03-05T14:31:50.123 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 773, in test_ps_s3_notification_records 2022-03-05T14:31:50.123 INFO:tasks.rgw_multisite_tests: verify_s3_records_by_elements(records, keys, exact_match=False) 2022-03-05T14:31:50.123 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 358, in verify_s3_records_by_elements 2022-03-05T14:31:50.124 INFO:tasks.rgw_multisite_tests: assert False, err 2022-03-05T14:31:50.124 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-72,0> 2022-03-05T14:31:50.125 INFO:tasks.rgw_multisite_tests: 2022-03-05T14:31:50.125 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-03-05T14:31:50.125 INFO:tasks.rgw_multisite_tests:FAIL: test set/get/delete of subscription 2022-03-05T14:31:50.126 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-03-05T14:31:50.126 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-03-05T14:31:50.126 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-03-05T14:31:50.127 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-03-05T14:31:50.127 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1291, in test_ps_subscription 2022-03-05T14:31:50.127 INFO:tasks.rgw_multisite_tests: verify_events_by_elements(events, keys, exact_match=False) 2022-03-05T14:31:50.128 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements 2022-03-05T14:31:50.128 INFO:tasks.rgw_multisite_tests: assert False, err 2022-03-05T14:31:50.128 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-78,0> 2022-03-05T14:31:50.129 INFO:tasks.rgw_multisite_tests: 2022-03-05T14:31:50.129 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-03-05T14:31:50.130 INFO:tasks.rgw_multisite_tests:FAIL: test radosgw-admin commands 2022-03-05T14:31:50.130 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-03-05T14:31:50.130 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-03-05T14:31:50.131 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-03-05T14:31:50.131 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-03-05T14:31:50.131 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1402, in test_ps_admin 2022-03-05T14:31:50.132 INFO:tasks.rgw_multisite_tests: verify_events_by_elements({"events": events1+events2}, keys, exact_match=False) 2022-03-05T14:31:50.132 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements 2022-03-05T14:31:50.132 INFO:tasks.rgw_multisite_tests: assert False, err 2022-03-05T14:31:50.133 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-79,0> 2022-03-05T14:31:50.133 INFO:tasks.rgw_multisite_tests: 2022-03-05T14:31:50.134 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-03-05T14:31:50.134 INFO:tasks.rgw_multisite_tests:FAIL: test that events are only sent on incremental sync 2022-03-05T14:31:50.134 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-03-05T14:31:50.135 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-03-05T14:31:50.135 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-03-05T14:31:50.135 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-03-05T14:31:50.136 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1474, in test_ps_incremental_sync 2022-03-05T14:31:50.136 INFO:tasks.rgw_multisite_tests: assert_equal(count, number_of_objects) 2022-03-05T14:31:50.136 INFO:tasks.rgw_multisite_tests:AssertionError: 0 != 10 2022-03-05T14:31:50.137 INFO:tasks.rgw_multisite_tests: 2022-03-05T14:31:50.137 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-03-05T14:31:50.138 INFO:tasks.rgw_multisite_tests:FAIL: test subscriptions for different events 2022-03-05T14:31:50.138 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-03-05T14:31:50.138 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-03-05T14:31:50.139 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-03-05T14:31:50.139 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-03-05T14:31:50.139 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1552, in test_ps_event_type_subscription 2022-03-05T14:31:50.140 INFO:tasks.rgw_multisite_tests: verify_events_by_elements(events, keys, exact_match=False) 2022-03-05T14:31:50.140 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements 2022-03-05T14:31:50.140 INFO:tasks.rgw_multisite_tests: assert False, err 2022-03-05T14:31:50.141 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-81,0> 2022-03-05T14:31:50.141 INFO:tasks.rgw_multisite_tests: 2022-03-05T14:31:50.142 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-03-05T14:31:50.142 INFO:tasks.rgw_multisite_tests:FAIL: test incremental fetching of events from a subscription 2022-03-05T14:31:50.142 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-03-05T14:31:50.143 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-03-05T14:31:50.143 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-03-05T14:31:50.143 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-03-05T14:31:50.144 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1670, in test_ps_event_fetching 2022-03-05T14:31:50.144 INFO:tasks.rgw_multisite_tests: verify_events_by_elements({'events': all_events}, keys, exact_match=False) 2022-03-05T14:31:50.145 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements 2022-03-05T14:31:50.145 INFO:tasks.rgw_multisite_tests: assert False, err 2022-03-05T14:31:50.145 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-82,0> 2022-03-05T14:31:50.146 INFO:tasks.rgw_multisite_tests: 2022-03-05T14:31:50.146 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-03-05T14:31:50.146 INFO:tasks.rgw_multisite_tests:FAIL: test acking of some events in a subscription 2022-03-05T14:31:50.147 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-03-05T14:31:50.147 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-03-05T14:31:50.148 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-03-05T14:31:50.148 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-03-05T14:31:50.148 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1720, in test_ps_event_acking 2022-03-05T14:31:50.149 INFO:tasks.rgw_multisite_tests: verify_events_by_elements(events, keys, exact_match=False) 2022-03-05T14:31:50.149 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 312, in verify_events_by_elements 2022-03-05T14:31:50.149 INFO:tasks.rgw_multisite_tests: assert False, err 2022-03-05T14:31:50.150 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-83,0> 2022-03-05T14:31:50.150 INFO:tasks.rgw_multisite_tests: 2022-03-05T14:31:50.150 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-03-05T14:31:50.151 INFO:tasks.rgw_multisite_tests:FAIL: test object creation notifications in using put/copy/post 2022-03-05T14:31:50.151 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-03-05T14:31:50.152 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-03-05T14:31:50.152 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-03-05T14:31:50.152 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-03-05T14:31:50.153 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 1797, in test_ps_creation_triggers 2022-03-05T14:31:50.153 INFO:tasks.rgw_multisite_tests: assert len(events['events']) >= 3 2022-03-05T14:31:50.153 INFO:tasks.rgw_multisite_tests:AssertionError 2022-03-05T14:31:50.154 INFO:tasks.rgw_multisite_tests: 2022-03-05T14:31:50.154 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-03-05T14:31:50.154 INFO:tasks.rgw_multisite_tests:FAIL: test notification status upon bucket deletion 2022-03-05T14:31:50.155 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-03-05T14:31:50.155 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-03-05T14:31:50.155 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-03-05T14:31:50.156 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-03-05T14:31:50.156 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 2217, in test_ps_delete_bucket 2022-03-05T14:31:50.157 INFO:tasks.rgw_multisite_tests: verify_s3_records_by_elements(records, keys, exact_match=False) 2022-03-05T14:31:50.157 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_27bad0cf8decf43ee4c77b0b76e8f0be5095a177/qa/../src/test/rgw/rgw_multi/tests_ps.py", line 358, in verify_s3_records_by_elements 2022-03-05T14:31:50.157 INFO:tasks.rgw_multisite_tests: assert False, err 2022-03-05T14:31:50.158 INFO:tasks.rgw_multisite_tests:AssertionError: no creation event found for key: <Key: qvbrjt-86,0> 2022-03-05T14:31:50.158 INFO:tasks.rgw_multisite_tests:
Related issues
History
#1 Updated by Casey Bodley almost 2 years ago
- Related to Bug #55498: "AssertionError: rgw multisite test failures" in upgrade:octopus added
#2 Updated by Yuval Lifshitz over 1 year ago
- Status changed from New to Duplicate
duplicate of https://tracker.ceph.com/issues/56572