Bug #44493
qa/tasks/rgw_multisite_tests.py fails with "RuntimeError: rgw multisite, rgw multisite pubsub, test failures"
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2020-03-06T07:14:35.289 INFO:tasks.rgw_multisite_tests:FAIL: test s3 topics set/get/delete on master 2020-03-06T07:14:35.289 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-06T07:14:35.290 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-06T07:14:35.290 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-06T07:14:35.290 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-06T07:14:35.290 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/rgw_multi/tests_ps.py", line 829, in test_ps_s3_topic_on_master 2020-03-06T07:14:35.290 INFO:tasks.rgw_multisite_tests: master_zone, _ = init_env(require_ps=False) 2020-03-06T07:14:35.290 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/rgw_multi/tests_ps.py", line 570, in init_env 2020-03-06T07:14:35.290 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2020-03-06T07:14:35.290 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2020-03-06T07:14:35.291 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2020-03-06T07:14:35.291 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2020-03-06T07:14:35.291 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2020-03-06T07:14:35.291 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3
rgw/multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml}
/a/kchai-2020-03-06_04:33:31-rgw-master-distro-basic-smithi/4829214
History
#1 Updated by Kefu Chai about 4 years ago
Yuval, could you take a look?
#2 Updated by Yuval Lifshitz about 4 years ago
looking into it. no necessarily related to pubsub, could be due to sync issues. both pubsub and multisite tests are failing here:
http://qa-proxy.ceph.com/teuthology/yuvalif-2020-03-04_09:07:58-rgw-fix_data_corruption_in_cls_queue_head-distro-basic-smithi/4824274/teuthology.log
and here:
http://qa-proxy.ceph.com/teuthology/yuvalif-2020-03-04_09:07:58-rgw-fix_data_corruption_in_cls_queue_head-distro-basic-smithi/4824327/teuthology.log
#3 Updated by Kefu Chai about 4 years ago
Yuval, could you assign this issue to the developer who's more familiar with sync?
#4 Updated by Yuval Lifshitz about 4 years ago
- Status changed from New to Duplicate
this is being fixed by cbodley in: https://tracker.ceph.com/issues/43768
this is a multisite issue - not specific to pubsub