Bug #39142: multisite: test_versioned_object_incremental_sync sometimes fails in three-zone configuration - rgw - Ceph

Actions

Copy link

Bug #39142

closed

multisite: test_versioned_object_incremental_sync sometimes fails in three-zone configuration

Added by Casey Bodley about 5 years ago. Updated about 4 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Tags:

multisite versioning

Backport:

nautilus

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

31325

Crash signature (v1):

Crash signature (v2):

Description

The three-zone multisite tests sometime fail to converge on the same version. This shows up in one of two failures:

2019-04-04T15:58:34.813 INFO:tasks.rgw_multisite_tests:======================================================================
2019-04-04T15:58:34.813 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_versioned_object_incremental_sync
2019-04-04T15:58:34.813 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2019-04-04T15:58:34.813 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2019-04-04T15:58:34.813 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
2019-04-04T15:58:34.813 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2019-04-04T15:58:34.813 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-rgw-non-existent-object/qa/tasks/rgw_multi/tests.py", line 711, in test_versioned_object_incremental_sync
2019-04-04T15:58:34.814 INFO:tasks.rgw_multisite_tests:    zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name)
2019-04-04T15:58:34.814 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-rgw-non-existent-object/qa/tasks/rgw_multi/tests.py", line 419, in zonegroup_bucket_checkpoint
2019-04-04T15:58:34.814 INFO:tasks.rgw_multisite_tests:    target_conn.check_bucket_eq(source_conn, bucket_name)
2019-04-04T15:58:34.814 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-rgw-non-existent-object/qa/tasks/rgw_multi/zone_rados.py", line 86, in check_bucket_eq
2019-04-04T15:58:34.814 INFO:tasks.rgw_multisite_tests:    check_object_eq(k1, k2)
2019-04-04T15:58:34.814 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-rgw-non-existent-object/qa/tasks/rgw_multi/zone_rados.py", line 21, in check_object_eq
2019-04-04T15:58:34.814 INFO:tasks.rgw_multisite_tests:    eq(k1.is_latest, k2.is_latest)
2019-04-04T15:58:34.815 INFO:tasks.rgw_multisite_tests:AssertionError: False != True
2019-04-04T15:58:34.815 INFO:tasks.rgw_multisite_tests:
2019-04-04T15:58:34.815 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------

2019-04-04T04:06:54.751 INFO:tasks.rgw_multisite_tests:======================================================================
2019-04-04T04:06:54.751 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_versioned_object_incremental_sync
2019-04-04T04:06:54.752 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2019-04-04T04:06:54.752 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2019-04-04T04:06:54.752 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
2019-04-04T04:06:54.752 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2019-04-04T04:06:54.752 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_luminous/qa/tasks/rgw_multi/tests.py", line 694, in test_versioned_object_incremental_sync
2019-04-04T04:06:54.752 INFO:tasks.rgw_multisite_tests:    zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name)
2019-04-04T04:06:54.752 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_luminous/qa/tasks/rgw_multi/tests.py", line 416, in zonegroup_bucket_checkpoint
2019-04-04T04:06:54.752 INFO:tasks.rgw_multisite_tests:    target_conn.check_bucket_eq(source_conn, bucket_name)
2019-04-04T04:06:54.753 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_luminous/qa/tasks/rgw_multi/zone_rados.py", line 83, in check_bucket_eq
2019-04-04T04:06:54.753 INFO:tasks.rgw_multisite_tests:    check_object_eq(k1, k2)
2019-04-04T04:06:54.753 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/github.com_ceph_ceph_luminous/qa/tasks/rgw_multi/zone_rados.py", line 20, in check_object_eq
2019-04-04T04:06:54.753 INFO:tasks.rgw_multisite_tests:    eq(k1.version_id, k2.version_id)
2019-04-04T04:06:54.753 INFO:tasks.rgw_multisite_tests:AssertionError: u'6vSjWdhmOD-m2Ax2nFUDCEo7N27m1aP' != u'RW8upN1jkR5m3Or7OpG-d5AVuDNdXwQ'
2019-04-04T04:06:54.753 INFO:tasks.rgw_multisite_tests:
2019-04-04T04:06:54.753 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------

Related issues 1 (0 open — 1 closed)