Project

General

Profile

Actions
Copy link

Bug #21591

closed

RGW multisite does not sync all objects

Added by Anonymous over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
High
Assignee:
Yehuda Sadeh
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v12.2.0
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I setup a multisite sync between 2 luminous clusters. The clusters were deployed with ceph-ansible. The Sync seemed to work fine in both directions when I tested with some bucket operations and small objects. However, when I really started using it as storage for a docker registry, I noticed not all objects seem to sync correctly.

master zone:

  1. s3cmd --config s3cfg_s3_bccl_tda du s3://tda-registry
    9090457213 1120 objects s3://tda-registry/

secondary zone:

  1. s3cmd --config s3cfg_s3_bccm_tda du s3://tda-registry
    851591006 943 objects s3://tda-registry/

Altough the buckets are clearly not in sync, the sync status keeps reporting everything is fine and caught up with the source:

master zone:

realm 0f33e8d4-825c-464b-90c5-87a44d99f6fc (tda)
      zonegroup 5ce69d1a-097d-4ef7-ae0f-6f356f76de0c (be)
           zone 6c82776a-a9c0-46ba-b89a-500958e65b15 (bccl-tda)
  metadata sync no sync (zone is master)
      data sync source: 2c7a4a95-1922-49fb-bf5f-f550309d611d (bccm-tda)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

secondary zone:
realm 0f33e8d4-825c-464b-90c5-87a44d99f6fc (tda)
zonegroup 5ce69d1a-097d-4ef7-ae0f-6f356f76de0c (be)
zone 2c7a4a95-1922-49fb-bf5f-f550309d611d (bccm-tda)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
data sync source: 6c82776a-a9c0-46ba-b89a-500958e65b15 (bccl-tda)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source

When I push some more containers to the registry that is using this bucket, it seems the sync is indeed still working, but some objects seem to be ignored:

master:
  1. s3cmd --config s3cfg_s3_bccl_tda du s3://tda-registry
    9585013621 1206 objects s3://tda-registry/
secondary:
  1. s3cmd --config s3cfg_s3_bccm_tda du s3://tda-registry
    960073106 1025 objects s3://tda-registry/

Judging by the size of the bucket, it looks to me the larger objects are not synced.

When I disable and re-enable the sync on this bucket, the buckets get in sync again. Sometimes I need to disable/enable the sync 2 or 3 times to have all objects in sync.

Any pointers are greatly appreciated.

Files

Download all files
secondary-rgw-rgw1.log00.bz2 (869 KB) secondary-rgw-rgw1.log00.bz2 secondary site, one of the rgw's Anonymous, 10/09/2017 12:07 PM
secondary-rgw-rgw1.log01.bz2 (868 KB) secondary-rgw-rgw1.log01.bz2 next part of debug log secondary site, first rgw Anonymous, 10/10/2017 05:45 AM
secondary-single_rgw.log00.bz2 (568 KB) secondary-single_rgw.log00.bz2 Anonymous, 10/10/2017 07:53 AM
secondary-single_rgw.log01.bz2 (555 KB) secondary-single_rgw.log01.bz2 Anonymous, 10/10/2017 07:53 AM
secondary-single_rgw.log02.bz2 (551 KB) secondary-single_rgw.log02.bz2 Anonymous, 10/10/2017 07:53 AM
secondary-single_rgw.log03.bz2 (556 KB) secondary-single_rgw.log03.bz2 Anonymous, 10/10/2017 07:53 AM
secondary-single_rgw.log04.bz2 (550 KB) secondary-single_rgw.log04.bz2 Anonymous, 10/10/2017 07:53 AM
secondary-single_rgw.log05.bz2 (554 KB) secondary-single_rgw.log05.bz2 Anonymous, 10/10/2017 07:53 AM
secondary-single_rgw.log06.bz2 (550 KB) secondary-single_rgw.log06.bz2 Anonymous, 10/10/2017 07:53 AM
secondary-single_rgw_sync.log00.bz2 (608 KB) secondary-single_rgw_sync.log00.bz2 Anonymous, 10/10/2017 07:54 AM
secondary-single_rgw_sync.log01.bz2 (640 KB) secondary-single_rgw_sync.log01.bz2 Anonymous, 10/10/2017 07:54 AM
secondary-single_rgw_sync.log02.bz2 (585 KB) secondary-single_rgw_sync.log02.bz2 Anonymous, 10/10/2017 07:54 AM
secondary-single_rgw_sync.log03.bz2 (380 KB) secondary-single_rgw_sync.log03.bz2 Anonymous, 10/10/2017 07:55 AM

Related issues 1 (0 open — 1 closed)

Related to rgw - Bug #21772: multisite: multipart uploads fail to syncResolvedCasey Bodley10/12/2017

Actions
Actions
Copy link

Also available in: Atom PDF