Project

General

Profile

Bug #39551

multisite: RGWListBucketIndexesCR for data full sync needs pagination

Added by Casey Bodley 5 months ago. Updated 3 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Target version:
-
Start date:
04/30/2019
Due date:
% Done:

0%

Source:
Tags:
multisite admin-api
Backport:
luminous mimic nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

When a new peer zone is added, rgw starts in 'full sync'. This starts by listing all of that zone's bucket instances, in order to build a list of all of the bucket index shards we need to sync.

from RGWListBucketIndexesCR::operate():

        /* FIXME: need a better scaling solution here, requires streaming output */
        call(new RGWReadRESTResourceCR<list<string> >(store->ctx(), sync_env->conn, sync_env->http_manager,
                                                      entrypoint, NULL, &result));

This attempts to read the entire list of bucket instance ids from the remote zone's "/admin/metadata/bucket.instance" admin api. RGWOp_Metadata_List is the op for this admin api, and it will return a maximum of 1000 entries - so if there are more bucket instances than that, data sync won't see them. RGWOp_Metadata_List accepts a "marker" parameter that allows you to resume a listing from where the last request left off. RGWListBucketIndexesCR should use this marker in order to continue listing keys until the end (where truncated = false).


Related issues

Copied to rgw - Backport #40352: nautilus: multisite: RGWListBucketIndexesCR for data full sync needs pagination Resolved
Copied to rgw - Backport #40353: luminous: multisite: RGWListBucketIndexesCR for data full sync needs pagination New
Copied to rgw - Backport #40354: mimic: multisite: RGWListBucketIndexesCR for data full sync needs pagination Resolved

History

#1 Updated by Casey Bodley 4 months ago

  • Status changed from New to In Progress
  • Assignee set to Shilpa MJ

#2 Updated by Casey Bodley 4 months ago

  • Status changed from In Progress to Need Review
  • Backport set to luminous mimic nautilus
  • Pull request ID set to 28146

#3 Updated by Casey Bodley 3 months ago

  • Status changed from Need Review to Pending Backport

#4 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #40352: nautilus: multisite: RGWListBucketIndexesCR for data full sync needs pagination added

#5 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #40353: luminous: multisite: RGWListBucketIndexesCR for data full sync needs pagination added

#6 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #40354: mimic: multisite: RGWListBucketIndexesCR for data full sync needs pagination added

Also available in: Atom PDF