Project

General

Profile

Bug #41230

multisite: better spread multisite sync load over cooperating gateways

Added by Casey Bodley over 4 years ago. Updated 11 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
multisite backport_processed
Backport:
reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

each radosgw tries to lock every shard of each multisite log for processing, and can hold the locks for a long time. this generally results in one gateway hogging most of the shards


Related issues

Copied to rgw - Backport #61172: reef: multisite: better spread multisite sync load over cooperating gateways In Progress

History

#1 Updated by Casey Bodley over 1 year ago

  • Assignee changed from J. Eric Ivancich to Or Friedmann
  • Pull request ID changed from 28119 to 45958

#2 Updated by Christian Rohmann over 1 year ago

Casey Bodley wrote:

each radosgw tries to lock every shard of each multisite log for processing, and can hold the locks for a long time. this generally results in one gateway hogging most of the shards

We use this exact setup with 3 gateways on each end and with multiple endpoints explicitly configured in the zone / zonegroup.
Apart from proper "cooperation" there seems it seems that a single non-responsive RGW causes the whole replication to suffer massive slowdowns and calls of radosgw-admin for "sync status" to time out.

#3 Updated by Casey Bodley 11 months ago

  • Status changed from In Progress to Fix Under Review
  • Assignee changed from Or Friedmann to Shilpa MJ

#4 Updated by Casey Bodley 11 months ago

metadata sync fairness merged in https://github.com/ceph/ceph/pull/45958

@Shilpa, should we resolve this issue and start a new one for data sync, or leave this one open to track data sync? do you want to backport meta sync fairness to reef now, or wait for data sync too?

#5 Updated by Shilpa MJ 11 months ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to reef

Casey Bodley wrote:

metadata sync fairness merged in https://github.com/ceph/ceph/pull/45958

@Shilpa, should we resolve this issue and start a new one for data sync, or leave this one open to track data sync? do you want to backport meta sync fairness to reef now, or wait for data sync too?

I have opened a new tracker for data sync. https://tracker.ceph.com/issues/61171
And I could backport this one first.

#6 Updated by Backport Bot 11 months ago

  • Copied to Backport #61172: reef: multisite: better spread multisite sync load over cooperating gateways added

#7 Updated by Backport Bot 11 months ago

  • Tags changed from multisite to multisite backport_processed

Also available in: Atom PDF