Project

General

Profile

Actions

Bug #48037

closed

Unnecessary bilogs are left in sync-disabled buckets

Added by Ilsoo Byun over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
multisite
Backport:
octopus,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

There is an issue where bilogs are left in the sync-disabled buckets in the multi-site sync-enabled cluster. In our cluster, these logs were accumulated for a long time and caused a performance problem.

This phenomenon can be reproduced through the following procedure:
First off, create multi-site cluster(c1,c2) using rgw/test-rgw-multisite.sh script. And then execute the below commands.

$ yum install s3cmd -y
$ cat >> c1-s3cmd << EOF
[default]
access_key = 1234567890
secret_key = pencil
host_base = 127.0.0.1:8001
host_bucket = 127.0.0.1:8001
use_https = False
EOF
$ s3cmd -c c1-s3cmd mb s3://mytest1
$ ../src/mrun c1 radosgw-admin bucket sync disable --bucket mytest1
$ s3cmd  -c c1-s3cmd put obj s3://mytest1
$ ../src/mrun c1 radosgw-admin bilog list --bucket mytest1 | grep -A1 write

    (Nothing left)

$ s3cmd -c c1-s3cmd mb s3://mytest2
$ ../src/mrun c1 radosgw-admin bucket sync disable --bucket mytest2
$ ../src/mrun c1 radosgw-admin bucket check --fix --bucket mytest2
$ s3cmd  -c c1-s3cmd put obj s3://mytest2
$ ../src/mrun c1 radosgw-admin bilog list --bucket mytest2 | grep -A1 write
        "op": "write",
        "object": "obj",

    (a bilog is left in the sync-disabled bucket)


Related issues 2 (0 open2 closed)

Copied to rgw - Backport #48127: octopus: Unnecessary bilogs are left in sync-disabled bucketsResolvedNathan CutlerActions
Copied to rgw - Backport #48128: nautilus: Unnecessary bilogs are left in sync-disabled bucketsResolvedNathan CutlerActions
Actions #1

Updated by Ilsoo Byun over 3 years ago

Actions #2

Updated by Ilsoo Byun over 3 years ago

The root cause is that when radosgw-admin bucket check --fix commands replace the bucket shard headers, syncstopped flag is missing.

Actions #3

Updated by J. Eric Ivancich over 3 years ago

  • Backport set to octopus,nautilus
Actions #4

Updated by J. Eric Ivancich over 3 years ago

  • Pull request ID set to 37892
Actions #5

Updated by J. Eric Ivancich over 3 years ago

  • Assignee set to Ilsoo Byun
Actions #6

Updated by Casey Bodley over 3 years ago

  • Status changed from New to Fix Under Review
  • Tags set to multisite
Actions #7

Updated by J. Eric Ivancich over 3 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #8

Updated by Nathan Cutler over 3 years ago

  • Copied to Backport #48127: octopus: Unnecessary bilogs are left in sync-disabled buckets added
Actions #9

Updated by Nathan Cutler over 3 years ago

  • Copied to Backport #48128: nautilus: Unnecessary bilogs are left in sync-disabled buckets added
Actions #10

Updated by Nathan Cutler over 3 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions

Also available in: Atom PDF