Project

General

Profile

Bug #8251

radosgw-agent does not sync objects uploaded to recreated buckets

Added by Brian Andrus almost 10 years ago. Updated almost 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Support
Tags:
Backport:
firefly
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

If a bucket is deleted and one with the same name is recreated, subsequently uploaded objects to this bucket are no longer synced to secondary zones.

This was reported in v067.7 and reproduced in v0.72.2.

Associated revisions

Revision e274e109 (diff)
Added by Yehuda Sadeh about 9 years ago

rgw: fix replica log indexing

Fixes: #8251

Previously we were indexing the replica log only by bucket name, even
though we were provided with the bucket instance id. Fix that, and also
add the option to be able to revert to the old behavior. For
radosgw-admin we can do it if --replicalog-index-by-instance=false is
provided. For the replica log REST api we now have the index-by-instance
http param (defaults to true).

Signed-off-by: Yehuda Sadeh <>

History

#1 Updated by Ian Colle almost 10 years ago

  • Assignee set to Yehuda Sadeh

#2 Updated by Ian Colle almost 10 years ago

  • Assignee changed from Yehuda Sadeh to Josh Durgin

#3 Updated by Josh Durgin almost 10 years ago

  • Assignee changed from Josh Durgin to Yehuda Sadeh

It appears this is due the replica log api using the bucket name only, not the bucket-instance which is given to it by the client. Note the bucket instance is different:

2014-05-06T12:44:29.224 9331:DEBUG:requests.packages.urllib3.connectionpool:"POST /admin/replica_log?work_bound&marker=00000000004.4.3&daemon_id=radosgw-agent&type=bucket
-index&bucket-instance=myfoodata%3Ar0z0.4137.4&time=1970-01-01+00%3A00%3A00 HTTP/1.1" 200 None
...
2014-05-06T12:45:38.751 9391:DEBUG:requests.packages.urllib3.connectionpool:"GET /admin/replica_log?bounds&type=bucket-index&bucket-instance=myfoodata%3Ar0z0.4137.5 HTTP/
1.1" 200 None
2014-05-06T12:45:38.752 9391:DEBUG:boto:get_worker_bound returned: {u'marker': u'00000000004.4.3', u'oldest_time': u'0.000000', u'markers': [{u'items_in_progress': [], u'position_time': u'0.000000', u'position_marker': u'00000000004.4.3', u'entity': u'radosgw-agent'}]}
...
2014-05-06T12:45:38.757 9391:DEBUG:requests.packages.urllib3.connectionpool:"GET /admin/log?marker=00000000004.4.3&type=bucket-index&bucket-instance=myfoodata%3Ar0z0.4137.5&max-entries=10 HTTP/1.1" 200 None
2014-05-06T12:45:38.768 9391:DEBUG:radosgw_agent.worker:bucket instance "myfoodata:r0z0.4137.5" has 0 entries after "00000000004.4.3" 
`

radosgw-admin bilog list confirms the log entries are there:

2014-05-06 14:11:22.020572 7fa563258780 10 moving .region0.r0z0.domain_root+.bucket.meta.myfoodata:r0z0.4137.5 to cache LRU end
[
    { "op_id": "00000000001.1.2",
      "op_tag": "r0z0.4137.1074",
      "op": "write",
      "object": "tiny_file",
      "state": "pending",
      "index_ver": 1,
      "timestamp": "0.000000",
      "ver": { "pool": -1,
          "epoch": 0}},
    { "op_id": "00000000002.2.3",
      "op_tag": "r0z0.4137.1074",
      "op": "write",
      "object": "tiny_file",
      "state": "complete",
      "index_ver": 2,
      "timestamp": "2014-05-06 19:44:32.000000Z",
      "ver": { "pool": 7,
          "epoch": 1}}]

This can be reproduced by using the wip-rgw-8251 branch of teuthology with the following configuration:

interactive-on-error: true
roles:
- [mon.a, osd.0, osd.1, client.0, client.1]
tasks:
- chef:
- install:
    branch: firefly
- ceph:
    conf:
      client:
        debug ms: 1
        debug rgw: 30
        rgw gc obj min wait: 15
        rgw data log window: 30
      client.0:
        rgw region: region0
        rgw zone: r0z0
        rgw region root pool: .rgw.region.0
        rgw zone root pool: .rgw.zone.0
        rgw gc pool: .rgw.gc.0
        rgw user uid pool: .users.uid.0
        rgw user keys pool: .users.0
        rgw log data: True
        rgw log meta: True
      client.1:
        rgw region: region0
        rgw zone: r0z1
        rgw region root pool: .rgw.region.0
        rgw zone root pool: .rgw.zone.1
        rgw gc pool: .rgw.gc.1
        rgw user uid pool: .users.uid.1
        rgw user keys pool: .users.1
        rgw log data: False
        rgw log meta: False
- rgw:
    regions:
      region0:
        api name: api1
        is master: True
        master zone: r0z0
        zones: [r0z0, r0z1]
    client.0:
      system user:
        name: client0-system-user
        access key: 0te6NH5mcdcq0Tc5i8i2
        secret key: Oy4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv
    client.1:
      system user:
        name: client1-system-user
        access key: 1te6NH5mcdcq0Tc5i8i3
        secret key: Py4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXw
- radosgw-agent:
    client.0:
      max-entries: 10
      src: client.0
      dest: client.1
- radosgw-admin:

#4 Updated by Yehuda Sadeh almost 10 years ago

  • Status changed from New to Fix Under Review
  • Assignee changed from Yehuda Sadeh to Josh Durgin

#5 Updated by Sage Weil over 9 years ago

#6 Updated by Yehuda Sadeh over 9 years ago

  • Backport set to giant, firefly

#7 Updated by Sage Weil about 9 years ago

  • Assignee changed from Josh Durgin to Yehuda Sadeh

#8 Updated by Loïc Dachary about 9 years ago

  • Backport changed from giant, firefly to firefly

giant is end of life

#9 Updated by Yehuda Sadeh almost 8 years ago

  • Status changed from Fix Under Review to Closed

Also available in: Atom PDF