Bug #8251
radosgw-agent does not sync objects uploaded to recreated buckets
0%
Description
If a bucket is deleted and one with the same name is recreated, subsequently uploaded objects to this bucket are no longer synced to secondary zones.
This was reported in v067.7 and reproduced in v0.72.2.
Associated revisions
rgw: fix replica log indexing
Fixes: #8251
Previously we were indexing the replica log only by bucket name, even
though we were provided with the bucket instance id. Fix that, and also
add the option to be able to revert to the old behavior. For
radosgw-admin we can do it if --replicalog-index-by-instance=false is
provided. For the replica log REST api we now have the index-by-instance
http param (defaults to true).
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
History
#1 Updated by Ian Colle almost 10 years ago
- Assignee set to Yehuda Sadeh
#2 Updated by Ian Colle almost 10 years ago
- Assignee changed from Yehuda Sadeh to Josh Durgin
#3 Updated by Josh Durgin almost 10 years ago
- Assignee changed from Josh Durgin to Yehuda Sadeh
It appears this is due the replica log api using the bucket name only, not the bucket-instance which is given to it by the client. Note the bucket instance is different:
2014-05-06T12:44:29.224 9331:DEBUG:requests.packages.urllib3.connectionpool:"POST /admin/replica_log?work_bound&marker=00000000004.4.3&daemon_id=radosgw-agent&type=bucket -index&bucket-instance=myfoodata%3Ar0z0.4137.4&time=1970-01-01+00%3A00%3A00 HTTP/1.1" 200 None ... 2014-05-06T12:45:38.751 9391:DEBUG:requests.packages.urllib3.connectionpool:"GET /admin/replica_log?bounds&type=bucket-index&bucket-instance=myfoodata%3Ar0z0.4137.5 HTTP/ 1.1" 200 None 2014-05-06T12:45:38.752 9391:DEBUG:boto:get_worker_bound returned: {u'marker': u'00000000004.4.3', u'oldest_time': u'0.000000', u'markers': [{u'items_in_progress': [], u'position_time': u'0.000000', u'position_marker': u'00000000004.4.3', u'entity': u'radosgw-agent'}]} ... 2014-05-06T12:45:38.757 9391:DEBUG:requests.packages.urllib3.connectionpool:"GET /admin/log?marker=00000000004.4.3&type=bucket-index&bucket-instance=myfoodata%3Ar0z0.4137.5&max-entries=10 HTTP/1.1" 200 None 2014-05-06T12:45:38.768 9391:DEBUG:radosgw_agent.worker:bucket instance "myfoodata:r0z0.4137.5" has 0 entries after "00000000004.4.3"`
radosgw-admin bilog list confirms the log entries are there:
2014-05-06 14:11:22.020572 7fa563258780 10 moving .region0.r0z0.domain_root+.bucket.meta.myfoodata:r0z0.4137.5 to cache LRU end [ { "op_id": "00000000001.1.2", "op_tag": "r0z0.4137.1074", "op": "write", "object": "tiny_file", "state": "pending", "index_ver": 1, "timestamp": "0.000000", "ver": { "pool": -1, "epoch": 0}}, { "op_id": "00000000002.2.3", "op_tag": "r0z0.4137.1074", "op": "write", "object": "tiny_file", "state": "complete", "index_ver": 2, "timestamp": "2014-05-06 19:44:32.000000Z", "ver": { "pool": 7, "epoch": 1}}]
This can be reproduced by using the wip-rgw-8251 branch of teuthology with the following configuration:
interactive-on-error: true roles: - [mon.a, osd.0, osd.1, client.0, client.1] tasks: - chef: - install: branch: firefly - ceph: conf: client: debug ms: 1 debug rgw: 30 rgw gc obj min wait: 15 rgw data log window: 30 client.0: rgw region: region0 rgw zone: r0z0 rgw region root pool: .rgw.region.0 rgw zone root pool: .rgw.zone.0 rgw gc pool: .rgw.gc.0 rgw user uid pool: .users.uid.0 rgw user keys pool: .users.0 rgw log data: True rgw log meta: True client.1: rgw region: region0 rgw zone: r0z1 rgw region root pool: .rgw.region.0 rgw zone root pool: .rgw.zone.1 rgw gc pool: .rgw.gc.1 rgw user uid pool: .users.uid.1 rgw user keys pool: .users.1 rgw log data: False rgw log meta: False - rgw: regions: region0: api name: api1 is master: True master zone: r0z0 zones: [r0z0, r0z1] client.0: system user: name: client0-system-user access key: 0te6NH5mcdcq0Tc5i8i2 secret key: Oy4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv client.1: system user: name: client1-system-user access key: 1te6NH5mcdcq0Tc5i8i3 secret key: Py4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXw - radosgw-agent: client.0: max-entries: 10 src: client.0 dest: client.1 - radosgw-admin:
#4 Updated by Yehuda Sadeh almost 10 years ago
- Status changed from New to Fix Under Review
- Assignee changed from Yehuda Sadeh to Josh Durgin
#5 Updated by Sage Weil over 9 years ago
closed and obsolete : https://github.com/ceph/ceph/pull/2765
#6 Updated by Yehuda Sadeh over 9 years ago
- Backport set to giant, firefly
#7 Updated by Sage Weil about 9 years ago
- Assignee changed from Josh Durgin to Yehuda Sadeh
#8 Updated by Loïc Dachary about 9 years ago
- Backport changed from giant, firefly to firefly
giant is end of life
#9 Updated by Yehuda Sadeh almost 8 years ago
- Status changed from Fix Under Review to Closed