Bug #43339: bucket sync init command failed - rgw - Ceph

Actions

Copy link

Bug #43339

closed

bucket sync init command failed

Added by Chang Liu over 4 years ago. Updated about 4 years ago.

Status:

Can't reproduce

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

[root@VM_71_139_centos build]# ./bin/radosgw-admin bucket sync init --bucket=test
ERROR: source zone not specified
[root@VM_71_139_centos build]# ./bin/radosgw-admin bucket sync init --bucket=test --source-zone=us-east
019-12-17T04:42:57.646+0000 7f86bbb0eec0  0 bucket sync zone:bb1d9f59 bucket:test connection object to zone bb1d9f59-4282-4576-8bbd-1b7a7adeb045 does not exist
ERROR: sync.init() returned ret=-22
[root@VM_71_139_centos build]# ./bin/radosgw-admin bucket sync init --bucket=test --source-zone=us-east --rgw-zone=es
could not get bucket info for bucket=test
ERROR: could not init bucket: (2) No such file or directory

bucket is created in sourze zone. rgw couldn't get bucket info in elastic search zone. here is a conflict in `rgw-zone` param.

Actions

Copy link

Updated by Chang Liu over 4 years ago

elastic search zone does not sync metadate, so all those `bucket sync *` commands are not able to used in elastic zone ?

Actions

Copy link

Updated by Chang Liu over 4 years ago

Chang Liu wrote:

elastic search zone does not sync metadate, so all those `bucket sync *` commands are not able to used in elastic zone ?

I found that bucket info will be synced in here https://github.com/ceph/ceph/blob/0f0ae55a0d1dd92612dab81f1ed0cc0ded703bbe/src/rgw/rgw_data_sync.cc#L3309-L3315

Actions

Copy link

Updated by Chang Liu over 4 years ago

I add some debug log, it shows that sync module could get bucket instance info. but radosgw-admin could not . as following:

2019-12-17T09:49:45.490+0000 7f82004c3ec0  0 RGW-SYNC:data:sync:shard[69]:entry[test:bb1d9f59-4282-4576-8bbd-1b7a7adeb045.4578.1]:bucket[test:bb1d9f59-4282-4576-8bbd-1b7a7adeb045.4578.1]: debuglc: get bucket instance info: 0
2019-12-17T09:49:45.498+0000 7f82004c3ec0  0 RGW-SYNC:data:sync:shard[120]:entry[test2:bb1d9f59-4282-4576-8bbd-1b7a7adeb045.4578.2]:bucket[test2:bb1d9f59-4282-4576-8bbd-1b7a7adeb045.4578.2]: debuglc: get bucket instance info: 0
2019-12-17T09:49:45.670+0000 7f82004c3ec0  0 RGW-SYNC:data:sync:shard[120]:entry[test2:bb1d9f59-4282-4576-8bbd-1b7a7adeb045.4578.2]:bucket[test2:bb1d9f59-4282-4576-8bbd-1b7a7adeb045.4578.2]: debuglc: get bucket instance info: 0
[root@VM_71_139_centos build]# ./bin/radosgw-admin bucket sync status --bucket=test --source-zone=us-east --rgw-zone=es
could not get bucket info for bucket=test
[root@VM_71_139_centos build]# ./bin/radosgw-admin bucket sync status --bucket=test2 --source-zone=us-east --rgw-zone=es
could not get bucket info for bucket=test2


    yield call(new RGWGetBucketInstanceInfoCR(sync_env->async_rados, sync_env->store, bs.bucket, &bucket_info));
    tn->log(0, SSTR("debuglc: get bucket instance info: " << retcode));   <<-- add logging here
    if (retcode == -ENOENT) {
      /* bucket instance info has not been synced in yet, fetch it now */

Actions

Copy link

Updated by Chang Liu over 4 years ago

2019-12-17T11:41:03.306+0000 7f593cc54ec0 20 get_system_obj_state: rctx=0x7ffc6cc59fd0 obj=es.rgw.meta:root:test2 state=0x7f593ef0cf80 s->prefetch_data=0
could not get bucket info for bucket=test2

[root@VM_71_139_centos build]# ./bin/rados ls -p es.rgw.meta --all 2>/dev/null
root    .bucket.meta.test:bb1d9f59-4282-4576-8bbd-1b7a7adeb045.4578.1
root    .bucket.meta.test2:bb1d9f59-4282-4576-8bbd-1b7a7adeb045.4578.2
[root@VM_71_139_centos build]# ./bin/rados ls -p us-east.rgw.meta --all 2>/dev/null
root    .bucket.meta.test:bb1d9f59-4282-4576-8bbd-1b7a7adeb045.4578.1
users.keys    BXAQKB0F10U19M903MND
users.keys    KBZYKLMUIG00XV6SFEO1
users.uid    test
users.uid    test2.buckets
users.keys    0555b35654ad1656d804
root    test
users.uid    test2
root    .bucket.meta.test2:bb1d9f59-4282-4576-8bbd-1b7a7adeb045.4578.2
root    test2
users.uid    synchronization-user

Actions

Copy link

Updated by Casey Bodley over 4 years ago

Chang Liu wrote:

Chang Liu wrote:

elastic search zone does not sync metadate, so all those `bucket sync *` commands are not able to used in elastic zone ?

I found that bucket info will be synced in here https://github.com/ceph/ceph/blob/0f0ae55a0d1dd92612dab81f1ed0cc0ded703bbe/src/rgw/rgw_data_sync.cc#L3309-L3315

data sync is only fetching the bucket instance object, and not its bucket entrypoint

commands like 'bucket sync init' that take --bucket have to look up its bucket entrypoint object by name. the bucket entrypoint stores the current bucket instance id, so we can use that to find the right bucket instance object. if metadata sync hasn't fetched the bucket entrypoint object yet, commands with --bucket will fail this way

Actions

Copy link