Ceph » rgw

https://github.com/ceph/ceph/pull/15545

I don't understand what step (4) is. What do you mean fail bucket index add op.

Yehuda Sadeh wrote:

I don't understand what step (4) is. What do you mean fail bucket index add op.

1. rados cluster abnormally lead bucket index add op failed in fetch remote object, but the object's data was put successfully

2. because step 1, that object can be download but will not be in s3 ListBucket results

3. That object will never be in s3 ListBucket results if it is not modified later, even we triger a new full sync

Surely, that object will be in s3 ListBucket if it is modified later.

Yehuda Sadeh wrote:

I don't understand what step (4) is. What do you mean fail bucket index add op.

I consider that rados cluster abnormal is very usual. Losts of objects will be "half syncing" state(they have whole data but don't have index), and they will stay same if not be modified in source site even we triger a new full sync using radosgw-admin, why? let's have a look at the code:

if (copy_if_newer) {  `always 'True' in syncing object`
    /* need to get mtime for destination */
    ret = get_obj_state(&obj_ctx, dest_obj, &dest_state, false); `will success for object data is whole`
    if (ret < 0)
      goto set_err_state;

if (!real_clock::is_zero(dest_state->mtime)) {
      dest_mtime_weight.init(dest_state);
      pmod = &dest_mtime_weight.mtime;
    }
  }

ret = conn->get_obj(user_id, info, src_obj, pmod, unmod_ptr,
                      dest_mtime_weight.zone_short_id, dest_mtime_weight.pg_ver,
                      true /* prepend_meta /, true / GET /, false / rgwx-stat */,
                      &cb, &in_stream_req);
  if (ret < 0) { 
`will fail for object is not modified in source site and we just copy if newer, will get ERR_NOT_MODIFIED`
    goto set_err_state;
}

In our environment, radosgw-admin bucket stats show that slave site only has half the objects of master site(nearlly 50,000 objects look lost) after rados cluster in slave site abnormal, which make me panic for that I retriger a full sync does't work.

Then I just do ListBucket on both site, and pick one object in master site but not in slave site.
I found that object can be downloaded from slave site event it's not in ListBucket results of slave site.

so my fix is to make full sync work after things happen.

fang yuxiang wrote:

Yehuda Sadeh wrote:

I don't understand what step (4) is. What do you mean fail bucket index add op.

1. rados cluster abnormally lead bucket index add op failed in fetch remote object, but the object's data was put successfully

Can you tell what errors the bucket index op was returning?

Either way, I tend to agree that the copy_if_newer check should be based on the bucket index's timestamp, if at all possible.

Casey Bodley wrote:

fang yuxiang wrote:

Yehuda Sadeh wrote:

I don't understand what step (4) is. What do you mean fail bucket index add op.

1. rados cluster abnormally lead bucket index add op failed in fetch remote object, but the object's data was put successfully

Can you tell what errors the bucket index op was returning?

Either way, I tend to agree that the copy_if_newer check should be based on the bucket index's timestamp, if at all possible.

110 ETIMEOUT

Casey Bodley wrote:

fang yuxiang wrote:

Yehuda Sadeh wrote:

I don't understand what step (4) is. What do you mean fail bucket index add op.

1. rados cluster abnormally lead bucket index add op failed in fetch remote object, but the object's data was put successfully

Can you tell what errors the bucket index op was returning?

Either way, I tend to agree that the copy_if_newer check should be based on the bucket index's timestamp, if at all possible.

consider that bucket index add op is two phase(prepare and complete), we just know the return code is -110 from do_compete