Ceph » rgw

Hi,
we run a multisite setup where only the metadata get synced, but not the actual data.

I wanted to reshard a bucket (we do this all the time, because a lot of buckets still have 0 shards, and we increase it when necessary), but entered the wrong shard number. I hit `ctrl-c` and tried to cancel with `radosgw-admin reshard cancel --bucket bucket`. After it was canceled I am not able to reshard this bucket anymore.

After some digging it looks like the metadata is broken, and now I fear that I broke the bucket.

Does anybody know what I can do?

Error message:

$ radosgw-admin bucket reshard --bucket bucket --num-shards 11
ERROR: the bucket is currently undergoing resharding and cannot be added to the reshard list at this time

Current reshard status:

$ radosgw-admin reshard status --bucket bucket
[
    {
        "reshard_status": "not-resharding",
        "new_bucket_instance_id": "",
        "num_shards": -1
    }
]

Bucket stats already use new id and marker, and still shows 0 shards

$ radosgw-admin bucket stats --bucket bucket
{
    "bucket": "bucket",
    "num_shards": 0,
...
    "id": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2296333939.14",
    "marker": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2296333939.14",
...
}

Metadata with old id show 211 shards and a different marker and bucket id

$ radosgw-admin metadata get bucket.instance:bucket:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2368407345.1
{
    "key": "bucket.instance:bucket:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2368407345.1",
    "ver": {
        "tag": "QndcbsKPFDjs6rYKKDHde9bM",
        "ver": 2
    },
    "mtime": "2022-10-07T07:16:49.231685Z",
    "data": {
        "bucket_info": {
            "bucket": {
                "name": "bucket",
                "marker": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2296333939.14",
                "bucket_id": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2368407345.1",
...
            },
...
            "num_shards": 211,
...
        },
}

There is no metadata with the new bucket id.

$ radosgw-admin metadata get bucket.instance:bucket:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2296333939.14
ERROR: can't get key: (2) No such file or directory

Cheers
Boris