Update on the issue after 2 days:
1- On the primary site, the buckets have been resharded dynamically.
2- On the secondary site, however, looks like the dynamic reshard process is stuck and the 'radosgw-admin reshard list' lists out the buckets that are in for resharding.
a snippet of radosgw-admin reshard list on the secondary:
==========================================================
[ceph: root@magna017 /]# radosgw-admin reshard list
2021-10-11T11:10:28.281+0000 7f20f637a340 1 Realm: data (d62bd711-d486-47be-9c3e-193e49334862)
2021-10-11T11:10:28.281+0000 7f20f637a340 1 ZoneGroup: us (8f3b29b1-ffc6-4c90-9d0c-9bd258028cd8)
2021-10-11T11:10:28.281+0000 7f20f637a340 1 Zone: west (3a571642-9f5e-46d8-8186-9eca8cc79ac6)
2021-10-11T11:10:28.281+0000 7f20f637a340 1 using period configuration: dd132fae-4457-4f49-88b9-55ca2f8adff9:2
[2021-10-11T11:10:28.532+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000000 marker= (2) No such file or directory
2021-10-11T11:10:28.533+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000001 marker= (2) No such file or directory
2021-10-11T11:10:28.533+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000002 marker= (2) No such file or directory
{
"time": "2021-10-10T05:22:11.644198Z",
"tenant": "tx",
"bucket_name": "ms-bkt-v1",
"bucket_id": "5d32949e-6245-422c-b315-9048855d3a9a.28133.1",
"old_num_shards": 11,
"new_num_shards": 43
}2021-10-11T11:10:28.536+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000004 marker= (2) No such file or directory
2021-10-11T11:10:28.538+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000005 marker= (2) No such file or directory
2021-10-11T11:10:28.538+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000006 marker= (2) No such file or directory
2021-10-11T11:10:28.538+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000007 marker= (2) No such file or directory
2021-10-11T11:10:28.539+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000008 marker= (2) No such file or directory
2021-10-11T11:10:28.539+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000009 marker= (2) No such file or directory
2021-10-11T11:10:28.539+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000010 marker= (2) No such file or directory
2021-10-11T11:10:28.540+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000011 marker= (2) No such file or directory
,
{
"time": "2021-10-10T05:22:11.813771Z",
"tenant": "",
"bucket_name": "elizabethd.615-bucky-488-0",
"bucket_id": "5d32949e-6245-422c-b315-9048855d3a9a.28133.3",
"old_num_shards": 11,
"new_num_shards": 227
},
{
"time": "2021-10-10T05:22:11.678128Z",
"tenant": "tx",
"bucket_name": "ms-bkt-v2",
"bucket_id": "5d32949e-6245-422c-b315-9048855d3a9a.28841.1",
"old_num_shards": 11,
"new_num_shards": 163
},
{
"time": "2021-10-10T05:22:11.711405Z",
"tenant": "tx",
"bucket_name": "ss-bkt-v1",
"bucket_id": "5d32949e-6245-422c-b315-9048855d3a9a.26963.1",
"old_num_shards": 11,
"new_num_shards": 59
},
{
"time": "2021-10-10T05:22:11.761627Z",
"tenant": "tx",
"bucket_name": "ss-bkt-v3",
"bucket_id": "5d32949e-6245-422c-b315-9048855d3a9a.26963.3",
"old_num_shards": 11,
"new_num_shards": 53
}2021-10-11T11:10:28.541+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000014 marker= (2) No such file or directory
]
2021-10-11T11:10:28.541+0000 7f20f637a340 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000015 marker= (2) No such file or directory
[ceph: root@magna017 /]#