Project

General

Profile

Actions

Bug #51595

closed

Incremental sync fails to complete post reshard on a bucket ownership changed.

Added by Tejas C almost 3 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
multisite-reshard
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ceph version 17.0.0-5278-g79eb0c85 (79eb0c853ca1ee491410e0c6c6796675a7449ef9) quincy (dev)

Steps:
1. Created a bucket with 50 objects .
2. changed the bucket ownership and created 1000 more objects. Sync worked as expected til here.
3. did a reshard from primary, and created 150 more objects. Sync fails here.

Primary:
/]# radosgw-admin bucket sync checkpoint --bucket chusr1
2021-07-08T11:54:41.083+0000 7f7b99ba5340 1 bucket sync caught up with source:
local status: [00000000114.256.5, 00000000107.231.5, 00000000112.316.5, 00000000108.233.5, 00000000105.227.5, 00000000111.309.5, 00000000114.245.5, 00000000103.370015.5, 00000000117.39200.5, 00000000103.474.5, 00000000112.241.5, , , , , , , , , , , , , , , , , , , , , , , , , , ]
remote markers: [00000000114.256.5, 00000000107.231.5, 00000000112.316.5, 00000000108.233.5, 00000000105.227.5, 00000000111.309.5, 00000000114.245.5, 00000000103.370015.5, 00000000117.39200.5, 00000000103.474.5, 00000000112.241.5]
2021-07-08T11:54:41.083+0000 7f7b99ba5340 0 bucket checkpoint complete
[ceph: root@clara001 /]#
[ceph: root@clara001 /]# radosgw-admin bucket stats --bucket chusr1 {
"bucket": "chusr1",
"num_shards": 37,
"tenant": "",
"zonegroup": "1785a4fa-f7d6-4081-8c72-74f9cc441d3a",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "b76458b4-55bc-41b4-8610-b4aa5df49661.36014.3",
"marker": "b76458b4-55bc-41b4-8610-b4aa5df49661.36014.3",
"index_type": "Normal",
"owner": "test2",
"ver": "0#5,1#5,2#6,3#6,4#6,5#8,6#6,7#6,8#5,9#6,10#7,11#6,12#7,13#6,14#5,15#6,16#6,17#6,18#5,19#7,20#7,21#5,22#7,23#6,24#5,25#6,26#5,27#6,28#6,29#5,30#7,31#5,32#6,33#7,34#6,35#7,36#8",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0",
"mtime": "0.000000",
"creation_time": "2021-07-08T11:21:41.700523Z",
"max_marker": "0#00000000004.15799.5,1#00000000004.28.5,2#00000000005.38782.5,3#00000000005.210.5,4#00000000005.30.5,5#00000000007.24.5,6#00000000005.20.5,7#00000000005.20.5,8#00000000004.480.5,9#00000000005.262.5,10#00000000006.218.5,11#00000000005.20.5,12#00000000006.295.5,13#00000000005.20.5,14#00000000004.18.5,15#00000000005.18060.5,16#00000000005.261.5,17#00000000005.38887.5,18#00000000004.31.5,19#00000000006.18407.5,20#00000000006.252.5,21#00000000004.38760.5,22#00000000006.32.5,23#00000000005.20.5,24#00000000004.5305.5,25#00000000005.38784.5,26#00000000004.18.5,27#00000000005.15925.5,28#00000000005.290.5,29#00000000004.16351.5,30#00000000006.237.5,31#00000000004.38786.5,32#00000000005.369563.5,33#00000000006.246.5,34#00000000005.20.5,35#00000000006.34.5,36#00000000007.24.5",
"usage": {
"rgw.main": {
"size": 1258291200,
"size_actual": 1258291200,
"size_utilized": 1258291200,
"size_kb": 1228800,
"size_kb_actual": 1228800,
"size_kb_utilized": 1228800,
"num_objects": 1200
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}

Sec:
/]# radosgw-admin bucket sync checkpoint --bucket chusr1
2021-07-08T11:54:50.696+0000 7f4cc9bb9340 1 bucket sync caught up with source:
local status: [00000000107.507.5, 00000000096.205.5, 00000000104.470.5, 00000000101.215.5, 00000000093.16042.5, 00000000101.224.5, 00000000103.230.5, 00000000095.370212.5, 00000000104.39167.5, 00000000093.199.5, 00000000103.218.5]
remote markers: [00000000004.15799.5, 00000000004.28.5, 00000000005.38782.5, 00000000005.210.5, 00000000005.30.5, 00000000007.24.5, 00000000005.20.5, 00000000005.20.5, 00000000004.480.5, 00000000005.262.5, 00000000006.218.5, 00000000005.20.5, 00000000006.295.5, 00000000005.20.5, 00000000004.18.5, 00000000005.18060.5, 00000000005.261.5, 00000000005.38887.5, 00000000004.31.5, 00000000006.18407.5, 00000000006.252.5, 00000000004.38760.5, 00000000006.32.5, 00000000005.20.5, 00000000004.5305.5, 00000000005.38784.5, 00000000004.18.5, 00000000005.15925.5, 00000000005.290.5, 00000000004.16351.5, 00000000006.237.5, 00000000004.38786.5, 00000000005.369563.5, 00000000006.246.5, 00000000005.20.5, 00000000006.34.5, 00000000007.24.5]
2021-07-08T11:54:50.696+0000 7f4cc9bb9340 0 bucket checkpoint complete
[ceph: root@magna017 /]#
[ceph: root@magna017 /]# radosgw-admin bucket stats --bucket chusr1 {
"bucket": "chusr1",
"num_shards": 11,
"tenant": "",
"zonegroup": "1785a4fa-f7d6-4081-8c72-74f9cc441d3a",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "b76458b4-55bc-41b4-8610-b4aa5df49661.36014.3",
"marker": "b76458b4-55bc-41b4-8610-b4aa5df49661.36014.3",
"index_type": "Normal",
"owner": "test2",
"ver": "0#115,1#108,2#113,3#109,4#106,5#112,6#115,7#104,8#118,9#104,10#113",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
"mtime": "0.000000",
"creation_time": "2021-07-08T11:21:41.700523Z",
"max_marker": "0#00000000114.256.5,1#00000000107.231.5,2#00000000112.316.5,3#00000000108.233.5,4#00000000105.227.5,5#00000000111.309.5,6#00000000114.245.5,7#00000000103.370015.5,8#00000000117.39200.5,9#00000000103.474.5,10#00000000112.241.5",
"usage": {
"rgw.main": {
"size": 1212153856,
"size_actual": 1212153856,
"size_utilized": 1212153856,
"size_kb": 1183744,
"size_kb_actual": 1183744,
"size_kb_utilized": 1183744,
"num_objects": 1156 <--------------------------
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}

Actions #1

Updated by Casey Bodley almost 3 years ago

  • Tags set to multisite-reshard
Actions #2

Updated by Casey Bodley over 2 years ago

  • Status changed from New to Closed
Actions

Also available in: Atom PDF