Project

General

Profile

Actions
Copy link

Bug #51427

closed

Multisite sync stuck if reshard is done while bucket sync is disabled

Added by Tejas C almost 3 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
Ceph - v17.0.0
% Done:

0%

Source:
Q/A
Tags:
multisite-reshard
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v17.0.0
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ceph version 17.0.0-5278-g79eb0c85 (79eb0c853ca1ee491410e0c6c6796675a7449ef9) quincy (dev)

Steps:
- Bucket has 13k objects synced on either site , and is idle.
- Do a bucket sync disable on buck1, on primary.
- Write 2k objects on secondary, and do a reshard on either site.
- Enable bucket sync again, sync fails to catch up.

Primary:
/]# radosgw-admin bucket sync enable --bucket buck1

/]# radosgw-admin bucket sync status --bucket buck1
realm 6d34d4f8-671c-4cf0-a6b7-48e1fa21fde7 (india)
zonegroup 1785a4fa-f7d6-4081-8c72-74f9cc441d3a (south)
zone b76458b4-55bc-41b4-8610-b4aa5df49661 (blr)
bucket :buck1[b76458b4-55bc-41b4-8610-b4aa5df49661.34772.1]) {
"bucket": "buck1",
"num_shards": 111,
"tenant": "",
"zonegroup": "1785a4fa-f7d6-4081-8c72-74f9cc441d3a",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "b76458b4-55bc-41b4-8610-b4aa5df49661.34772.1",
"marker": "b76458b4-55bc-41b4-8610-b4aa5df49661.34772.1",
"index_type": "Normal",
"owner": "test1",
"ver": "0#5,1#5,2#5,3#5,4#5,5#5,6#5,7#5,8#5,9#5,10#5,11#5,12#5,13#5,14#5,15#5,16#5,17#5,18#5,19#5,20#5,21#5,22#5,23#5,24#5,25#5,26#5,27#5,28#5,29#5,30#6,31#5,32#5,33#5,34#5,35#5,36#5,37#5,38#5,39#5,40#5,41#5,42#5,43#5,44#5,45#5,46#5,47#5,48#5,49#5,50#5,51#5,52#6,53#5,54#5,55#5,56#5,57#5,58#5,59#5,60#5,61#5,62#5,63#5,64#5,65#5,66#5,67#5,68#5,69#5,70#5,71#5,72#5,73#5,74#5,75#5,76#5,77#5,78#5,79#5,80#5,81#5,82#5,83#5,84#5,85#5,86#5,87#5,88#5,89#5,90#5,91#5,92#5,93#5,94#5,95#5,96#5,97#5,98#5,99#5,100#5,101#5,102#5,103#5,104#5,105#5,106#5,107#5,108#5,109#5,110#5",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0",
"mtime": "0.000000",
"creation_time": "2021-06-28T07:45:16.289450Z",
"max_marker": "0#00000000004.15.1,1#00000000004.15.1,2#00000000004.19.1,3#00000000004.15.1,4#00000000004.23.1,5#00000000004.15.1,6#00000000004.15.1,7#00000000004.15.1,8#00000000004.15.1,9#00000000004.15.1,10#00000000004.15.1,11#00000000004.15.1,12#00000000004.15.1,13#00000000004.15.1,14#00000000004.19.1,15#00000000004.19.1,16#00000000004.19.1,17#00000000004.19.1,18#00000000004.15.1,19#00000000004.19.1,20#00000000004.15.1,21#00000000004.15.1,22#00000000004.20.1,23#00000000004.15.1,24#00000000004.15.1,25#00000000004.4526.1,26#00000000004.15.1,27#00000000004.15.1,28#00000000004.15.1,29#00000000004.19.1,30#00000000005.16.1,31#00000000004.15.1,32#00000000004.24.1,33#00000000004.15.1,34#00000000004.15.1,35#00000000004.19.1,36#00000000004.19.1,37#00000000004.15.1,38#00000000004.15.1,39#00000000004.19.1,40#00000000004.19.1,41#00000000004.20.1,42#00000000004.15.1,43#00000000004.19.1,44#00000000004.15.1,45#00000000004.15.1,46#00000000004.15.1,47#00000000004.15.1,48#00000000004.15.1,49#00000000004.15.1,50#00000000004.15.1,51#00000000004.4602.1,52#00000000005.16.1,53#00000000004.20.1,54#00000000004.15.1,55#00000000004.20.1,56#00000000004.15.1,57#00000000004.19.1,58#00000000004.15.1,59#00000000004.15.1,60#00000000004.15.1,61#00000000004.4544.1,62#00000000004.15.1,63#00000000004.15.1,64#00000000004.15.1,65#00000000004.15.1,66#00000000004.15.1,67#00000000004.15.1,68#00000000004.15.1,69#00000000004.20.1,70#00000000004.15.1,71#00000000004.20.1,72#00000000004.15.1,73#00000000004.15.1,74#00000000004.15.1,75#00000000004.15.1,76#00000000004.15.1,77#00000000004.15.1,78#00000000004.4545.1,79#00000000004.20.1,80#00000000004.15.1,81#00000000004.4582.1,82#00000000004.20.1,83#00000000004.4546.1,84#00000000004.15.1,85#00000000004.15.1,86#00000000004.15.1,87#00000000004.15.1,88#00000000004.15.1,89#00000000004.19.1,90#00000000004.15.1,91#00000000004.25.1,92#00000000004.15.1,93#00000000004.15.1,94#00000000004.129538.1,95#00000000004.15.1,96#00000000004.20.1,97#00000000004.15.1,98#00000000004.15.1,99#00000000004.15.1,100#00000000004.15.1,101#00000000004.15.1,102#00000000004.15.1,103#00000000004.20.1,104#00000000004.15.1,105#00000000004.20.1,106#00000000004.20.1,107#00000000004.15.1,108#00000000004.20.1,109#00000000004.20.1,110#00000000004.15.1",
"usage": {
"rgw.main": {
"size": 63045928640,
"size_actual": 63053168640,
"size_utilized": 63045928640,
"size_kb": 61568290,
"size_kb_actual": 61575360,
"size_kb_utilized": 61568290,
"num_objects": 13001
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 27,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 1,
"num_objects": 1
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}

/]# radosgw-admin bucket sync checkpoint --bucket buck1
2021-06-29T10:32:57.890+0000 7fd0a6083340  1 waiting to reach latest gen 1, on 0..
2021-06-29T10:32:59.891+0000 7fd0a6083340  1 waiting to reach latest gen 1, on 0..
2021-06-29T10:33:01.892+0000 7fd0a6083340  1 waiting to reach latest gen 1, on 0..
2021-06-29T10:33:03.893+0000 7fd0a6083340  1 waiting to reach latest gen 1, on 0..
2021-06-29T10:33:05.893+0000 7fd0a6083340  1 waiting to reach latest gen 1, on 0..
2021-06-29T10:33:07.894+0000 7fd0a6083340  1 waiting to reach latest gen 1, on 0..
...

Secondary:
/]# radosgw-admin bucket stats --bucket buck1 {
"bucket": "buck1",
"num_shards": 37,
"tenant": "",
"zonegroup": "1785a4fa-f7d6-4081-8c72-74f9cc441d3a",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "b76458b4-55bc-41b4-8610-b4aa5df49661.34772.1",
"marker": "b76458b4-55bc-41b4-8610-b4aa5df49661.34772.1",
"index_type": "Normal",
"owner": "test1",
"ver": "0#9,1#9,2#9,3#9,4#9,5#9,6#9,7#9,8#9,9#9,10#9,11#9,12#9,13#9,14#9,15#9,16#9,17#9,18#9,19#9,20#9,21#9,22#9,23#9,24#9,25#9,26#9,27#9,28#9,29#9,30#9,31#9,32#9,33#9,34#9,35#9,36#9",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0",
"mtime": "0.000000",
"creation_time": "2021-06-28T07:45:16.289450Z",
"max_marker": "0#00000000008.19.1,1#00000000008.19.1,2#00000000008.19.1,3#00000000008.19.1,4#00000000008.27.1,5#00000000008.19.1,6#00000000008.19.1,7#00000000008.19.1,8#00000000008.19.1,9#00000000008.19.1,10#00000000008.19.1,11#00000000008.19.1,12#00000000008.19.1,13#00000000008.19.1,14#00000000008.19.1,15#00000000008.19.1,16#00000000008.27.1,17#00000000008.19.1,18#00000000008.19.1,19#00000000008.19.1,20#00000000008.19.1,21#00000000008.19.1,22#00000000008.28.1,23#00000000008.19.1,24#00000000008.19.1,25#00000000008.2748.1,26#00000000008.19.1,27#00000000008.19.1,28#00000000008.19.1,29#00000000008.19.1,30#00000000008.19.1,31#00000000008.19.1,32#00000000008.28.1,33#00000000008.19.1,34#00000000008.19.1,35#00000000008.19.1,36#00000000008.19.1",
"usage": {
"rgw.main": {
"size": 63053328000,
"size_actual": 63062016000,
"size_utilized": 63053328000,
"size_kb": 61575516,
"size_kb_actual": 61584000,
"size_kb_utilized": 61575516,
"num_objects": 15000
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}

Actions
Copy link
 #1

Updated by Casey Bodley almost 3 years ago

  • Tags set to multisite-reshard
Actions
Copy link
 #2

Updated by Vidushi Mishra over 2 years ago

Not seeing an issue as of now on ceph version 17.0.0-8051-g15b54dc9 (15b54dc9eaa835e95809e32e8ddf109d416320c9) quincy (dev)

Steps followed:
1. On a versioned bucket tx/ms-bkt-v4, upload 2K objects from each site simultaneously.
2. After the workload completes, verify bucket stats and sync is consistent on both sites.
3. Now do bucket sync disable from the primary site.
4. Upload another 1K objects from the secondary site.
5. Reshard the bucket from the primary site to 41 shards.
6. Enable bucket sync and verify sync is consistent.

Actions
Copy link
 #3

Updated by Casey Bodley over 2 years ago

  • Status changed from New to Closed
Actions
Copy link

Also available in: Atom PDF