Project

General

Profile

Actions
Copy link

Bug #52684

closed

Observing sync inconsistencies on a bucket that has been resharded.

Added by Vidushi Mishra over 2 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
Ceph - v17.0.0
% Done:

0%

Source:
Tags:
multisite-reshard
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Issue: We observe sync inconsistencies for data written to a bucket post it has been resharded.

Steps:
1. Create a bucket via awscli from the primary.
[aws s3api create-bucket --bucket fs6 --endpoint http://magna101:8080]
2. Verify bucket stats on the primary and secondary for num_shards and the num_objects.
3. Upload an object from the primary site.
4. Verify bucket list and bucket stats. They should list 1 object on both sites.
5. Now reshard the bucket from the primary [11 to 29 shards]
6. After the reshard is successful verify the bucket stats and the bucket list.
'num-shards' on primary should be 29 and on secondary should be 11.
7. Now write 600 more objects from the primary to secondary.
8. Verify the bucket stats and bucket list on both sites.

Actual results:

1. bucket stats on primary site fails :
[ceph: root@magna017 /]# radosgw-admin bucket stats --bucket fs5
error getting bucket stats bucket=fs5 ret=-2
failure: (2) No such file or directory:
[ceph: root@magna017 /]#

2. The Secondary has only 436 objects synced out of 600 written post the bucket was resharded from the primary.
[ceph: root@clara001 /]# radosgw-admin bucket stats --bucket fs6 | grep num_objects
"num_objects": 436

Expected results:

1. Bucket stats on both sites should go fine without reporting any error.
2. No data sync issues should be observed post a reshard.

Issue observed on ceph version :
ceph version 17.0.0-5278-gbdea385a (bdea385a57ec55831eab75134de20be2ca9349dd) quincy (dev)

Actions
Copy link
 #1

Updated by Vidushi Mishra over 2 years ago

We are observing the issue on a larger scale for a tenanted and versioned bucket.

Steps done:
1. create a bucket 'books-ver-1' and enable bucket versioning.
2. Manually reshard the bucket from the primary site to 47 shards and upload around 8K objects from both sites.
3. Manually reshard the bucket from the secondary site to 89 shards and upload more objects 10K from the secondary.
4. Since the rgw_max_objs_per_shard = 100, and also the IOs in progress, the bucket gets dynamically resharded again on both sites, 367 on primary and 223 on the secondary.
5. We observed sync inconsistencies after the reshard.

====================
primary: ====================

[ceph: root@clara001 /]# radosgw-admin bucket stats --bucket tx/books-ver-1
2021-10-19T06:58:32.196+0000 7fc6a37d0340 1 Realm: data (d62bd711-d486-47be-9c3e-193e49334862)
2021-10-19T06:58:32.196+0000 7fc6a37d0340 1 ZoneGroup: us (8f3b29b1-ffc6-4c90-9d0c-9bd258028cd8)
2021-10-19T06:58:32.196+0000 7fc6a37d0340 1 Zone: east (5d32949e-6245-422c-b315-9048855d3a9a)
2021-10-19T06:58:32.196+0000 7fc6a37d0340 1 using period configuration: dd132fae-4457-4f49-88b9-55ca2f8adff9:2 {
"bucket": "books-ver-1",
"num_shards": 367,
"tenant": "tx",
"zonegroup": "8f3b29b1-ffc6-4c90-9d0c-9bd258028cd8",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "5d32949e-6245-422c-b315-9048855d3a9a.30686.4",
"marker": "5d32949e-6245-422c-b315-9048855d3a9a.30686.4",
"index_type": "Normal",
"owner": "tx$user1",
"ver": "0#4,1#4,2#4,3#4,4#4,5#4,6#4,7#4,8#4,9#4,10#4,11#4,12#4,13#4,14#4,15#3,16#4,17#3,18#5,19#4,20#4,21#4,22#5,23#3,24#4,25#4,26#3,27#4,28#4,29#4,30#4,31#4,32#4,33#4,34#4,35#4,36#4,37#4,38#4,39#4,40#4,41#4,42#4,43#4,44#4,45#3,46#5,47#4,48#4,49#4,50#4,51#4,52#4,53#4,54#3,55#4,56#4,57#5,58#4,59#4,60#4,61#4,62#4,63#3,64#4,65#4,66#4,67#4,68#4,69#4,70#4,71#4,72#4,73#4,74#4,75#4,76#4,77#4,78#4,79#4,80#4,81#4,82#4,83#4,84#4,85#4,86#4,87#4,88#4,89#4,90#4,91#4,92#3,93#4,94#4,95#4,96#4,97#4,98#5,99#4,100#4,101#4,102#4,103#4,104#4,105#4,106#4,107#4,108#4,109#4,110#4,111#4,112#4,113#4,114#4,115#4,116#3,117#4,118#4,119#4,120#3,121#4,122#4,123#4,124#4,125#4,126#4,127#4,128#4,129#4,130#4,131#4,132#4,133#4,134#4,135#4,136#4,137#4,138#4,139#4,140#3,141#4,142#4,143#4,144#4,145#4,146#4,147#4,148#4,149#4,150#3,151#4,152#4,153#4,154#4,155#3,156#4,157#4,158#4,159#4,160#4,161#4,162#4,163#4,164#4,165#4,166#4,167#4,168#4,169#4,170#4,171#4,172#4,173#4,174#4,175#4,176#4,177#4,178#4,179#4,180#3,181#4,182#4,183#4,184#4,185#4,186#4,187#4,188#3,189#4,190#4,191#4,192#4,193#4,194#4,195#4,196#4,197#4,198#4,199#4,200#4,201#4,202#3,203#4,204#4,205#4,206#4,207#4,208#4,209#4,210#4,211#4,212#4,213#4,214#4,215#4,216#3,217#3,218#4,219#3,220#4,221#4,222#4,223#4,224#4,225#3,226#4,227#4,228#3,229#4,230#4,231#4,232#4,233#4,234#4,235#4,236#4,237#4,238#4,239#4,240#4,241#4,242#4,243#4,244#4,245#4,246#4,247#3,248#4,249#3,250#4,251#4,252#4,253#4,254#4,255#3,256#4,257#4,258#4,259#4,260#4,261#4,262#4,263#4,264#3,265#4,266#4,267#4,268#4,269#4,270#4,271#4,272#4,273#4,274#4,275#4,276#4,277#4,278#4,279#4,280#4,281#4,282#4,283#4,284#4,285#4,286#4,287#4,288#4,289#4,290#4,291#4,292#4,293#4,294#3,295#4,296#4,297#4,298#4,299#4,300#4,301#3,302#4,303#3,304#4,305#4,306#4,307#4,308#4,309#4,310#4,311#4,312#4,313#4,314#4,315#4,316#4,317#4,318#4,319#4,320#4,321#4,322#3,323#4,324#4,325#4,326#4,327#4,328#4,329#4,330#4,331#3,332#4,333#4,334#4,335#4,336#4,337#4,338#4,339#4,340#4,341#4,342#4,343#4,344#4,345#4,346#4,347#4,348#4,349#4,350#4,351#4,352#4,353#4,354#4,355#4,356#4,357#4,358#4,359#4,360#4,361#4,362#4,363#4,364#4,365#4,366#4",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0,197#0,198#0,199#0,200#0,201#0,202#0,203#0,204#0,205#0,206#0,207#0,208#0,209#0,210#0,211#0,212#0,213#0,214#0,215#0,216#0,217#0,218#0,219#0,220#0,221#0,222#0,223#0,224#0,225#0,226#0,227#0,228#0,229#0,230#0,231#0,232#0,233#0,234#0,235#0,236#0,237#0,238#0,239#0,240#0,241#0,242#0,243#0,244#0,245#0,246#0,247#0,248#0,249#0,250#0,251#0,252#0,253#0,254#0,255#0,256#0,257#0,258#0,259#0,260#0,261#0,262#0,263#0,264#0,265#0,266#0,267#0,268#0,269#0,270#0,271#0,272#0,273#0,274#0,275#0,276#0,277#0,278#0,279#0,280#0,281#0,282#0,283#0,284#0,285#0,286#0,287#0,288#0,289#0,290#0,291#0,292#0,293#0,294#0,295#0,296#0,297#0,298#0,299#0,300#0,301#0,302#0,303#0,304#0,305#0,306#0,307#0,308#0,309#0,310#0,311#0,312#0,313#0,314#0,315#0,316#0,317#0,318#0,319#0,320#0,321#0,322#0,323#0,324#0,325#0,326#0,327#0,328#0,329#0,330#0,331#0,332#0,333#0,334#0,335#0,336#0,337#0,338#0,339#0,340#0,341#0,342#0,343#0,344#0,345#0,346#0,347#0,348#0,349#0,350#0,351#0,352#0,353#0,354#0,355#0,356#0,357#0,358#0,359#0,360#0,361#0,362#0,363#0,364#0,365#0,366#0",
"mtime": "0.000000",
"creation_time": "2021-10-18T16:53:53.270721Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#,197#,198#,199#,200#,201#,202#,203#,204#,205#,206#,207#,208#,209#,210#,211#,212#,213#,214#,215#,216#,217#,218#,219#,220#,221#,222#,223#,224#,225#,226#,227#,228#,229#,230#,231#,232#,233#,234#,235#,236#,237#,238#,239#,240#,241#,242#,243#,244#,245#,246#,247#,248#,249#,250#,251#,252#,253#,254#,255#,256#,257#,258#,259#,260#,261#,262#,263#,264#,265#,266#,267#,268#,269#,270#,271#,272#,273#,274#,275#,276#,277#,278#,279#,280#,281#,282#,283#,284#,285#,286#,287#,288#,289#,290#,291#,292#,293#,294#,295#,296#,297#,298#,299#,300#,301#,302#,303#,304#,305#,306#,307#,308#,309#,310#,311#,312#,313#,314#,315#,316#,317#,318#,319#,320#,321#,322#,323#,324#,325#,326#,327#,328#,329#,330#,331#,332#,333#,334#,335#,336#,337#,338#,339#,340#,341#,342#,343#,344#,345#,346#,347#,348#,349#,350#,351#,352#,353#,354#,355#,356#,357#,358#,359#,360#,361#,362#,363#,364#,365#,366#",
"usage": {
"rgw.main": {
"size": 209227452,
"size_actual": 222326784,
"size_utilized": 209227452,
"size_kb": 204324,
"size_kb_actual": 217116,
"size_kb_utilized": 204324,
"num_objects": 18093 <<<<<<<<<<<<<<<<<<
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
[ceph: root@clara001 /]#

====================
secondary: ====================

[ceph: root@magna017 /]# radosgw-admin bucket stats --bucket tx/books-ver-1
2021-10-19T06:58:22.924+0000 7fb61c089340 1 Realm: data (d62bd711-d486-47be-9c3e-193e49334862)
2021-10-19T06:58:22.924+0000 7fb61c089340 1 ZoneGroup: us (8f3b29b1-ffc6-4c90-9d0c-9bd258028cd8)
2021-10-19T06:58:22.924+0000 7fb61c089340 1 Zone: west (3a571642-9f5e-46d8-8186-9eca8cc79ac6)
2021-10-19T06:58:22.924+0000 7fb61c089340 1 using period configuration: dd132fae-4457-4f49-88b9-55ca2f8adff9:2 {
"bucket": "books-ver-1",
"num_shards": 223,
"tenant": "tx",
"zonegroup": "8f3b29b1-ffc6-4c90-9d0c-9bd258028cd8",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "5d32949e-6245-422c-b315-9048855d3a9a.30686.4",
"marker": "5d32949e-6245-422c-b315-9048855d3a9a.30686.4",
"index_type": "Normal",
"owner": "tx$user1",
"ver": "0#6,1#6,2#6,3#6,4#6,5#5,6#5,7#5,8#6,9#5,10#6,11#6,12#6,13#6,14#6,15#6,16#6,17#6,18#5,19#6,20#5,21#5,22#5,23#5,24#5,25#5,26#5,27#6,28#5,29#6,30#6,31#6,32#6,33#6,34#6,35#6,36#5,37#5,38#6,39#5,40#5,41#6,42#5,43#6,44#5,45#6,46#6,47#5,48#5,49#6,50#6,51#6,52#6,53#5,54#5,55#5,56#4,57#5,58#5,59#5,60#5,61#5,62#6,63#6,64#6,65#6,66#6,67#5,68#5,69#6,70#6,71#5,72#6,73#4,74#6,75#5,76#5,77#5,78#5,79#5,80#5,81#6,82#6,83#6,84#6,85#5,86#5,87#5,88#5,89#5,90#5,91#5,92#5,93#5,94#5,95#5,96#5,97#5,98#5,99#5,100#6,101#6,102#5,103#6,104#5,105#6,106#5,107#5,108#5,109#5,110#5,111#6,112#5,113#6,114#5,115#5,116#6,117#6,118#6,119#6,120#6,121#6,122#5,123#6,124#5,125#5,126#5,127#5,128#5,129#6,130#5,131#6,132#5,133#6,134#5,135#6,136#6,137#6,138#6,139#5,140#6,141#5,142#5,143#5,144#5,145#5,146#5,147#6,148#5,149#5,150#6,151#5,152#6,153#5,154#5,155#5,156#5,157#5,158#6,159#5,160#6,161#5,162#5,163#5,164#5,165#5,166#5,167#5,168#5,169#6,170#5,171#6,172#6,173#5,174#6,175#5,176#5,177#5,178#5,179#5,180#6,181#5,182#5,183#5,184#5,185#6,186#6,187#6,188#6,189#5,190#5,191#6,192#5,193#5,194#5,195#5,196#5,197#5,198#5,199#5,200#5,201#6,202#6,203#6,204#5,205#5,206#5,207#6,208#5,209#6,210#5,211#6,212#5,213#5,214#5,215#5,216#6,217#5,218#6,219#6,220#6,221#6,222#5",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0,197#0,198#0,199#0,200#0,201#0,202#0,203#0,204#0,205#0,206#0,207#0,208#0,209#0,210#0,211#0,212#0,213#0,214#0,215#0,216#0,217#0,218#0,219#0,220#0,221#0,222#0",
"mtime": "0.000000",
"creation_time": "2021-10-18T16:53:53.270721Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#,197#,198#,199#,200#,201#,202#,203#,204#,205#,206#,207#,208#,209#,210#,211#,212#,213#,214#,215#,216#,217#,218#,219#,220#,221#,222#",
"usage": {
"rgw.main": {
"size": 209262144,
"size_actual": 222363648,
"size_utilized": 209262144,
"size_kb": 204358,
"size_kb_actual": 217152,
"size_kb_utilized": 204358,
"num_objects": 18096 <<<<<<<<<<<<<<<<<
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
[ceph: root@magna017 /]#

Actions
Copy link
 #2

Updated by Casey Bodley over 2 years ago

  • Status changed from New to Need More Info

can you please share the outputs of 'bucket sync status' on each zone? also check 'sync error list' for errors?

Actions
Copy link
 #3

Updated by Casey Bodley over 2 years ago

  • Status changed from Need More Info to Closed
Actions
Copy link

Also available in: Atom PDF