Project

General

Profile

Bug #15515

[rgw multisite] Unable to list the container and object using swift from secondary rgw node after a full sync

Added by Kumar Hemanth almost 8 years ago. Updated almost 8 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Synchronized Container and object is not seen on the remote rgw node using swift api.

Steps :-
1. Created rgw active/active multisite configuration.
2. Created Container and an Object on the master rgw node using swift.

On rgw secondary node - sync did not start immediately after the object upload was done using swift. Sync started only after rgw service was restarted on both the nodes. After some time created Container and object was seen with "rados ls".

Tried listing the Container and File from rgw secondary node, swift command to list the container fails with "Auth GET failed: http://rgw2:8080/auth 403 Forbidden"

====================
On Master Node :-
-----------------
[root@rgw1 /]# rados ls -p us-1.rgw.buckets.data | grep newfile2
ee31c3bd-8907-407c-a58c-30aa23c2be62.54149.1_newfile2

[root@rgw1 /]# swift -V 1.0 -A http://rgw1:8080/auth -U testuser:swift -K 'pbG50bUbrv8uejc9CREOhGEINWqH3auWXOr7CTLd' stat Container2 newfile2
Account: v1
Container: Container2
Object: newfile2
Content Type: binary/octet-stream
Content Length: 0
Last Modified: Fri, 15 Apr 2016 10:23:56 GMT
ETag: d41d8cd98f00b204e9800998ecf8427e
Accept-Ranges: bytes
X-Timestamp: 1460715836.92832
X-Trans-Id: tx0000000000000000086eb-005710dd14-d394-us-1

On Secondary Node :-
---------------------
[root@rgw2 ceph]# rados ls -p us-2.rgw.buckets.data | grep newfile2
ee31c3bd-8907-407c-a58c-30aa23c2be62.54149.1_newfile2

[root@rgw2 ceph]# swift -V 1.0 -A http://rgw2:8080/auth -U testuser:swift -K 'pbG50bUbrv8uejc9CREOhGEINWqH3auWXOr7CTLd' stat Container2 newfile2
Auth GET failed: http://rgw2:8080/auth 403 Forbidden

Full execution steps to hit the issue :- http://pastebin.test.redhat.com/366089

History

#1 Updated by Kumar Hemanth almost 8 years ago

Ceph-RGW Version :- ceph-radosgw-10.1.1-1.el7cp.x86_64

#2 Updated by Abhishek Lekshmanan almost 8 years ago

Can you paste the contents of pastebin somewhere else, swift apis worked for me with multisite for most common operations

#3 Updated by Kumar Hemanth almost 8 years ago

Full execution steps to hit the issue :- http://pastebin.com/9s1vxr1H

#4 Updated by Abhishek Lekshmanan almost 8 years ago

Tried reproducing this issue, couldn't hit this one yet.. I'm only seeing that stats are not exactly in sync.. (ie. object counts are not updated.. though objects are themselves present) but didn't hit the 403 error yet.. could you always reproduce this? Can we have rgw logs at debug_rgw=20 debug_ms=1 when you do a swift stat?

#5 Updated by Yehuda Sadeh almost 8 years ago

  • Status changed from New to Can't reproduce

Also available in: Atom PDF