Project

General

Profile

Bug #19042

rgw: multiple zonegroups: bucket can't be created if the user name isn't registered on master zone in master zonegroup

Added by Osamu KIMURA about 7 years ago. Updated over 4 years ago.

Status:
New
Priority:
Normal
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This is a part of issues discussed on ceph-devel ML: http://marc.info/?t=148671813300008
It may be related to #19041 and #18346.

I tentatively configured multiple zonegroups in a single realm on single ceph cluster for evaluation of behavior of multiple zonegroups:

RGW: v10.2.5
Ceph cluster: RHCS 2.0

Any bucket creation is rejected with "404 NoSuchKey" if the master zone in the master zonegroup doesn't have the same user id.

# radosgw-admin metadata list user --rgw-zonegroup west --rgw-zone osaka
[
    "sync-user",
    "nishi" 
]

# radosgw-admin metadata list user --rgw-zonegroup jp --rgw-zone jp-east
[ "sync-user", "test" ] # s3cmd -c s3jp-east.cfg ls (access jp-east endpoint with uid "test")
2017-02-22 02:58 s3://bucket2
2017-02-22 05:41 s3://bucket3
2017-02-22 05:58 s3://bucket4 # s3cmd -c s3nishi.cfg mb s3://bucket5 (access osaka endpoint with uid "nishi")
ERROR: S3 error: 404 (NoSuchKey)

ceph-client.radosgw.osaka.log:

2017-02-22 16:05:48.750177 7f60cd7ca700  1 ====== starting new request req=0x7f60cd7c4710 =====
2017-02-22 16:05:48.750217 7f60cd7ca700  2 req 23:0.000041::PUT /bucket5/::initializing for trans_id = tx000000000000000000017-0058ad384c-15e1-osaka
2017-02-22 16:05:48.750226 7f60cd7ca700 10 rgw api priority: s3=5 s3website=4
2017-02-22 16:05:48.750228 7f60cd7ca700 10 host=node5
2017-02-22 16:05:48.750249 7f60cd7ca700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2017-02-22 16:05:48.750255 7f60cd7ca700 10 meta>> HTTP_X_AMZ_DATE
2017-02-22 16:05:48.750258 7f60cd7ca700 10 x>> x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-22 16:05:48.750270 7f60cd7ca700 10 x>> x-amz-date:20170222T070548Z
2017-02-22 16:05:48.750288 7f60cd7ca700 10 handler=25RGWHandler_REST_Bucket_S3
2017-02-22 16:05:48.750290 7f60cd7ca700  2 req 23:0.000114:s3:PUT /bucket5/::getting op 1
2017-02-22 16:05:48.750295 7f60cd7ca700 10 op=27RGWCreateBucket_ObjStore_S3
2017-02-22 16:05:48.750296 7f60cd7ca700  2 req 23:0.000121:s3:PUT /bucket5/:create_bucket:authorizing
2017-02-22 16:05:48.750316 7f60cd7ca700 10 v4 signature format = c7d17192abba0f2671330007790577f09b154c64c24799599fc93f65f435b7ac
2017-02-22 16:05:48.750330 7f60cd7ca700 10 v4 credential format = 87AU1NRJAI7ZN1AQXB6W/20170222/west/s3/aws4_request
2017-02-22 16:05:48.750333 7f60cd7ca700 10 access key id = 87AU1NRJAI7ZN1AQXB6W
2017-02-22 16:05:48.750334 7f60cd7ca700 10 credential scope = 20170222/west/s3/aws4_request
2017-02-22 16:05:48.750356 7f60cd7ca700 10 canonical headers format = host:node5:8082
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
x-amz-date:20170222T070548Z

2017-02-22 16:05:48.750361 7f60cd7ca700 10 delaying v4 auth
2017-02-22 16:05:48.750364 7f60cd7ca700  2 req 23:0.000189:s3:PUT /bucket5/:create_bucket:normalizing buckets and tenants
2017-02-22 16:05:48.750367 7f60cd7ca700 10 s->object=<NULL> s->bucket=bucket5
2017-02-22 16:05:48.750369 7f60cd7ca700  2 req 23:0.000193:s3:PUT /bucket5/:create_bucket:init permissions
2017-02-22 16:05:48.750370 7f60cd7ca700  2 req 23:0.000195:s3:PUT /bucket5/:create_bucket:recalculating target
2017-02-22 16:05:48.750372 7f60cd7ca700  2 req 23:0.000196:s3:PUT /bucket5/:create_bucket:reading permissions
2017-02-22 16:05:48.750374 7f60cd7ca700  2 req 23:0.000198:s3:PUT /bucket5/:create_bucket:init op
2017-02-22 16:05:48.750375 7f60cd7ca700  2 req 23:0.000200:s3:PUT /bucket5/:create_bucket:verifying op mask
2017-02-22 16:05:48.750377 7f60cd7ca700  2 req 23:0.000201:s3:PUT /bucket5/:create_bucket:verifying op permissions
2017-02-22 16:05:48.751251 7f60cd7ca700  2 req 23:0.001074:s3:PUT /bucket5/:create_bucket:verifying op params
2017-02-22 16:05:48.751266 7f60cd7ca700  2 req 23:0.001091:s3:PUT /bucket5/:create_bucket:pre-executing
2017-02-22 16:05:48.751269 7f60cd7ca700  2 req 23:0.001093:s3:PUT /bucket5/:create_bucket:executing
2017-02-22 16:05:48.751339 7f60cd7ca700 10 payload request hash = d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-22 16:05:48.751356 7f60cd7ca700 10 canonical request = PUT
/bucket5/

host:node5:8082
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
x-amz-date:20170222T070548Z

host;x-amz-content-sha256;x-amz-date
d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-22 16:05:48.751357 7f60cd7ca700 10 canonical request hash = e428cbc9081953102984d2a0dce6ae70dc076e6f95b3c875ecbc055c446aaf40
2017-02-22 16:05:48.751361 7f60cd7ca700 10 string to sign = AWS4-HMAC-SHA256
20170222T070548Z
20170222/west/s3/aws4_request
e428cbc9081953102984d2a0dce6ae70dc076e6f95b3c875ecbc055c446aaf40
2017-02-22 16:05:48.751389 7f60cd7ca700 10 date_k        = ad5444f7116af8aa782d6879e68977cd165288cdcb78c01fa4ea284a75a07b76
2017-02-22 16:05:48.751407 7f60cd7ca700 10 region_k      = 8faf00bd25092a41a7332d76e31426eecb9171835f4e7b3ccdc7def8074cb72d
2017-02-22 16:05:48.751424 7f60cd7ca700 10 service_k     = f1ba1b42c778178162919b9de68fa5e771a0b34f5f8b6d3a50261cf243448b20
2017-02-22 16:05:48.751441 7f60cd7ca700 10 signing_k     = 5e166a8f00366d1e85579744e69cea8ffa236aa82c5ace730750b8e1120a1373
2017-02-22 16:05:48.751468 7f60cd7ca700 10 signature_k   = c7d17192abba0f2671330007790577f09b154c64c24799599fc93f65f435b7ac
2017-02-22 16:05:48.751473 7f60cd7ca700 10 new signature = c7d17192abba0f2671330007790577f09b154c64c24799599fc93f65f435b7ac
2017-02-22 16:05:48.751474 7f60cd7ca700 10 ----------------------------- Verifying signatures
2017-02-22 16:05:48.751474 7f60cd7ca700 10 Signature     = c7d17192abba0f2671330007790577f09b154c64c24799599fc93f65f435b7ac
2017-02-22 16:05:48.751475 7f60cd7ca700 10 New Signature = c7d17192abba0f2671330007790577f09b154c64c24799599fc93f65f435b7ac
2017-02-22 16:05:48.751475 7f60cd7ca700 10 -----------------------------
2017-02-22 16:05:48.751476 7f60cd7ca700 10 v4 auth ok
2017-02-22 16:05:48.751499 7f60cd7ca700 10 create bucket location constraint: west
2017-02-22 16:05:48.751521 7f60cd7ca700 10 cache get: name=osaka.rgw.data.root+bucket5 : miss
2017-02-22 16:05:48.752905 7f60cd7ca700 10 cache put: name=osaka.rgw.data.root+bucket5 info.flags=0
2017-02-22 16:05:48.752918 7f60cd7ca700 10 adding osaka.rgw.data.root+bucket5 to cache LRU end
2017-02-22 16:05:48.752924 7f60cd7ca700  0 sending create_bucket request to master zonegroup
2017-02-22 16:05:48.752946 7f60cd7ca700 10 get_canon_resource(): dest=/bucket5/
2017-02-22 16:05:48.752953 7f60cd7ca700 10 generated canonical header: PUT

Wed Feb 22 07:05:48 2017
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
/bucket5/
2017-02-22 16:05:48.758740 7f60cd7ca700 10 receive_http_header
2017-02-22 16:05:48.758745 7f60cd7ca700 10 received header:HTTP/1.1 404 Not Found
2017-02-22 16:05:48.758747 7f60cd7ca700 10 receive_http_header
2017-02-22 16:05:48.758748 7f60cd7ca700 10 received header:x-amz-request-id: tx000000000000000000fa9-0058ad384c-1582-jp-east
2017-02-22 16:05:48.758752 7f60cd7ca700 10 receive_http_header
2017-02-22 16:05:48.758752 7f60cd7ca700 10 received header:Content-Length: 177
2017-02-22 16:05:48.758755 7f60cd7ca700 10 receive_http_header
2017-02-22 16:05:48.758755 7f60cd7ca700 10 received header:Accept-Ranges: bytes
2017-02-22 16:05:48.758757 7f60cd7ca700 10 receive_http_header
2017-02-22 16:05:48.758757 7f60cd7ca700 10 received header:Content-Type: application/xml
2017-02-22 16:05:48.758765 7f60cd7ca700 10 receive_http_header
2017-02-22 16:05:48.758766 7f60cd7ca700 10 received header:Date: Wed, 22 Feb 2017 07:05:48 GMT
2017-02-22 16:05:48.758767 7f60cd7ca700 10 receive_http_header
2017-02-22 16:05:48.758768 7f60cd7ca700 10 received header:
2017-02-22 16:05:48.758855 7f60cd7ca700  2 req 23:0.008679:s3:PUT /bucket5/:create_bucket:completing
2017-02-22 16:05:48.758910 7f60cd7ca700  2 req 23:0.008734:s3:PUT /bucket5/:create_bucket:op status=-2
2017-02-22 16:05:48.758916 7f60cd7ca700  2 req 23:0.008741:s3:PUT /bucket5/:create_bucket:http status=404
2017-02-22 16:05:48.758920 7f60cd7ca700  1 ====== req done req=0x7f60cd7c4710 op status=-2 http_status=404 ======
2017-02-22 16:05:48.758941 7f60cd7ca700  1 civetweb: 0x7f60f4043870: 192.168.20.15 - - [22/Feb/2017:16:05:48 +0900] "PUT /bucket5/ HTTP/1.1" 404 0 - -

ceph-client.radosgw.jp-east.log:

2017-02-22 16:05:48.757493 7f8c5bc8c700  1 ====== starting new request req=0x7f8c5bc86710 =====
2017-02-22 16:05:48.757514 7f8c5bc8c700  2 req 4009:0.000023::PUT /bucket5/::initializing for trans_id = tx000000000000000000fa9-0058ad384c-1582-jp-east
2017-02-22 16:05:48.757521 7f8c5bc8c700 10 rgw api priority: s3=5 s3website=4
2017-02-22 16:05:48.757522 7f8c5bc8c700 10 host=node5
2017-02-22 16:05:48.757530 7f8c5bc8c700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2017-02-22 16:05:48.757533 7f8c5bc8c700 10 x>> x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-22 16:05:48.757550 7f8c5bc8c700 10 handler=25RGWHandler_REST_Bucket_S3
2017-02-22 16:05:48.757552 7f8c5bc8c700  2 req 4009:0.000061:s3:PUT /bucket5/::getting op 1
2017-02-22 16:05:48.757556 7f8c5bc8c700 10 op=27RGWCreateBucket_ObjStore_S3
2017-02-22 16:05:48.757557 7f8c5bc8c700  2 req 4009:0.000066:s3:PUT /bucket5/:create_bucket:authorizing
2017-02-22 16:05:48.757577 7f8c5bc8c700 10 get_canon_resource(): dest=/bucket5/
2017-02-22 16:05:48.757579 7f8c5bc8c700 10 auth_hdr:
PUT

Wed Feb 22 07:05:48 2017
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
/bucket5/
2017-02-22 16:05:48.757628 7f8c5bc8c700 10 cache get: name=jp-east.rgw.users.uid+nishi : type miss (requested=6, cached=0)
2017-02-22 16:05:48.758658 7f8c5bc8c700 10 cache put: name=jp-east.rgw.users.uid+nishi info.flags=0
2017-02-22 16:05:48.758665 7f8c5bc8c700 10 moving jp-east.rgw.users.uid+nishi to cache LRU end
2017-02-22 16:05:48.758670 7f8c5bc8c700  0 User lookup failed!
2017-02-22 16:05:48.758672 7f8c5bc8c700 10 failed to authorize request
2017-02-22 16:05:48.758745 7f8c5bc8c700  2 req 4009:0.001253:s3:PUT /bucket5/:create_bucket:op status=0
2017-02-22 16:05:48.758748 7f8c5bc8c700  2 req 4009:0.001257:s3:PUT /bucket5/:create_bucket:http status=404
2017-02-22 16:05:48.758751 7f8c5bc8c700  1 ====== req done req=0x7f8c5bc86710 op status=0 http_status=404 ======
2017-02-22 16:05:48.758821 7f8c5bc8c700  1 civetweb: 0x7f8cb0039ab0: 192.168.20.15 - - [22/Feb/2017:16:05:48 +0900] "PUT /bucket5/ HTTP/1.1" 404 0 - -

After a user account was created with the same user id on jp-east zone, a bucket creation is succeeded. Note that the uid must be same, but key may differ.

[root@gateway-node5 ~]# radosgw-admin user create --uid=nishi --display-name="Nisshi WEST on jp" --rgw-zonegroup=jp --rgw-zone=jp-east
{
    "user_id": "nishi",
    "display_name": "Nisshi WEST on jp",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "nishi",
            "access_key": "KKC0R5T1M9ZFZ870NEG2",
            "secret_key": "YMAPY74hBCvE8nJPpvKPHY8PM4pizfFc8GBWbAbR" 
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
       "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

# radosgw-admin metadata list user --rgw-zonegroup jp --rgw-zone jp-east
[ "sync-user", "test", "nishi" ] # s3cmd -c s3nishi.cfg mb s3://bucket5
Bucket 's3://bucket5/' created # s3cmd -c s3nishi.cfg ls
2017-02-22 07:06 s3://bucket5

ceph-client.radosgw.osaka.log:

2017-02-22 16:06:31.434044 7f60ca7c4700  1 ====== starting new request req=0x7f60ca7be710 =====
2017-02-22 16:06:31.434086 7f60ca7c4700  2 req 25:0.000043::PUT /bucket5/::initializing for trans_id = tx000000000000000000019-0058ad3877-15e1-osaka
2017-02-22 16:06:31.434096 7f60ca7c4700 10 rgw api priority: s3=5 s3website=4
2017-02-22 16:06:31.434098 7f60ca7c4700 10 host=node5
2017-02-22 16:06:31.434112 7f60ca7c4700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2017-02-22 16:06:31.434118 7f60ca7c4700 10 meta>> HTTP_X_AMZ_DATE
2017-02-22 16:06:31.434121 7f60ca7c4700 10 x>> x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-22 16:06:31.434123 7f60ca7c4700 10 x>> x-amz-date:20170222T070631Z
2017-02-22 16:06:31.434144 7f60ca7c4700 10 handler=25RGWHandler_REST_Bucket_S3
2017-02-22 16:06:31.434147 7f60ca7c4700  2 req 25:0.000105:s3:PUT /bucket5/::getting op 1
2017-02-22 16:06:31.434152 7f60ca7c4700 10 op=27RGWCreateBucket_ObjStore_S3
2017-02-22 16:06:31.434154 7f60ca7c4700  2 req 25:0.000111:s3:PUT /bucket5/:create_bucket:authorizing
2017-02-22 16:06:31.434168 7f60ca7c4700 10 v4 signature format = 84cf93c40d11a2b8a47688726e1d776a29c71944b7709a864d44ab25f83cb408
2017-02-22 16:06:31.434178 7f60ca7c4700 10 v4 credential format = 87AU1NRJAI7ZN1AQXB6W/20170222/west/s3/aws4_request
2017-02-22 16:06:31.434181 7f60ca7c4700 10 access key id = 87AU1NRJAI7ZN1AQXB6W
2017-02-22 16:06:31.434182 7f60ca7c4700 10 credential scope = 20170222/west/s3/aws4_request
2017-02-22 16:06:31.434204 7f60ca7c4700 10 canonical headers format = host:node5:8082
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
x-amz-date:20170222T070631Z

2017-02-22 16:06:31.434211 7f60ca7c4700 10 delaying v4 auth
2017-02-22 16:06:31.434214 7f60ca7c4700  2 req 25:0.000172:s3:PUT /bucket5/:create_bucket:normalizing buckets and tenants
2017-02-22 16:06:31.434216 7f60ca7c4700 10 s->object=<NULL> s->bucket=bucket5
2017-02-22 16:06:31.434218 7f60ca7c4700  2 req 25:0.000176:s3:PUT /bucket5/:create_bucket:init permissions
2017-02-22 16:06:31.434219 7f60ca7c4700  2 req 25:0.000177:s3:PUT /bucket5/:create_bucket:recalculating target
2017-02-22 16:06:31.434220 7f60ca7c4700  2 req 25:0.000178:s3:PUT /bucket5/:create_bucket:reading permissions
2017-02-22 16:06:31.434222 7f60ca7c4700  2 req 25:0.000180:s3:PUT /bucket5/:create_bucket:init op
2017-02-22 16:06:31.434223 7f60ca7c4700  2 req 25:0.000181:s3:PUT /bucket5/:create_bucket:verifying op mask
2017-02-22 16:06:31.434224 7f60ca7c4700  2 req 25:0.000182:s3:PUT /bucket5/:create_bucket:verifying op permissions
2017-02-22 16:06:31.435415 7f60ca7c4700  2 req 25:0.001369:s3:PUT /bucket5/:create_bucket:verifying op params
2017-02-22 16:06:31.435429 7f60ca7c4700  2 req 25:0.001387:s3:PUT /bucket5/:create_bucket:pre-executing
2017-02-22 16:06:31.435433 7f60ca7c4700  2 req 25:0.001391:s3:PUT /bucket5/:create_bucket:executing
2017-02-22 16:06:31.435514 7f60ca7c4700 10 payload request hash = d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-22 16:06:31.435540 7f60ca7c4700 10 canonical request = PUT
/bucket5/

host:node5:8082
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
x-amz-date:20170222T070631Z

host;x-amz-content-sha256;x-amz-date
d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-22 16:06:31.435542 7f60ca7c4700 10 canonical request hash = c94f75a2248b288af5ec961f2fbfd2b1eb73ce59df96ea985163db69680fe2b1
2017-02-22 16:06:31.435550 7f60ca7c4700 10 string to sign = AWS4-HMAC-SHA256
20170222T070631Z
20170222/west/s3/aws4_request
c94f75a2248b288af5ec961f2fbfd2b1eb73ce59df96ea985163db69680fe2b1
2017-02-22 16:06:31.435607 7f60ca7c4700 10 date_k        = ad5444f7116af8aa782d6879e68977cd165288cdcb78c01fa4ea284a75a07b76
2017-02-22 16:06:31.435653 7f60ca7c4700 10 region_k      = 8faf00bd25092a41a7332d76e31426eecb9171835f4e7b3ccdc7def8074cb72d
2017-02-22 16:06:31.435665 7f60ca7c4700 10 service_k     = f1ba1b42c778178162919b9de68fa5e771a0b34f5f8b6d3a50261cf243448b20
2017-02-22 16:06:31.435687 7f60ca7c4700 10 signing_k     = 5e166a8f00366d1e85579744e69cea8ffa236aa82c5ace730750b8e1120a1373
2017-02-22 16:06:31.435699 7f60ca7c4700 10 signature_k   = 84cf93c40d11a2b8a47688726e1d776a29c71944b7709a864d44ab25f83cb408
2017-02-22 16:06:31.435700 7f60ca7c4700 10 new signature = 84cf93c40d11a2b8a47688726e1d776a29c71944b7709a864d44ab25f83cb408
2017-02-22 16:06:31.435700 7f60ca7c4700 10 ----------------------------- Verifying signatures
2017-02-22 16:06:31.435701 7f60ca7c4700 10 Signature     = 84cf93c40d11a2b8a47688726e1d776a29c71944b7709a864d44ab25f83cb408
2017-02-22 16:06:31.435701 7f60ca7c4700 10 New Signature = 84cf93c40d11a2b8a47688726e1d776a29c71944b7709a864d44ab25f83cb408
2017-02-22 16:06:31.435702 7f60ca7c4700 10 -----------------------------
2017-02-22 16:06:31.435703 7f60ca7c4700 10 v4 auth ok
2017-02-22 16:06:31.435730 7f60ca7c4700 10 create bucket location constraint: west
2017-02-22 16:06:31.435748 7f60ca7c4700 10 cache get: name=osaka.rgw.data.root+bucket5 : type miss (requested=22, cached=0)
2017-02-22 16:06:31.436369 7f60ca7c4700 10 cache put: name=osaka.rgw.data.root+bucket5 info.flags=0
2017-02-22 16:06:31.436376 7f60ca7c4700 10 moving osaka.rgw.data.root+bucket5 to cache LRU end
2017-02-22 16:06:31.436382 7f60ca7c4700  0 sending create_bucket request to master zonegroup
2017-02-22 16:06:31.436405 7f60ca7c4700 10 get_canon_resource(): dest=/bucket5/
2017-02-22 16:06:31.436406 7f60ca7c4700 10 generated canonical header: PUT

Wed Feb 22 07:06:31 2017
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
/bucket5/
2017-02-22 16:06:31.443190 7f60ca7c4700 10 receive_http_header
2017-02-22 16:06:31.443199 7f60ca7c4700 10 received header:HTTP/1.1 100 CONTINUE
2017-02-22 16:06:31.443200 7f60ca7c4700 10 received header:HTTP/1.1
2017-02-22 16:06:31.443201 7f60ca7c4700 10 receive_http_header
2017-02-22 16:06:31.443201 7f60ca7c4700 10 received header:
2017-02-22 16:06:31.467206 7f60ca7c4700 10 receive_http_header
2017-02-22 16:06:31.467229 7f60ca7c4700 10 received header:HTTP/1.1 200 OK
2017-02-22 16:06:31.467232 7f60ca7c4700 10 receive_http_header
2017-02-22 16:06:31.467232 7f60ca7c4700 10 received header:x-amz-request-id: tx00000000000000000112b-0058ad3877-1582-jp-east
2017-02-22 16:06:31.467238 7f60ca7c4700 10 receive_http_header
2017-02-22 16:06:31.467238 7f60ca7c4700 10 received header:Content-Length: 779
2017-02-22 16:06:31.467241 7f60ca7c4700 10 receive_http_header
2017-02-22 16:06:31.467242 7f60ca7c4700 10 received header:Date: Wed, 22 Feb 2017 07:06:31 GMT
2017-02-22 16:06:31.467243 7f60ca7c4700 10 receive_http_header
2017-02-22 16:06:31.467244 7f60ca7c4700 10 received header:
2017-02-22 16:06:32.725087 7f60ca7c4700 10 cache put: name=osaka.rgw.data.root+.bucket.meta.bucket5:22719b44-532d-41ee-974e-fc89cb93255f.5506.1 info.flags=23
2017-02-22 16:06:32.725100 7f60ca7c4700 10 adding osaka.rgw.data.root+.bucket.meta.bucket5:22719b44-532d-41ee-974e-fc89cb93255f.5506.1 to cache LRU end
2017-02-22 16:06:32.725102 7f60ca7c4700 10 updating xattr: name=user.rgw.acl bl.length()=139
2017-02-22 16:06:32.725118 7f60ca7c4700 10 distributing notification oid=notify.2 bl.length()=811
2017-02-22 16:06:32.727359 7f611bfff700 10 RGWWatcher::handle_notify()  notify_id 755914244098 cookie 140055919525568 notifier 5601 bl.length()=811
2017-02-22 16:06:32.727389 7f611bfff700 10 cache put: name=osaka.rgw.data.root+.bucket.meta.bucket5:22719b44-532d-41ee-974e-fc89cb93255f.5506.1 info.flags=23
2017-02-22 16:06:32.727392 7f611bfff700 10 moving osaka.rgw.data.root+.bucket.meta.bucket5:22719b44-532d-41ee-974e-fc89cb93255f.5506.1 to cache LRU end
2017-02-22 16:06:32.727394 7f611bfff700 10 updating xattr: name=user.rgw.acl bl.length()=139
2017-02-22 16:06:32.735773 7f60ca7c4700 10 cache put: name=osaka.rgw.data.root+bucket5 info.flags=23
2017-02-22 16:06:32.735776 7f60ca7c4700 10 moving osaka.rgw.data.root+bucket5 to cache LRU end
2017-02-22 16:06:32.735789 7f60ca7c4700 10 distributing notification oid=notify.1 bl.length()=510
2017-02-22 16:06:32.736366 7f611bfff700 10 RGWWatcher::handle_notify()  notify_id 755914244099 cookie 140055919510768 notifier 5601 bl.length()=510
2017-02-22 16:06:32.736386 7f611bfff700 10 cache put: name=osaka.rgw.data.root+bucket5 info.flags=23
2017-02-22 16:06:32.736390 7f611bfff700 10 moving osaka.rgw.data.root+bucket5 to cache LRU end
2017-02-22 16:06:32.741159 7f60ca7c4700  2 req 25:1.307114:s3:PUT /bucket5/:create_bucket:completing
2017-02-22 16:06:32.741242 7f60ca7c4700  2 req 25:1.307200:s3:PUT /bucket5/:create_bucket:op status=0
2017-02-22 16:06:32.741245 7f60ca7c4700  2 req 25:1.307203:s3:PUT /bucket5/:create_bucket:http status=200
2017-02-22 16:06:32.741252 7f60ca7c4700  1 ====== req done req=0x7f60ca7be710 op status=0 http_status=200 ======
2017-02-22 16:06:32.741294 7f60ca7c4700  1 civetweb: 0x7f60f404c2f0: 192.168.20.15 - - [22/Feb/2017:16:06:31 +0900] "PUT /bucket5/ HTTP/1.1" 200 0 - -

ceph-client.radosgw.jp-east.log:

2017-02-22 16:06:31.440936 7f8c66ca2700  1 ====== starting new request req=0x7f8c66c9c710 =====
2017-02-22 16:06:31.440965 7f8c66ca2700  2 req 4395:0.000026::PUT /bucket5/::initializing for trans_id = tx00000000000000000112b-0058ad3877-1582-jp-east
2017-02-22 16:06:31.440973 7f8c66ca2700 10 rgw api priority: s3=5 s3website=4
2017-02-22 16:06:31.440974 7f8c66ca2700 10 host=node5
2017-02-22 16:06:31.440984 7f8c66ca2700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2017-02-22 16:06:31.440987 7f8c66ca2700 10 x>> x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-22 16:06:31.441006 7f8c66ca2700 10 handler=25RGWHandler_REST_Bucket_S3
2017-02-22 16:06:31.441509 7f8c66ca2700  2 req 4395:0.000573:s3:PUT /bucket5/::getting op 1
2017-02-22 16:06:31.442014 7f8c66ca2700 10 op=27RGWCreateBucket_ObjStore_S3
2017-02-22 16:06:31.442021 7f8c66ca2700  2 req 4395:0.001085:s3:PUT /bucket5/:create_bucket:authorizing
2017-02-22 16:06:31.442056 7f8c66ca2700 10 get_canon_resource(): dest=/bucket5/
2017-02-22 16:06:31.442059 7f8c66ca2700 10 auth_hdr:
PUT

Wed Feb 22 07:06:31 2017
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
/bucket5/
2017-02-22 16:06:31.442116 7f8c66ca2700 10 cache get: name=jp-east.rgw.users.uid+nishi : hit (requested=6, cached=23)
2017-02-22 16:06:31.442131 7f8c66ca2700 10 cache get: name=jp-east.rgw.users.uid+nishi : hit (requested=1, cached=23)
2017-02-22 16:06:31.442164 7f8c66ca2700  2 req 4395:0.001228:s3:PUT /bucket5/:create_bucket:normalizing buckets and tenants
2017-02-22 16:06:31.442180 7f8c66ca2700 10 s->object=<NULL> s->bucket=bucket5
2017-02-22 16:06:31.442197 7f8c66ca2700  2 req 4395:0.001261:s3:PUT /bucket5/:create_bucket:init permissions
2017-02-22 16:06:31.442199 7f8c66ca2700  2 req 4395:0.001263:s3:PUT /bucket5/:create_bucket:recalculating target
2017-02-22 16:06:31.442200 7f8c66ca2700  2 req 4395:0.001265:s3:PUT /bucket5/:create_bucket:reading permissions
2017-02-22 16:06:31.442203 7f8c66ca2700  2 req 4395:0.001268:s3:PUT /bucket5/:create_bucket:init op
2017-02-22 16:06:31.442206 7f8c66ca2700  2 req 4395:0.001269:s3:PUT /bucket5/:create_bucket:verifying op mask
2017-02-22 16:06:31.442208 7f8c66ca2700  2 req 4395:0.001272:s3:PUT /bucket5/:create_bucket:verifying op permissions
2017-02-22 16:06:31.443130 7f8c66ca2700  2 req 4395:0.002194:s3:PUT /bucket5/:create_bucket:verifying op params
2017-02-22 16:06:31.443139 7f8c66ca2700  2 req 4395:0.002203:s3:PUT /bucket5/:create_bucket:pre-executing
2017-02-22 16:06:31.443158 7f8c66ca2700  2 req 4395:0.002222:s3:PUT /bucket5/:create_bucket:executing
2017-02-22 16:06:31.443293 7f8c66ca2700 10 create bucket location constraint: west
2017-02-22 16:06:31.443317 7f8c66ca2700 10 cache get: name=jp-east.rgw.data.root+bucket5 : miss
2017-02-22 16:06:31.445043 7f8c66ca2700 10 cache put: name=jp-east.rgw.data.root+bucket5 info.flags=0
2017-02-22 16:06:31.445056 7f8c66ca2700 10 adding jp-east.rgw.data.root+bucket5 to cache LRU end
2017-02-22 16:06:31.454359 7f8c66ca2700 10 cache put: name=jp-east.rgw.data.root+.bucket.meta.bucket5:22719b44-532d-41ee-974e-fc89cb93255f.5506.1 info.flags=23
2017-02-22 16:06:31.454369 7f8c66ca2700 10 adding jp-east.rgw.data.root+.bucket.meta.bucket5:22719b44-532d-41ee-974e-fc89cb93255f.5506.1 to cache LRU end
2017-02-22 16:06:31.454371 7f8c66ca2700 10 updating xattr: name=user.rgw.acl bl.length()=151
2017-02-22 16:06:31.454396 7f8c66ca2700 10 distributing notification oid=notify.2 bl.length()=837
2017-02-22 16:06:31.455068 7f8cc3fff700 10 RGWWatcher::handle_notify()  notify_id 747324309541 cookie 140243475537840 notifier 5506 bl.length()=837
2017-02-22 16:06:31.455086 7f8cc3fff700 10 cache put: name=jp-east.rgw.data.root+.bucket.meta.bucket5:22719b44-532d-41ee-974e-fc89cb93255f.5506.1 info.flags=23
2017-02-22 16:06:31.455090 7f8cc3fff700 10 moving jp-east.rgw.data.root+.bucket.meta.bucket5:22719b44-532d-41ee-974e-fc89cb93255f.5506.1 to cache LRU end
2017-02-22 16:06:31.455094 7f8cc3fff700 10 updating xattr: name=user.rgw.acl bl.length()=151
2017-02-22 16:06:31.462383 7f8c66ca2700 10 cache put: name=jp-east.rgw.data.root+bucket5 info.flags=23
2017-02-22 16:06:31.462396 7f8c66ca2700 10 moving jp-east.rgw.data.root+bucket5 to cache LRU end
2017-02-22 16:06:31.462408 7f8c66ca2700 10 distributing notification oid=notify.1 bl.length()=524
2017-02-22 16:06:31.463078 7f8cc3fff700 10 RGWWatcher::handle_notify()  notify_id 747324309508 cookie 140243475523040 notifier 5506 bl.length()=524
2017-02-22 16:06:31.463099 7f8cc3fff700 10 cache put: name=jp-east.rgw.data.root+bucket5 info.flags=23
2017-02-22 16:06:31.463102 7f8cc3fff700 10 moving jp-east.rgw.data.root+bucket5 to cache LRU end
2017-02-22 16:06:31.467072 7f8c66ca2700  2 req 4395:0.026136:s3:PUT /bucket5/:create_bucket:completing
2017-02-22 16:06:31.467143 7f8c66ca2700  2 req 4395:0.026207:s3:PUT /bucket5/:create_bucket:op status=0
2017-02-22 16:06:31.467156 7f8c66ca2700  2 req 4395:0.026221:s3:PUT /bucket5/:create_bucket:http status=200
2017-02-22 16:06:31.467161 7f8c66ca2700  1 ====== req done req=0x7f8c66c9c710 op status=0 http_status=200 ======
2017-02-22 16:06:31.467191 7f8c66ca2700  1 civetweb: 0x7f8cb8035ca0: 192.168.20.15 - - [22/Feb/2017:16:06:31 +0900] "PUT /bucket5/ HTTP/1.1" 200 0 - -

plantuml_bucket_create1.png View - sequence chart (can't create bucket) (18.4 KB) Osamu KIMURA, 03/01/2017 04:34 AM

plantuml_bucket_create2.png View - sequence chart (can create bucket) (17.4 KB) Osamu KIMURA, 03/01/2017 04:34 AM

History

#1 Updated by Yehuda Sadeh about 7 years ago

  • Assignee set to Orit Wasserman

#2 Updated by Osamu KIMURA about 7 years ago

I stick sequence charts created by PlantUML plugin

If the user account doesn't exist on the master zone in the master zonegroup, bucket creation fails.
sequence chart (can't create bucket)

If the user account exists on the master zone in the master zonegroup, bucket creation succeeds even if the keys are differnt.
sequence chart (can create bucket)

#3 Updated by Osamu KIMURA almost 7 years ago

Is this related to #19756 / #19757 ?

#4 Updated by Orit Wasserman over 6 years ago

We will need to forward all user creation operation to the master zonegroup.
At the moment all users needs to be created in the master zone.

#5 Updated by Orit Wasserman over 6 years ago

  • Status changed from New to 12

#6 Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New

Also available in: Atom PDF