Project

General

Profile

Actions
Copy link

Bug #19043

open

rgw: multiple zonegroups: asymmetric behavior of creating bucket on secondary zonegroup

Added by Osamu KIMURA about 7 years ago. Updated about 7 years ago.

Status:
New
Priority:
Normal
Assignee:
Orit Wasserman
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This is a part of issues discussed on ceph-devel ML: http://marc.info/?t=148671813300008

I tentatively configured multiple zonegroups in a single realm on single ceph cluster for evaluation of behavior of multiple zonegroups:

zonegroup default (api_name east)
        zone default http://node5:80
    zonegroup jp (master)
        zone jp-east http://node5:8080 (master)
        zone jp-west http://node5:8081
    zonegroup west
        zone osaka http://node5:8082

RGW: v10.2.5
Ceph cluster: RHCS 2.0

Common user accounts were created on the master zone in the master zonegroup as shown in #19042.
Zonegroup map is as shown in #19040-1.

When I tried to create a bucket on west zonegroup which was the same bucket name as one on jp zonegroup, the request was rejected. It seems bucket namespace is shared among multiple zonegroups, it is different from #18346.

# radosgw-admin metadata list user --rgw-zonegroup west --rgw-zone osaka
[
    "sync-user",
    "nishi" 
]

# radosgw-admin metadata list user --rgw-zonegroup jp --rgw-zone jp-east
[
    "sync-user",
    "test",
    "nishi" 
]

# s3cmd -c s3jp-east.cfg ls              (access jp-east endpoint with uid "test")
2017-02-22 02:58  s3://bucket2
2017-02-22 05:41  s3://bucket3
2017-02-22 05:58  s3://bucket4
# s3cmd -c s3nishi.cfg ls                (access osaka endpoint with uid "nishi")
2017-02-22 07:06  s3://bucket5
# s3cmd -c s3nishi-jp.cfg ls             (access jp-east endpoint with uid "nishi")
2017-02-22 07:06  s3://bucket5
# s3cmd -c s3nishi.cfg mb s3://bucket3   (access osaka endpoint with uid "nishi")
ERROR: S3 error: 409 (BucketNotEmpty)

On the other hand, when I tried to create a bucket on default zonegroup which was the same bucket name as one on jp zonegroup, the bucket was successfully created.

# radosgw-admin metadata list user --rgw-zonegroup default --rgw-zone default
[
    "sync-user",
    "test" 
]

# radosgw-admin metadata list user --rgw-zonegroup jp --rgw-zone jp-east
[
    "sync-user",
    "test",
    "nishi" 
]

# s3cmd -c s3jp-east.cfg ls                 (access jp-east endpoint with uid "test")
2017-02-22 02:58  s3://bucket2
2017-02-22 05:41  s3://bucket3
2017-02-22 05:58  s3://bucket4
# s3cmd -c s3jp-east.cfg info s3://bucket3  (access jp-east endpoint with uid "test")
s3://bucket3/ (bucket):
   Location:  jp
   Payer:     BucketOwner
   Expiration Rule: none
   policy:    <?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>bucket3</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>
   cors:      none
   ACL:       Test User on jp-east: FULL_CONTROL
# s3cmd -c s3default.cfg ls                 (access default endpoint with uid "test")
2017-02-22 02:02  s3://bucket1
2017-02-22 02:58  s3://bucket2
# s3cmd -c s3default.cfg mb s3://bucket3    (access default endpoint with uid "test")
Bucket 's3://bucket3/' created
# s3cmd -c s3default.cfg ls                 (access default endpoint with uid "test")
2017-02-22 02:02  s3://bucket1
2017-02-22 02:58  s3://bucket2
2017-02-22 07:09  s3://bucket3
# s3cmd -c s3default.cfg info s3://bucket3  (access default endpoint with uid "test")
s3://bucket3/ (bucket):
   Location:  east
   Payer:     BucketOwner
   Expiration Rule: none
   policy:    <?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>bucket3</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>
   cors:      none
   ACL:       Test User: FULL_CONTROL

I'm not sure which is right behavior.
I'm also not sure the reason:

  • "default" zone in "default" zonegroup acts different?
  • Same bucket name can be created by same uid?

Probably it is caused by latter one.

ceph-client.radosgw.osaka.log:

2017-02-22 16:08:29.747383 7f60b2794700  1 ====== starting new request req=0x7f60b278e710 =====
2017-02-22 16:08:29.747415 7f60b2794700  2 req 32:0.000034::PUT /bucket3/::initializing for trans_id = tx000000000000000000020-0058ad38ed-15e1-osaka
2017-02-22 16:08:29.747423 7f60b2794700 10 rgw api priority: s3=5 s3website=4
2017-02-22 16:08:29.747424 7f60b2794700 10 host=node5
2017-02-22 16:08:29.747433 7f60b2794700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2017-02-22 16:08:29.747440 7f60b2794700 10 meta>> HTTP_X_AMZ_DATE
2017-02-22 16:08:29.747442 7f60b2794700 10 x>> x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-22 16:08:29.747444 7f60b2794700 10 x>> x-amz-date:20170222T070829Z
2017-02-22 16:08:29.747459 7f60b2794700 10 handler=25RGWHandler_REST_Bucket_S3
2017-02-22 16:08:29.747465 7f60b2794700  2 req 32:0.000084:s3:PUT /bucket3/::getting op 1
2017-02-22 16:08:29.747470 7f60b2794700 10 op=27RGWCreateBucket_ObjStore_S3
2017-02-22 16:08:29.747471 7f60b2794700  2 req 32:0.000090:s3:PUT /bucket3/:create_bucket:authorizing
2017-02-22 16:08:29.747482 7f60b2794700 10 v4 signature format = 95b97d254f830017f12778229e226c4ce9061d1917fff2e8bb899fd563f5e21b
2017-02-22 16:08:29.747488 7f60b2794700 10 v4 credential format = 87AU1NRJAI7ZN1AQXB6W/20170222/west/s3/aws4_request
2017-02-22 16:08:29.747490 7f60b2794700 10 access key id = 87AU1NRJAI7ZN1AQXB6W
2017-02-22 16:08:29.747490 7f60b2794700 10 credential scope = 20170222/west/s3/aws4_request
2017-02-22 16:08:29.747507 7f60b2794700 10 canonical headers format = host:node5:8082
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
x-amz-date:20170222T070829Z

2017-02-22 16:08:29.747513 7f60b2794700 10 delaying v4 auth
2017-02-22 16:08:29.747515 7f60b2794700  2 req 32:0.000135:s3:PUT /bucket3/:create_bucket:normalizing buckets and tenants
2017-02-22 16:08:29.747518 7f60b2794700 10 s->object=<NULL> s->bucket=bucket3
2017-02-22 16:08:29.747519 7f60b2794700  2 req 32:0.000139:s3:PUT /bucket3/:create_bucket:init permissions
2017-02-22 16:08:29.747520 7f60b2794700  2 req 32:0.000140:s3:PUT /bucket3/:create_bucket:recalculating target
2017-02-22 16:08:29.747522 7f60b2794700  2 req 32:0.000140:s3:PUT /bucket3/:create_bucket:reading permissions
2017-02-22 16:08:29.747523 7f60b2794700  2 req 32:0.000143:s3:PUT /bucket3/:create_bucket:init op
2017-02-22 16:08:29.747525 7f60b2794700  2 req 32:0.000144:s3:PUT /bucket3/:create_bucket:verifying op mask
2017-02-22 16:08:29.747526 7f60b2794700  2 req 32:0.000145:s3:PUT /bucket3/:create_bucket:verifying op permissions
2017-02-22 16:08:29.748654 7f60b2794700  2 req 32:0.001272:s3:PUT /bucket3/:create_bucket:verifying op params
2017-02-22 16:08:29.748664 7f60b2794700  2 req 32:0.001283:s3:PUT /bucket3/:create_bucket:pre-executing
2017-02-22 16:08:29.748666 7f60b2794700  2 req 32:0.001285:s3:PUT /bucket3/:create_bucket:executing
2017-02-22 16:08:29.748712 7f60b2794700 10 payload request hash = d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-22 16:08:29.748728 7f60b2794700 10 canonical request = PUT
/bucket3/

host:node5:8082
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
x-amz-date:20170222T070829Z

host;x-amz-content-sha256;x-amz-date
d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-22 16:08:29.748729 7f60b2794700 10 canonical request hash = 9d6a0953485a6ed972ee50ae8a4ef7ca16d2f7b2748dfc6b02e7779b126ba0c0
2017-02-22 16:08:29.748732 7f60b2794700 10 string to sign = AWS4-HMAC-SHA256
20170222T070829Z
20170222/west/s3/aws4_request
9d6a0953485a6ed972ee50ae8a4ef7ca16d2f7b2748dfc6b02e7779b126ba0c0
2017-02-22 16:08:29.748759 7f60b2794700 10 date_k        = ad5444f7116af8aa782d6879e68977cd165288cdcb78c01fa4ea284a75a07b76
2017-02-22 16:08:29.748778 7f60b2794700 10 region_k      = 8faf00bd25092a41a7332d76e31426eecb9171835f4e7b3ccdc7def8074cb72d
2017-02-22 16:08:29.748794 7f60b2794700 10 service_k     = f1ba1b42c778178162919b9de68fa5e771a0b34f5f8b6d3a50261cf243448b20
2017-02-22 16:08:29.748811 7f60b2794700 10 signing_k     = 5e166a8f00366d1e85579744e69cea8ffa236aa82c5ace730750b8e1120a1373
2017-02-22 16:08:29.748840 7f60b2794700 10 signature_k   = 95b97d254f830017f12778229e226c4ce9061d1917fff2e8bb899fd563f5e21b
2017-02-22 16:08:29.748845 7f60b2794700 10 new signature = 95b97d254f830017f12778229e226c4ce9061d1917fff2e8bb899fd563f5e21b
2017-02-22 16:08:29.748845 7f60b2794700 10 ----------------------------- Verifying signatures
2017-02-22 16:08:29.748846 7f60b2794700 10 Signature     = 95b97d254f830017f12778229e226c4ce9061d1917fff2e8bb899fd563f5e21b
2017-02-22 16:08:29.748847 7f60b2794700 10 New Signature = 95b97d254f830017f12778229e226c4ce9061d1917fff2e8bb899fd563f5e21b
2017-02-22 16:08:29.748847 7f60b2794700 10 -----------------------------
2017-02-22 16:08:29.748849 7f60b2794700 10 v4 auth ok
2017-02-22 16:08:29.748882 7f60b2794700 10 create bucket location constraint: west
2017-02-22 16:08:29.748901 7f60b2794700 10 cache get: name=osaka.rgw.data.root+bucket3 : miss
2017-02-22 16:08:29.749484 7f60b2794700 10 cache put: name=osaka.rgw.data.root+bucket3 info.flags=0
2017-02-22 16:08:29.749495 7f60b2794700 10 adding osaka.rgw.data.root+bucket3 to cache LRU end
2017-02-22 16:08:29.749500 7f60b2794700  0 sending create_bucket request to master zonegroup
2017-02-22 16:08:29.749520 7f60b2794700 10 get_canon_resource(): dest=/bucket3/
2017-02-22 16:08:29.749525 7f60b2794700 10 generated canonical header: PUT

Wed Feb 22 07:08:29 2017
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
/bucket3/
2017-02-22 16:08:29.755519 7f60b2794700 10 receive_http_header
2017-02-22 16:08:29.755527 7f60b2794700 10 received header:HTTP/1.1 100 CONTINUE
2017-02-22 16:08:29.755528 7f60b2794700 10 received header:HTTP/1.1
2017-02-22 16:08:29.755529 7f60b2794700 10 receive_http_header
2017-02-22 16:08:29.755529 7f60b2794700 10 received header:
2017-02-22 16:08:29.757026 7f60b2794700 10 receive_http_header
2017-02-22 16:08:29.757034 7f60b2794700 10 received header:HTTP/1.1 409 Conflict
2017-02-22 16:08:29.757037 7f60b2794700 10 receive_http_header
2017-02-22 16:08:29.757038 7f60b2794700 10 received header:x-amz-request-id: tx0000000000000000015b6-0058ad38ed-1582-jp-east
2017-02-22 16:08:29.757060 7f60b2794700 10 receive_http_header
2017-02-22 16:08:29.757062 7f60b2794700 10 received header:Content-Length: 219
2017-02-22 16:08:29.757065 7f60b2794700 10 receive_http_header
2017-02-22 16:08:29.757067 7f60b2794700 10 received header:Accept-Ranges: bytes
2017-02-22 16:08:29.757070 7f60b2794700 10 receive_http_header
2017-02-22 16:08:29.757071 7f60b2794700 10 received header:Content-Type: application/xml
2017-02-22 16:08:29.757073 7f60b2794700 10 receive_http_header
2017-02-22 16:08:29.757074 7f60b2794700 10 received header:Date: Wed, 22 Feb 2017 07:08:29 GMT
2017-02-22 16:08:29.757076 7f60b2794700 10 receive_http_header
2017-02-22 16:08:29.757076 7f60b2794700 10 received header:
2017-02-22 16:08:29.796469 7f60b2794700  2 req 32:0.049088:s3:PUT /bucket3/:create_bucket:completing
2017-02-22 16:08:29.796528 7f60b2794700  2 req 32:0.049147:s3:PUT /bucket3/:create_bucket:op status=-39
2017-02-22 16:08:29.796533 7f60b2794700  2 req 32:0.049152:s3:PUT /bucket3/:create_bucket:http status=409
2017-02-22 16:08:29.796536 7f60b2794700  1 ====== req done req=0x7f60b278e710 op status=-39 http_status=409 ======
2017-02-22 16:08:29.796559 7f60b2794700  1 civetweb: 0x7f60f406a7b0: 192.168.20.15 - - [22/Feb/2017:16:08:29 +0900] "PUT /bucket3/ HTTP/1.1" 409 0 - -

ceph-client.radosgw.jp-east.log:

2017-02-22 16:08:29.754151 7f8c7dcd0700  1 ====== starting new request req=0x7f8c7dcca710 =====
2017-02-22 16:08:29.754197 7f8c7dcd0700  2 req 5558:0.000046::PUT /bucket3/::initializing for trans_id = tx0000000000000000015b6-0058ad38ed-1582-jp-east
2017-02-22 16:08:29.754210 7f8c7dcd0700 10 rgw api priority: s3=5 s3website=4
2017-02-22 16:08:29.754213 7f8c7dcd0700 10 host=node5
2017-02-22 16:08:29.754228 7f8c7dcd0700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2017-02-22 16:08:29.754235 7f8c7dcd0700 10 x>> x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-22 16:08:29.754267 7f8c7dcd0700 10 handler=25RGWHandler_REST_Bucket_S3
2017-02-22 16:08:29.754280 7f8c7dcd0700  2 req 5558:0.000130:s3:PUT /bucket3/::getting op 1
2017-02-22 16:08:29.754292 7f8c7dcd0700 10 op=27RGWCreateBucket_ObjStore_S3
2017-02-22 16:08:29.754295 7f8c7dcd0700  2 req 5558:0.000145:s3:PUT /bucket3/:create_bucket:authorizing
2017-02-22 16:08:29.754352 7f8c7dcd0700 10 get_canon_resource(): dest=/bucket3/
2017-02-22 16:08:29.754359 7f8c7dcd0700 10 auth_hdr:
PUT

Wed Feb 22 07:08:29 2017
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
/bucket3/
2017-02-22 16:08:29.754406 7f8c7dcd0700 10 cache get: name=jp-east.rgw.users.uid+nishi : hit (requested=6, cached=23)
2017-02-22 16:08:29.754418 7f8c7dcd0700 10 cache get: name=jp-east.rgw.users.uid+nishi : hit (requested=1, cached=23)
2017-02-22 16:08:29.754432 7f8c7dcd0700  2 req 5558:0.000282:s3:PUT /bucket3/:create_bucket:normalizing buckets and tenants
2017-02-22 16:08:29.754438 7f8c7dcd0700 10 s->object=<NULL> s->bucket=bucket3
2017-02-22 16:08:29.754439 7f8c7dcd0700  2 req 5558:0.000289:s3:PUT /bucket3/:create_bucket:init permissions
2017-02-22 16:08:29.754440 7f8c7dcd0700  2 req 5558:0.000290:s3:PUT /bucket3/:create_bucket:recalculating target
2017-02-22 16:08:29.754442 7f8c7dcd0700  2 req 5558:0.000292:s3:PUT /bucket3/:create_bucket:reading permissions
2017-02-22 16:08:29.754444 7f8c7dcd0700  2 req 5558:0.000294:s3:PUT /bucket3/:create_bucket:init op
2017-02-22 16:08:29.754445 7f8c7dcd0700  2 req 5558:0.000295:s3:PUT /bucket3/:create_bucket:verifying op mask
2017-02-22 16:08:29.754447 7f8c7dcd0700  2 req 5558:0.000297:s3:PUT /bucket3/:create_bucket:verifying op permissions
2017-02-22 16:08:29.755449 7f8c7dcd0700  2 req 5558:0.001299:s3:PUT /bucket3/:create_bucket:verifying op params
2017-02-22 16:08:29.755459 7f8c7dcd0700  2 req 5558:0.001309:s3:PUT /bucket3/:create_bucket:pre-executing
2017-02-22 16:08:29.755483 7f8c7dcd0700  2 req 5558:0.001333:s3:PUT /bucket3/:create_bucket:executing
2017-02-22 16:08:29.756066 7f8c7dcd0700 10 create bucket location constraint: west
2017-02-22 16:08:29.756086 7f8c7dcd0700 10 cache get: name=jp-east.rgw.data.root+bucket3 : type miss (requested=22, cached=19)
2017-02-22 16:08:29.756854 7f8c7dcd0700 10 cache put: name=jp-east.rgw.data.root+bucket3 info.flags=22
2017-02-22 16:08:29.756863 7f8c7dcd0700 10 moving jp-east.rgw.data.root+bucket3 to cache LRU end
2017-02-22 16:08:29.756869 7f8c7dcd0700 10 cache get: name=jp-east.rgw.data.root+bucket3 : hit (requested=17, cached=23)
2017-02-22 16:08:29.756883 7f8c7dcd0700 10 cache get: name=jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5404.2 : hit (requested=22, cached=23)
2017-02-22 16:08:29.756892 7f8c7dcd0700 10 cache get: name=jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5404.2 : hit (requested=17, cached=23)
2017-02-22 16:08:29.756899 7f8c7dcd0700 10 chain_cache_entry: cache_locator=jp-east.rgw.data.root+bucket3
2017-02-22 16:08:29.756901 7f8c7dcd0700 10 chain_cache_entry: cache_locator=jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5404.2
2017-02-22 16:08:29.756908 7f8c7dcd0700  0 WARNING: couldn't find acl header for object, generating default
2017-02-22 16:08:29.756915 7f8c7dcd0700 10 cache get: name=jp-east.rgw.users.uid+test : hit (requested=6, cached=23)
2017-02-22 16:08:29.756919 7f8c7dcd0700 10 cache get: name=jp-east.rgw.users.uid+test : hit (requested=1, cached=23)
2017-02-22 16:08:29.756929 7f8c7dcd0700  2 req 5558:0.002779:s3:PUT /bucket3/:create_bucket:completing
2017-02-22 16:08:29.756978 7f8c7dcd0700  2 req 5558:0.002828:s3:PUT /bucket3/:create_bucket:op status=-17
2017-02-22 16:08:29.756985 7f8c7dcd0700  2 req 5558:0.002835:s3:PUT /bucket3/:create_bucket:http status=409
2017-02-22 16:08:29.756988 7f8c7dcd0700  1 ====== req done req=0x7f8c7dcca710 op status=-17 http_status=409 ======
2017-02-22 16:08:29.757009 7f8c7dcd0700  1 civetweb: 0x7f8ccc0048c0: 192.168.20.15 - - [22/Feb/2017:16:08:29 +0900] "PUT /bucket3/ HTTP/1.1" 409 0 - -

ceph-client.radosgw.gateway.log:

2017-02-22 16:09:13.882427 7fab48fd9700  1 ====== starting new request req=0x7fab48fd3710 =====
2017-02-22 16:09:13.882451 7fab48fd9700  2 req 3:0.000025::PUT /bucket3/::initializing for trans_id = tx000000000000000000003-0058ad3919-15ef-default
2017-02-22 16:09:13.882460 7fab48fd9700 10 rgw api priority: s3=5 s3website=4
2017-02-22 16:09:13.882461 7fab48fd9700 10 host=node5
2017-02-22 16:09:13.882470 7fab48fd9700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2017-02-22 16:09:13.882476 7fab48fd9700 10 meta>> HTTP_X_AMZ_DATE
2017-02-22 16:09:13.882478 7fab48fd9700 10 x>> x-amz-content-sha256:2ffcfc8f4f0c7f98a2ab2f86264f388d2fce13bb0f5591f1e9b9560e2d68a614
2017-02-22 16:09:13.882479 7fab48fd9700 10 x>> x-amz-date:20170222T070913Z
2017-02-22 16:09:13.882506 7fab48fd9700 10 handler=25RGWHandler_REST_Bucket_S3
2017-02-22 16:09:13.882508 7fab48fd9700  2 req 3:0.000082:s3:PUT /bucket3/::getting op 1
2017-02-22 16:09:13.882514 7fab48fd9700 10 op=27RGWCreateBucket_ObjStore_S3
2017-02-22 16:09:13.882515 7fab48fd9700  2 req 3:0.000090:s3:PUT /bucket3/:create_bucket:authorizing
2017-02-22 16:09:13.882532 7fab48fd9700 10 v4 signature format = 5587ed62219d23f25134061d92ce4fbf5acaa2438c5ca025aae93a6ee6b55fba
2017-02-22 16:09:13.882538 7fab48fd9700 10 v4 credential format = A4DS3XOFE1L5KM5ZX2LL/20170222/east/s3/aws4_request
2017-02-22 16:09:13.882542 7fab48fd9700 10 access key id = A4DS3XOFE1L5KM5ZX2LL
2017-02-22 16:09:13.882543 7fab48fd9700 10 credential scope = 20170222/east/s3/aws4_request
2017-02-22 16:09:13.882568 7fab48fd9700 10 canonical headers format = host:node5
x-amz-content-sha256:2ffcfc8f4f0c7f98a2ab2f86264f388d2fce13bb0f5591f1e9b9560e2d68a614
x-amz-date:20170222T070913Z

2017-02-22 16:09:13.882571 7fab48fd9700 10 delaying v4 auth
2017-02-22 16:09:13.882574 7fab48fd9700  2 req 3:0.000148:s3:PUT /bucket3/:create_bucket:normalizing buckets and tenants
2017-02-22 16:09:13.882576 7fab48fd9700 10 s->object=<NULL> s->bucket=bucket3
2017-02-22 16:09:13.882579 7fab48fd9700  2 req 3:0.000153:s3:PUT /bucket3/:create_bucket:init permissions
2017-02-22 16:09:13.882580 7fab48fd9700  2 req 3:0.000154:s3:PUT /bucket3/:create_bucket:recalculating target
2017-02-22 16:09:13.882581 7fab48fd9700  2 req 3:0.000155:s3:PUT /bucket3/:create_bucket:reading permissions
2017-02-22 16:09:13.882587 7fab48fd9700  2 req 3:0.000158:s3:PUT /bucket3/:create_bucket:init op
2017-02-22 16:09:13.882588 7fab48fd9700  2 req 3:0.000162:s3:PUT /bucket3/:create_bucket:verifying op mask
2017-02-22 16:09:13.882589 7fab48fd9700  2 req 3:0.000163:s3:PUT /bucket3/:create_bucket:verifying op permissions
2017-02-22 16:09:13.883734 7fab48fd9700  2 req 3:0.001307:s3:PUT /bucket3/:create_bucket:verifying op params
2017-02-22 16:09:13.883738 7fab48fd9700  2 req 3:0.001312:s3:PUT /bucket3/:create_bucket:pre-executing
2017-02-22 16:09:13.883739 7fab48fd9700  2 req 3:0.001313:s3:PUT /bucket3/:create_bucket:executing
2017-02-22 16:09:13.883792 7fab48fd9700 10 payload request hash = 2ffcfc8f4f0c7f98a2ab2f86264f388d2fce13bb0f5591f1e9b9560e2d68a614
2017-02-22 16:09:13.883804 7fab48fd9700 10 canonical request = PUT
/bucket3/

host:node5
x-amz-content-sha256:2ffcfc8f4f0c7f98a2ab2f86264f388d2fce13bb0f5591f1e9b9560e2d68a614
x-amz-date:20170222T070913Z

host;x-amz-content-sha256;x-amz-date
2ffcfc8f4f0c7f98a2ab2f86264f388d2fce13bb0f5591f1e9b9560e2d68a614
2017-02-22 16:09:13.883805 7fab48fd9700 10 canonical request hash = ca28000a2504103f749da3ad7862eb4ae3937ea4f07a6d863a6d29bc2979193d
2017-02-22 16:09:13.883808 7fab48fd9700 10 string to sign = AWS4-HMAC-SHA256
20170222T070913Z
20170222/east/s3/aws4_request
ca28000a2504103f749da3ad7862eb4ae3937ea4f07a6d863a6d29bc2979193d
2017-02-22 16:09:13.883835 7fab48fd9700 10 date_k        = 451f0bb57aabb396f838924c00f93d672d9dfafab62a8cd79c9200f583623047
2017-02-22 16:09:13.883848 7fab48fd9700 10 region_k      = 6e9970204a9ba8dc62288c9adeaf09aa7fa54eacb4de795fb33706814922cffb
2017-02-22 16:09:13.883860 7fab48fd9700 10 service_k     = 25de605c7dd13a1bd41444d1e3bbc81885e7d2c00df21fea1c7b32e361e73008
2017-02-22 16:09:13.883872 7fab48fd9700 10 signing_k     = bd8004fc2aef7c121f1a09eda897977ad300aabef1914631baa555d58d09da36
2017-02-22 16:09:13.883906 7fab48fd9700 10 signature_k   = 5587ed62219d23f25134061d92ce4fbf5acaa2438c5ca025aae93a6ee6b55fba
2017-02-22 16:09:13.883907 7fab48fd9700 10 new signature = 5587ed62219d23f25134061d92ce4fbf5acaa2438c5ca025aae93a6ee6b55fba
2017-02-22 16:09:13.883907 7fab48fd9700 10 ----------------------------- Verifying signatures
2017-02-22 16:09:13.883908 7fab48fd9700 10 Signature     = 5587ed62219d23f25134061d92ce4fbf5acaa2438c5ca025aae93a6ee6b55fba
2017-02-22 16:09:13.883908 7fab48fd9700 10 New Signature = 5587ed62219d23f25134061d92ce4fbf5acaa2438c5ca025aae93a6ee6b55fba
2017-02-22 16:09:13.883908 7fab48fd9700 10 -----------------------------
2017-02-22 16:09:13.883910 7fab48fd9700 10 v4 auth ok
2017-02-22 16:09:13.883973 7fab48fd9700 10 create bucket location constraint: east
2017-02-22 16:09:13.883990 7fab48fd9700 10 cache get: name=default.rgw.data.root+bucket3 : miss
2017-02-22 16:09:13.885599 7fab48fd9700 10 cache put: name=default.rgw.data.root+bucket3 info.flags=0
2017-02-22 16:09:13.885613 7fab48fd9700 10 adding default.rgw.data.root+bucket3 to cache LRU end
2017-02-22 16:09:13.885618 7fab48fd9700  0 sending create_bucket request to master zonegroup
2017-02-22 16:09:13.885657 7fab48fd9700 10 get_canon_resource(): dest=/bucket3/
2017-02-22 16:09:13.885664 7fab48fd9700 10 generated canonical header: PUT

Wed Feb 22 07:09:13 2017
x-amz-content-sha256:2ffcfc8f4f0c7f98a2ab2f86264f388d2fce13bb0f5591f1e9b9560e2d68a614
/bucket3/
2017-02-22 16:09:13.891419 7fab48fd9700 10 receive_http_header
2017-02-22 16:09:13.891425 7fab48fd9700 10 received header:HTTP/1.1 100 CONTINUE
2017-02-22 16:09:13.891426 7fab48fd9700 10 received header:HTTP/1.1
2017-02-22 16:09:13.891427 7fab48fd9700 10 receive_http_header
2017-02-22 16:09:13.891427 7fab48fd9700 10 received header:
2017-02-22 16:09:13.919534 7fab48fd9700 10 receive_http_header
2017-02-22 16:09:13.919544 7fab48fd9700 10 received header:HTTP/1.1 200 OK
2017-02-22 16:09:13.919546 7fab48fd9700 10 receive_http_header
2017-02-22 16:09:13.919547 7fab48fd9700 10 received header:x-amz-request-id: tx00000000000000000173a-0058ad3919-1582-jp-east
2017-02-22 16:09:13.919555 7fab48fd9700 10 receive_http_header
2017-02-22 16:09:13.919555 7fab48fd9700 10 received header:Content-Length: 778
2017-02-22 16:09:13.919558 7fab48fd9700 10 receive_http_header
2017-02-22 16:09:13.919558 7fab48fd9700 10 received header:Date: Wed, 22 Feb 2017 07:09:13 GMT
2017-02-22 16:09:13.919560 7fab48fd9700 10 receive_http_header
2017-02-22 16:09:13.919560 7fab48fd9700 10 received header:
2017-02-22 16:09:13.967481 7fab48fd9700 10 cache put: name=default.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5404.2 info.flags=23
2017-02-22 16:09:13.967488 7fab48fd9700 10 adding default.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5404.2 to cache LRU end
2017-02-22 16:09:13.967490 7fab48fd9700 10 updating xattr: name=user.rgw.acl bl.length()=131
2017-02-22 16:09:13.967500 7fab48fd9700 10 distributing notification oid=notify.1 bl.length()=816
2017-02-22 16:09:13.968151 7fab9affd700 10 RGWWatcher::handle_notify()  notify_id 755914244112 cookie 140375653508400 notifier 5615 bl.length()=816
2017-02-22 16:09:13.968166 7fab9affd700 10 cache put: name=default.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5404.2 info.flags=23
2017-02-22 16:09:13.968169 7fab9affd700 10 moving default.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5404.2 to cache LRU end
2017-02-22 16:09:13.968172 7fab9affd700 10 updating xattr: name=user.rgw.acl bl.length()=131
2017-02-22 16:09:13.971868 7fab48fd9700 10 cache put: name=default.rgw.data.root+bucket3 info.flags=23
2017-02-22 16:09:13.971872 7fab48fd9700 10 moving default.rgw.data.root+bucket3 to cache LRU end
2017-02-22 16:09:13.971881 7fab48fd9700 10 distributing notification oid=notify.6 bl.length()=522
2017-02-22 16:09:13.972549 7fab9affd700 10 RGWWatcher::handle_notify()  notify_id 755914244102 cookie 140375653569232 notifier 5615 bl.length()=522
2017-02-22 16:09:13.972563 7fab9affd700 10 cache put: name=default.rgw.data.root+bucket3 info.flags=23
2017-02-22 16:09:13.972570 7fab9affd700 10 moving default.rgw.data.root+bucket3 to cache LRU end
2017-02-22 16:09:13.976403 7fab48fd9700  2 req 3:0.093977:s3:PUT /bucket3/:create_bucket:completing
2017-02-22 16:09:13.976464 7fab48fd9700  2 req 3:0.094038:s3:PUT /bucket3/:create_bucket:op status=0
2017-02-22 16:09:13.976470 7fab48fd9700  2 req 3:0.094044:s3:PUT /bucket3/:create_bucket:http status=200
2017-02-22 16:09:13.976476 7fab48fd9700  1 ====== req done req=0x7fab48fd3710 op status=0 http_status=200 ======
2017-02-22 16:09:13.976505 7fab48fd9700  1 civetweb: 0x7fab5c0040b0: 192.168.20.15 - - [22/Feb/2017:16:09:13 +0900] "PUT /bucket3/ HTTP/1.1" 200 0 - -

ceph-client.radosgw.jp-east.log:

2017-02-22 16:09:13.890198 7f8c6449d700  1 ====== starting new request req=0x7f8c64497710 =====
2017-02-22 16:09:13.890218 7f8c6449d700  2 req 5946:0.000022::PUT /bucket3/::initializing for trans_id = tx00000000000000000173a-0058ad3919-1582-jp-east
2017-02-22 16:09:13.890225 7f8c6449d700 10 rgw api priority: s3=5 s3website=4
2017-02-22 16:09:13.890226 7f8c6449d700 10 host=node5
2017-02-22 16:09:13.890234 7f8c6449d700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2017-02-22 16:09:13.890237 7f8c6449d700 10 x>> x-amz-content-sha256:2ffcfc8f4f0c7f98a2ab2f86264f388d2fce13bb0f5591f1e9b9560e2d68a614
2017-02-22 16:09:13.890253 7f8c6449d700 10 handler=25RGWHandler_REST_Bucket_S3
2017-02-22 16:09:13.890255 7f8c6449d700  2 req 5946:0.000058:s3:PUT /bucket3/::getting op 1
2017-02-22 16:09:13.890265 7f8c6449d700 10 op=27RGWCreateBucket_ObjStore_S3
2017-02-22 16:09:13.890266 7f8c6449d700  2 req 5946:0.000070:s3:PUT /bucket3/:create_bucket:authorizing
2017-02-22 16:09:13.890286 7f8c6449d700 10 get_canon_resource(): dest=/bucket3/
2017-02-22 16:09:13.890288 7f8c6449d700 10 auth_hdr:
PUT

Wed Feb 22 07:09:13 2017
x-amz-content-sha256:2ffcfc8f4f0c7f98a2ab2f86264f388d2fce13bb0f5591f1e9b9560e2d68a614
/bucket3/
2017-02-22 16:09:13.890346 7f8c6449d700 10 cache get: name=jp-east.rgw.users.uid+test : hit (requested=6, cached=23)
2017-02-22 16:09:13.890354 7f8c6449d700 10 cache get: name=jp-east.rgw.users.uid+test : hit (requested=1, cached=23)
2017-02-22 16:09:13.890370 7f8c6449d700  2 req 5946:0.000174:s3:PUT /bucket3/:create_bucket:normalizing buckets and tenants
2017-02-22 16:09:13.890373 7f8c6449d700 10 s->object=<NULL> s->bucket=bucket3
2017-02-22 16:09:13.890374 7f8c6449d700  2 req 5946:0.000178:s3:PUT /bucket3/:create_bucket:init permissions
2017-02-22 16:09:13.890376 7f8c6449d700  2 req 5946:0.000180:s3:PUT /bucket3/:create_bucket:recalculating target
2017-02-22 16:09:13.890377 7f8c6449d700  2 req 5946:0.000181:s3:PUT /bucket3/:create_bucket:reading permissions
2017-02-22 16:09:13.890379 7f8c6449d700  2 req 5946:0.000183:s3:PUT /bucket3/:create_bucket:init op
2017-02-22 16:09:13.890381 7f8c6449d700  2 req 5946:0.000185:s3:PUT /bucket3/:create_bucket:verifying op mask
2017-02-22 16:09:13.890382 7f8c6449d700  2 req 5946:0.000186:s3:PUT /bucket3/:create_bucket:verifying op permissions
2017-02-22 16:09:13.891349 7f8c6449d700  2 req 5946:0.001150:s3:PUT /bucket3/:create_bucket:verifying op params
2017-02-22 16:09:13.891359 7f8c6449d700  2 req 5946:0.001162:s3:PUT /bucket3/:create_bucket:pre-executing
2017-02-22 16:09:13.891384 7f8c6449d700  2 req 5946:0.001187:s3:PUT /bucket3/:create_bucket:executing
2017-02-22 16:09:13.891524 7f8c6449d700 10 create bucket location constraint: east
2017-02-22 16:09:13.892161 7f8c6449d700  0 WARNING: couldn't find acl header for object, generating default
2017-02-22 16:09:13.892177 7f8c6449d700 10 cache get: name=jp-east.rgw.users.uid+test : hit (requested=6, cached=23)
2017-02-22 16:09:13.892182 7f8c6449d700 10 cache get: name=jp-east.rgw.users.uid+test : hit (requested=1, cached=23)
2017-02-22 16:09:13.901781 7f8c6449d700 10 cache put: name=jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5506.2 info.flags=23
2017-02-22 16:09:13.901798 7f8c6449d700 10 adding jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5506.2 to cache LRU end
2017-02-22 16:09:13.901801 7f8c6449d700 10 updating xattr: name=user.rgw.acl bl.length()=153
2017-02-22 16:09:13.901814 7f8c6449d700 10 distributing notification oid=notify.2 bl.length()=838
2017-02-22 16:09:13.902595 7f8cc3fff700 10 RGWWatcher::handle_notify()  notify_id 755914244134 cookie 140243475537840 notifier 5506 bl.length()=838
2017-02-22 16:09:13.902612 7f8cc3fff700 10 cache put: name=jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5506.2 info.flags=23
2017-02-22 16:09:13.902615 7f8cc3fff700 10 moving jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5506.2 to cache LRU end
2017-02-22 16:09:13.902618 7f8cc3fff700 10 updating xattr: name=user.rgw.acl bl.length()=153
2017-02-22 16:09:13.907034 7f8c6449d700 10 removing jp-east.rgw.data.root+bucket3 from cache
2017-02-22 16:09:13.908541 7f8c6449d700 10 cache get: name=jp-east.rgw.data.root+bucket3 : miss
2017-02-22 16:09:13.909050 7f8c6449d700 10 cache put: name=jp-east.rgw.data.root+bucket3 info.flags=22
2017-02-22 16:09:13.909061 7f8c6449d700 10 adding jp-east.rgw.data.root+bucket3 to cache LRU end
2017-02-22 16:09:13.909066 7f8c6449d700 10 cache get: name=jp-east.rgw.data.root+bucket3 : type miss (requested=17, cached=22)
2017-02-22 16:09:13.909624 7f8c6449d700 10 cache put: name=jp-east.rgw.data.root+bucket3 info.flags=17
2017-02-22 16:09:13.909628 7f8c6449d700 10 moving jp-east.rgw.data.root+bucket3 to cache LRU end
2017-02-22 16:09:13.909643 7f8c6449d700 10 cache get: name=jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5404.2 : hit (requested=22, cached=23)
2017-02-22 16:09:13.909650 7f8c6449d700 10 cache get: name=jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5404.2 : hit (requested=17, cached=23)
2017-02-22 16:09:13.909659 7f8c6449d700 10 chain_cache_entry: cache_locator=jp-east.rgw.data.root+bucket3
2017-02-22 16:09:13.909666 7f8c6449d700 10 chain_cache_entry: cache_locator=jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5404.2
2017-02-22 16:09:13.909680 7f8c6449d700 10 cache get: name=jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5506.2 : hit (requested=22, cached=23)
2017-02-22 16:09:13.909684 7f8c6449d700 10 cache get: name=jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5506.2 : hit (requested=17, cached=23)
2017-02-22 16:09:13.911280 7f8c6449d700 10 removing jp-east.rgw.data.root+.bucket.meta.bucket3:22719b44-532d-41ee-974e-fc89cb93255f.5506.2 from cache
2017-02-22 16:09:13.911296 7f8c6449d700 10 distributing notification oid=notify.2 bl.length()=304
2017-02-22 16:09:13.911933 7f8cc3fff700 10 RGWWatcher::handle_notify()  notify_id 755914244135 cookie 140243475537840 notifier 5506 bl.length()=304
2017-02-22 16:09:13.919399 7f8c6449d700  2 req 5946:0.029202:s3:PUT /bucket3/:create_bucket:completing
2017-02-22 16:09:13.919480 7f8c6449d700  2 req 5946:0.029283:s3:PUT /bucket3/:create_bucket:op status=0
2017-02-22 16:09:13.919502 7f8c6449d700  2 req 5946:0.029305:s3:PUT /bucket3/:create_bucket:http status=200
2017-02-22 16:09:13.919508 7f8c6449d700  1 ====== req done req=0x7f8c64497710 op status=0 http_status=200 ======
2017-02-22 16:09:13.919534 7f8c6449d700  1 civetweb: 0x7f8cb0098e30: 192.168.20.15 - - [22/Feb/2017:16:09:13 +0900] "PUT /bucket3/ HTTP/1.1" 200 0 - -

Files

Download all files
plantuml_zone_config.png (11.2 KB) plantuml_zone_config.png zone/zonegroup configuration Osamu KIMURA, 03/01/2017 04:42 AM
plantuml_bucket_sync.png (41 KB) plantuml_bucket_sync.png sequence chart (bucket metadata sync) Osamu KIMURA, 03/01/2017 04:42 AM
plantuml_bucket_create3.png (30.1 KB) plantuml_bucket_create3.png sequence chart (can't create a bucket with same name) Osamu KIMURA, 03/01/2017 04:43 AM
plantuml_bucket_create4.png (23.2 KB) plantuml_bucket_create4.png sequence chart (can create buckets with same name in different zonegroup) Osamu KIMURA, 03/01/2017 04:44 AM
Actions
Copy link
 #1

Updated by Yehuda Sadeh about 7 years ago

  • Assignee set to Orit Wasserman
Actions
Copy link Download all files
 #2

Updated by Osamu KIMURA about 7 years ago

I stick sequence charts created by PlantUML plugin

zone/zonegroup configuration

Basically, bucket metadata is not synchronized to secondary zonegroup.
sequence chart (bucket metadata sync)

A case that creating bucket with same name is rejected.
sequence chart (can't create a bucket with same name)

A case that creating bucket with same name is accepted.
sequence chart (can create buckets with same name in different zonegroup)

Actions
Copy link

Also available in: Atom PDF