Ceph » rgw

Hello everyone, Thank you very much community created ceph.
I have a multisite question to ask. Why does not the same realm under the multiple zone group metadata synchronization.There may be a problem with my configuration.Please give me some tips community.
I saw this on the site(A zone realm is a collection of zonegroups that share the same user and bucket namespace.):
http://tracker.ceph.com/projects/ceph/wiki/Rgw_new_multisite_configuration

My two cluster addresses:
192.168.250.68 realm is earth,zonegroup is cn,zone is cn-sy-south.
192.168.250.57 realm is earth,zonegroup is us,zone is us-tx-east.

192.168.250.68 Configuration operation
radosgw-admin realm pull --url=http://192.168.250.68:80 --access-key=admin --secret=admin
radosgw-admin period pull --url=http://192.168.250.68:80 --access-key=admin --secret=admin
radosgw-admin realm default --rgw-realm=earth
radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://192.168.250.57:80 --default
radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-tx-east --endpoints=http://192.168.250.57:80 --access-key=admin --secret=admin --default --master
radosgw-admin period update --commit --url=http://192.168.250.68:80 --access-key=admin --secret=admin
/etc/init.d/radosgw start

192.168.250.57 Configuration operation
radosgw-admin realm pull --url=http://192.168.250.68:80 --access-key=admin --secret=admin
radosgw-admin period pull --url=http://192.168.250.68:80 --access-key=admin --secret=admin
radosgw-admin realm default --rgw-realm=earth
radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://192.168.250.57:80 --default
radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-tx-east --endpoints=http://192.168.250.57:80 --access-key=admin --secret=admin --default --master
radosgw-admin period update --commit --url=http://192.168.250.68:80 --access-key=admin --secret=admin
/etc/init.d/radosgw start

192.168.250.68 radosgw log
2016-12-23 17:09:54.454316 7f3634ff9700 20 cr:s=0x7f35fc001520:op=0x7f35fc000c90:14PushAndRetryCR: operate()
2016-12-23 17:09:54.454328 7f3634ff9700 10 rgw period pusher: pushing period 31bfaf25-2e87-4920-8478-bc2b8804ccf0 to b217b5e7-3eeb-4a53-9f1b-ad51798703ae
2016-12-23 17:09:54.454433 7f3634ff9700 20 cr:s=0x7f35fc001520:op=0x7f35fc00d4a0:21RGWPostRESTResourceCRI9RGWPeriodiE: operate()
2016-12-23 17:09:54.454437 7f3634ff9700 20 cr:s=0x7f35fc001520:op=0x7f35fc00d4a0:21RGWPostRESTResourceCRI9RGWPeriodiE: operate()
2016-12-23 17:09:54.454601 7f3634ff9700 20 RGWEnv::set(): HTTP_DATE: Fri Dec 23 09:09:54 2016
2016-12-23 17:09:54.454608 7f3634ff9700 20 > HTTP_DATE > Fri Dec 23 09:09:54 2016
2016-12-23 17:09:54.454616 7f3634ff9700 10 get_canon_resource(): dest=/admin/realm/period
2016-12-23 17:09:54.454617 7f3634ff9700 10 generated canonical header: POST
Fri Dec 23 09:09:54 2016
/admin/realm/period
2016-12-23 17:09:54.454658 7f3634ff9700 15 generated auth header: AWS admin:ICaJTi1RTgVgysVKsIVymNeFoKM=
2016-12-23 17:09:54.454672 7f3634ff9700 20 sending request to http://192.168.250.57:80/admin/realm/period?period=31bfaf25-2e87-4920-8478-bc2b8804ccf0&epoch=2&rgwx-zonegroup=1335bb49-51a5-45fc-8c47-eb1fa9335759
2016-12-23 17:09:54.454685 7f3634ff9700 20 register_request mgr=0x7f3608001e70 req_data>id=467, easy_handle=0x7f35fc058f80
2016-12-23 17:09:54.454701 7f3634ff9700 20 run: stack=0x7f35fc001520 is io blocked
2016-12-23 17:09:54.454704 7f3603fff700 20 link_request req_data=0x7f35fc001f50 req_data->id=467, easy_handle=0x7f35fc058f80
2016-12-23 17:09:54.456895 7f3603fff700 10 receive_http_header
2016-12-23 17:09:54.456900 7f3603fff700 10 received header:HTTP/1.1 403 Forbidden
2016-12-23 17:09:54.456902 7f3603fff700 10 receive_http_header
2016-12-23 17:09:54.456903 7f3603fff700 10 received header:x-amz-request-id: tx0000000000000000001d3-00585ce9dd-38de-us-tx-east
2016-12-23 17:09:54.456908 7f3603fff700 10 receive_http_header
2016-12-23 17:09:54.456908 7f3603fff700 10 received header:Content-Length: 124
2016-12-23 17:09:54.456911 7f3603fff700 10 receive_http_header
2016-12-23 17:09:54.456912 7f3603fff700 10 received header:Accept-Ranges: bytes
2016-12-23 17:09:54.456914 7f3603fff700 10 receive_http_header
2016-12-23 17:09:54.456914 7f3603fff700 10 received header:Content-Type: application/json
2016-12-23 17:09:54.456916 7f3603fff700 10 receive_http_header
2016-12-23 17:09:54.456917 7f3603fff700 10 received header:Date: Fri, 23 Dec 2016 09:09:49 GMT
2016-12-23 17:09:54.456918 7f3603fff700 10 receive_http_header
2016-12-23 17:09:54.456919 7f3603fff700 10 received header:
2016-12-23 17:09:54.457079 7f3634ff9700 20 cr:s=0x7f35fc001520:op=0x7f35fc00d4a0:21RGWPostRESTResourceCRI9RGWPeriodiE: operate()
2016-12-23 17:09:54.457092 7f3634ff9700 5 failed to wait for op, ret=-13: POST http://192.168.250.57:80/admin/realm/period?period=31bfaf25-2e87-4920-8478-bc2b8804ccf0&epoch=2&rgwx-zonegroup=1335bb49-51a5-45fc-8c47-eb1fa9335759
2016-12-23 17:09:54.457127 7f3634ff9700 20 cr:s=0x7f35fc001520:op=0x7f35fc00d4a0:21RGWPostRESTResourceCRI9RGWPeriodiE: operate() returned r=-13
2016-12-23 17:09:54.457140 7f3634ff9700 20 cr:s=0x7f35fc001520:op=0x7f35fc000c90:14PushAndRetryCR: operate()
2016-12-23 17:09:54.457143 7f3634ff9700 10 rgw period pusher: waiting 30.000000s for retry..
2016-12-23 17:09:54.457151 7f3634ff9700 20 run: stack=0x7f35fc001520 is io blocked

192.168.250.57 radosgw log
2016-12-23 17:11:49.139901 7fdbc4ff9700 20 RGWEnv::set(): HTTP_HOST: 192.168.250.57
2016-12-23 17:11:49.139921 7fdbc4ff9700 20 RGWEnv::set(): HTTP_ACCEPT: /*
2016-12-23 17:11:49.139925 7fdbc4ff9700 20 RGWEnv::set(): HTTP_TRANSFER_ENCODING: chunked
2016-12-23 17:11:49.139930 7fdbc4ff9700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS admin:/Vtjm6gyk/t2+Db8wOjFWXiA2Kw=
2016-12-23 17:11:49.139933 7fdbc4ff9700 20 RGWEnv::set(): HTTP_DATE: Fri Dec 23 09:11:54 2016
2016-12-23 17:11:49.139936 7fdbc4ff9700 20 RGWEnv::set(): HTTP_EXPECT: 100-continue
2016-12-23 17:11:49.139938 7fdbc4ff9700 20 RGWEnv::set(): REQUEST_METHOD: POST
2016-12-23 17:11:49.139940 7fdbc4ff9700 20 RGWEnv::set(): REQUEST_URI: /admin/realm/period
2016-12-23 17:11:49.139942 7fdbc4ff9700 20 RGWEnv::set(): QUERY_STRING: period=31bfaf25-2e87-4920-8478-bc2b8804ccf0&epoch=2&rgwx-zonegroup=1335bb49-51a5-45fc-8c47-eb1fa9335759
2016-12-23 17:11:49.139945 7fdbc4ff9700 20 RGWEnv::set(): REMOTE_USER:
2016-12-23 17:11:49.139950 7fdbc4ff9700 20 RGWEnv::set(): SCRIPT_URI: /admin/realm/period
2016-12-23 17:11:49.139955 7fdbc4ff9700 20 RGWEnv::set(): SERVER_PORT: 80
2016-12-23 17:11:49.139957 7fdbc4ff9700 20 HTTP_ACCEPT=/*
2016-12-23 17:11:49.139958 7fdbc4ff9700 20 HTTP_AUTHORIZATION=AWS admin:/Vtjm6gyk/t2+Db8wOjFWXiA2Kw=
2016-12-23 17:11:49.139960 7fdbc4ff9700 20 HTTP_DATE=Fri Dec 23 09:11:54 2016
2016-12-23 17:11:49.139961 7fdbc4ff9700 20 HTTP_EXPECT=100-continue
2016-12-23 17:11:49.139966 7fdbc4ff9700 20 HTTP_HOST=192.168.250.57
2016-12-23 17:11:49.139968 7fdbc4ff9700 20 HTTP_TRANSFER_ENCODING=chunked
2016-12-23 17:11:49.139969 7fdbc4ff9700 20 QUERY_STRING=period=31bfaf25-2e87-4920-8478-bc2b8804ccf0&epoch=2&rgwx-zonegroup=1335bb49-51a5-45fc-8c47-eb1fa9335759
2016-12-23 17:11:49.139970 7fdbc4ff9700 20 REMOTE_USER=
2016-12-23 17:11:49.139972 7fdbc4ff9700 20 REQUEST_METHOD=POST
2016-12-23 17:11:49.139973 7fdbc4ff9700 20 REQUEST_URI=/admin/realm/period
2016-12-23 17:11:49.139974 7fdbc4ff9700 20 SCRIPT_URI=/admin/realm/period
2016-12-23 17:11:49.139976 7fdbc4ff9700 20 SERVER_PORT=80
2016-12-23 17:11:49.139979 7fdbc4ff9700 1 ====== starting new request req=0x7fdbc4ff37d0 =====
2016-12-23 17:11:49.140004 7fdbc4ff9700 2 req 471:0.000025::POST /admin/realm/period::initializing for trans_id = tx0000000000000000001d7-00585cea55-38de-us-tx-east
2016-12-23 17:11:49.140017 7fdbc4ff9700 10 rgw api priority: s3=5 s3website=4
2016-12-23 17:11:49.140020 7fdbc4ff9700 10 host=192.168.250.57
2016-12-23 17:11:49.140022 7fdbc4ff9700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2016-12-23 17:11:49.140026 7fdbc4ff9700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/admin/realm/period
2016-12-23 17:11:49.140070 7fdbc4ff9700 10 handler=17RGWHandler_Period
2016-12-23 17:11:49.140077 7fdbc4ff9700 2 req 471:0.000095::POST /admin/realm/period::getting op 4
2016-12-23 17:11:49.140084 7fdbc4ff9700 10 op=17RGWOp_Period_Post
2016-12-23 17:11:49.140085 7fdbc4ff9700 2 req 471:0.000107::POST /admin/realm/period:post_period:authorizing
2016-12-23 17:11:49.140107 7fdbc4ff9700 20 get_system_obj_state: rctx=0x7fdbc4ff2660 obj=us-tx-east.rgw.users.keys:admin state=0x7fdba00086b8 s->prefetch_data=0
2016-12-23 17:11:49.140118 7fdbc4ff9700 10 cache get: name=us-tx-east.rgw.users.keys+admin : type miss (requested=6, cached=0)
2016-12-23 17:11:49.141165 7fdbc4ff9700 10 cache put: name=us-tx-east.rgw.users.keys+admin info.flags=0
2016-12-23 17:11:49.141187 7fdbc4ff9700 10 moving us-tx-east.rgw.users.keys+admin to cache LRU end
2016-12-23 17:11:49.141201 7fdbc4ff9700 5 error reading user info, uid=admin can't authenticate
2016-12-23 17:11:49.141205 7fdbc4ff9700 10 failed to authorize request
2016-12-23 17:11:49.141208 7fdbc4ff9700 20 handler->ERRORHANDLER: err_no=-2028 new_err_no=-2028
2016-12-23 17:11:49.141308 7fdbc4ff9700 2 req 471:0.001328::POST /admin/realm/period:post_period:op status=0
2016-12-23 17:11:49.141314 7fdbc4ff9700 2 req 471:0.001336::POST /admin/realm/period:post_period:http status=403
2016-12-23 17:11:49.141320 7fdbc4ff9700 1 ====== req done req=0x7fdbc4ff37d0 op status=0 http_status=403 ======
2016-12-23 17:11:49.141333 7fdbc4ff9700 20 process_request() returned -2028
2016-12-23 17:11:49.141387 7fdbc4ff9700 1 civetweb: 0x7fdba00008c0: 192.168.250.68 - - [23/Dec/2016:17:11:49 +0800] "POST /admin/realm/period HTTP/1.1" 403 0 - -

I found 192.168.250.57 this machine does not metadata_heap,I manually modify the zone json, and then import. Still invalid.
radosgw-admin zone get {
"id": "100cce8e-8398-435c-b013-0b42f1c2ec8c",
"name": "us-tx-east",
"domain_root": "us-tx-east.rgw.data.root",
"control_pool": "us-tx-east.rgw.control",
"gc_pool": "us-tx-east.rgw.gc",
"log_pool": "us-tx-east.rgw.log",
"intent_log_pool": "us-tx-east.rgw.intent-log",
"usage_log_pool": "us-tx-east.rgw.usage",
"user_keys_pool": "us-tx-east.rgw.users.keys",
"user_email_pool": "us-tx-east.rgw.users.email",
"user_swift_pool": "us-tx-east.rgw.users.swift",
"user_uid_pool": "us-tx-east.rgw.users.uid",
"system_key": {
"access_key": "admin",
"secret_key": "admin"
},
"placement_pools": [ {
"key": "default-placement",
"val": {
"index_pool": "us-tx-east.rgw.buckets.index",
"data_pool": "us-tx-east.rgw.buckets.data",
"data_extra_pool": "us-tx-east.rgw.buckets.non-ec",
"index_type": 0
}
}
],
"metadata_heap": "",
"realm_id": "37d520aa-b700-4f9b-b017-65c20ee2e4b3"
}

Where is my problem?