Actions
Bug #19268
openCeilometer receives NoSuchBucket via Swift API
Added by Jens Offenbach about 7 years ago. Updated about 7 years ago.
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I am running OpenStack Ocata on Ubuntu 16.04. using Ceph Jewel (10.2.6) as common storage backend. The RGW Swift API is added to OpenStack's endpoint catalog. The object store is fully functional, but the Ceilometer log contains the following entry which occurrs every 10 minutes:
11:41:22.851 14501 INFO swiftclient [-] REQ: curl i http://os-objectstore.materna.com:7480/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac?format=json -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: 165af4078f66482b..." ] RESP STATUS: 404 Not Found
2017-03-13 11:41:22.851 14501 INFO swiftclient [
2017-03-13 11:41:22.851 14501 INFO swiftclient [-] RESP HEADERS: {u'Date': u'Mon, 13 Mar 2017 10:41:22 GMT', u'Content-Length': u'174', u'Content-Type': u'application/json; charset=utf-8', u'Accept-Ranges': u'bytes', u'X-Trans-Id': u'tx000000000000000000054-0058c67752-23a86f-default'}
2017-03-13 11:41:22.851 14501 INFO swiftclient [-] RESP BODY: {"Code":"NoSuchBucket","BucketName":"AUTH_53eed6212bd54ff5acc4ad110e9899ac","RequestId":"tx000000000000000000054-0058c67752-23a86f-default","HostId":"23a86f-default-default"}
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager [-] Continue after error from storage.containers.objects.size: Account GET failed: http://os-objectstore.materna.com:7480/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac?format=json 404 Not Found [first 60 chars of response] {"Code":"NoSuchBucket","BucketName":"AUTH_53eed6212bd54ff5ac
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager Traceback (most recent call last):
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 192, in poll_and_notify
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager for sample in samples:
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 187, in get_samples
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager cache, tenants):
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 80, in _iter_accounts
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager ksclient, tenants))
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 93, in _get_account_info
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager keystone_client.get_auth_token(ksclient)))
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 766, in get_account
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager raise ClientException.from_response(resp, 'Account GET failed', body)
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager ClientException: Account GET failed: http://os-objectstore.materna.com:7480/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac?format=json 404 Not Found [first 60 chars of response] {"Code":"NoSuchBucket","BucketName":"AUTH_53eed6212bd54ff5ac
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager
2017-03-13 11:41:22.855 14501 WARNING ceilometer.agent.discovery.endpoint [-] No endpoints found for service energy
2017-03-13 11:41:23.037 14501 INFO swiftclient [-] REQ: curl i http://os-objectstore.materna.com:7480/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac -I -H "X-Auth-Token: 165af4078f66482b..." ] RESP STATUS: 404 Not Found
2017-03-13 11:41:23.037 14501 INFO swiftclient [
2017-03-13 11:41:23.038 14501 INFO swiftclient [-] RESP HEADERS: {u'Date': u'Mon, 13 Mar 2017 10:41:23 GMT', u'Content-Length': u'12', u'Content-Type': u'text/plain; charset=utf-8', u'Accept-Ranges': u'bytes', u'X-Trans-Id': u'tx000000000000000000055-0058c67753-23a86f-default'}
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager [-] Continue after error from storage.objects.size: Account HEAD failed: http://os-objectstore.materna.com:7480/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac 404 Not Found
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager Traceback (most recent call last):
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 192, in poll_and_notify
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager for sample in samples:
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 125, in get_samples
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager cache, tenants):
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 80, in _iter_accounts
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager ksclient, tenants))
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 93, in _get_account_info
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager keystone_client.get_auth_token(ksclient)))
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 803, in head_account
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager raise ClientException.from_response(resp, 'Account HEAD failed', body)
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager ClientException: Account HEAD failed: http://os-objectstore.materna.com:7480/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac 404 Not Found
$ ceph --version
ceph version 10.2.6 (656b5b63ed7c43bd014bcafd81b001959d5f089f)
Is there any fix or workaround available to solve this issue?
Updated by Jens Offenbach about 7 years ago
I am running OpenStack Ocata on Ubuntu 16.04. using Ceph Jewel (10.2.6) as common storage backend. The RGW Swift API is added to OpenStack's endpoint catalog. The object store is fully functional, but the Ceilometer log contains the following entry which occurrs every 10 minutes:
11:41:22.851 14501 INFO swiftclient [-] REQ: curl i http://os-objectstore.mycompany.com:7480/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac?format=json X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: 165af4078f66482b..."
2017-03-13 11:41:22.851 14501 INFO swiftclient [] RESP STATUS: 404 Not Found
2017-03-13 11:41:22.851 14501 INFO swiftclient [] RESP HEADERS: {u'Date': u'Mon, 13 Mar 2017 10:41:22 GMT', u'Content-Length': u'174', u'Content-Type': u'application/json; charset=utf-8', u'Accept-Ranges': u'bytes', u'X-Trans-Id': u'tx000000000000000000054-0058c67752-23a86f-default'}
2017-03-13 11:41:22.851 14501 INFO swiftclient [-] RESP BODY: {"Code":"NoSuchBucket","BucketName":"AUTH_53eed6212bd54ff5acc4ad110e9899ac","RequestId":"tx000000000000000000054-0058c67752-23a86f-default","HostId":"23a86f-default-default"}
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager [-] Continue after error from storage.containers.objects.size: Account GET failed: http://os-objectstore.mycompany.com:7480/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac?format=json 404 Not Found [first 60 chars of response] {"Code":"NoSuchBucket","BucketName":"AUTH_53eed6212bd54ff5ac
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager Traceback (most recent call last):
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 192, in poll_and_notify
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager for sample in samples:
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 187, in get_samples
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager cache, tenants):
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 80, in _iter_accounts
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager ksclient, tenants))
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 93, in _get_account_info
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager keystone_client.get_auth_token(ksclient)))
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 766, in get_account
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager raise ClientException.from_response(resp, 'Account GET failed', body)
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager ClientException: Account GET failed: http://os-objectstore.mycompany.com:7480/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac?format=json 404 Not Found [first 60 chars of response] {"Code":"NoSuchBucket","BucketName":"AUTH_53eed6212bd54ff5ac
2017-03-13 11:41:22.853 14501 ERROR ceilometer.agent.manager
2017-03-13 11:41:22.855 14501 WARNING ceilometer.agent.discovery.endpoint [-] No endpoints found for service energy
2017-03-13 11:41:23.037 14501 INFO swiftclient [-] REQ: curl i http://os-objectstore.mycompany.com:7480/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac I -H "X-Auth-Token: 165af4078f66482b..."
2017-03-13 11:41:23.037 14501 INFO swiftclient [] RESP STATUS: 404 Not Found
2017-03-13 11:41:23.038 14501 INFO swiftclient [] RESP HEADERS: {u'Date': u'Mon, 13 Mar 2017 10:41:23 GMT', u'Content-Length': u'12', u'Content-Type': u'text/plain; charset=utf-8', u'Accept-Ranges': u'bytes', u'X-Trans-Id': u'tx000000000000000000055-0058c67753-23a86f-default'}
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager [-] Continue after error from storage.objects.size: Account HEAD failed: http://os-objectstore.mycompany.com:7480/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac 404 Not Found
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager Traceback (most recent call last):
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 192, in poll_and_notify
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager for sample in samples:
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 125, in get_samples
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager cache, tenants):
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 80, in _iter_accounts
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager ksclient, tenants))
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 93, in _get_account_info
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager keystone_client.get_auth_token(ksclient)))
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 803, in head_account
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager raise ClientException.from_response(resp, 'Account HEAD failed', body)
2017-03-13 11:41:23.038 14501 ERROR ceilometer.agent.manager ClientException: Account HEAD failed: http://os-objectstore.mycompany.com:7480/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac 404 Not Found$ ceph --Version
ceph version 10.2.6 (656b5b63ed7c43bd014bcafd81b001959d5f089f)Is there any fix or workaround available to solve this issue?
Updated by Yehuda Sadeh about 7 years ago
Can you provide radosgw logs? (debug rgw = 20, debug ms = 1).
Maybe you have some special swift path configuration? Or hitting the dns virtual bucket name?
Updated by Jens Offenbach about 7 years ago
I will add the debug settings to my ceph configuration and provide you the output as soon as possible.
Thanks a lot!
Updated by Jens Offenbach about 7 years ago
Sorry for the delay. Now, I can provide you the radosgw log excerpts.
ceph-client.rgw.os-storage01.log
2017-03-30 15:48:50.473512 7ff4913c9700 20 RGWEnv::set(): HTTP_HOST: os-objectstore.mycompany.com:7480
2017-03-30 15:48:50.473533 7ff4913c9700 20 RGWEnv::set(): HTTP_ACCEPT_ENCODING: identity
2017-03-30 15:48:50.473535 7ff4913c9700 20 RGWEnv::set(): HTTP_USER_AGENT: python-swiftclient-3.3.0
2017-03-30 15:48:50.473538 7ff4913c9700 20 RGWEnv::set(): HTTP_X_AUTH_TOKEN: 6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.473540 7ff4913c9700 20 RGWEnv::set(): HTTP_X_FORWARDED_PORT: 7480
2017-03-30 15:48:50.473542 7ff4913c9700 20 RGWEnv::set(): REQUEST_METHOD: HEAD
2017-03-30 15:48:50.473544 7ff4913c9700 20 RGWEnv::set(): REQUEST_URI: /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.473547 7ff4913c9700 20 RGWEnv::set(): QUERY_STRING:
2017-03-30 15:48:50.473555 7ff4913c9700 20 RGWEnv::set(): REMOTE_USER:
2017-03-30 15:48:50.473557 7ff4913c9700 20 RGWEnv::set(): SCRIPT_URI: /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.473559 7ff4913c9700 20 RGWEnv::set(): SERVER_PORT: 7480
2017-03-30 15:48:50.473560 7ff4913c9700 20 HTTP_ACCEPT_ENCODING=identity
2017-03-30 15:48:50.473561 7ff4913c9700 20 HTTP_HOST=os-objectstore.mycompany.com:7480
2017-03-30 15:48:50.473562 7ff4913c9700 20 HTTP_USER_AGENT=python-swiftclient-3.3.0
2017-03-30 15:48:50.473563 7ff4913c9700 20 HTTP_X_AUTH_TOKEN=6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.473563 7ff4913c9700 20 HTTP_X_FORWARDED_PORT=7480
2017-03-30 15:48:50.473564 7ff4913c9700 20 QUERY_STRING=
2017-03-30 15:48:50.473565 7ff4913c9700 20 REMOTE_USER=
2017-03-30 15:48:50.473566 7ff4913c9700 20 REQUEST_METHOD=HEAD
2017-03-30 15:48:50.473566 7ff4913c9700 20 REQUEST_URI=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.473567 7ff4913c9700 20 SCRIPT_URI=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.473567 7ff4913c9700 20 SERVER_PORT=7480
2017-03-30 15:48:50.473570 7ff4913c9700 1 ====== starting new request req=0x7ff4913c37e0 =====
2017-03-30 15:48:50.473603 7ff4913c9700 2 req 7:0.000033::HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac::initializing for trans_id = tx000000000000000000007-0058dd0cc2-297664-default
2017-03-30 15:48:50.473612 7ff4913c9700 10 rgw api priority: s3=5 s3website=4
2017-03-30 15:48:50.473614 7ff4913c9700 10 host=os-objectstore.mycompany.com
2017-03-30 15:48:50.473617 7ff4913c9700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2017-03-30 15:48:50.473618 7ff4913c9700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.473646 7ff4913c9700 10 ver=v1 first=AUTH_53eed6212bd54ff5acc4ad110e9899ac req=
2017-03-30 15:48:50.473650 7ff4913c9700 10 handler=28RGWHandler_REST_Bucket_SWIFT
2017-03-30 15:48:50.473652 7ff4913c9700 2 req 7:0.000083:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac::getting op 3
2017-03-30 15:48:50.473673 7ff4913c9700 10 op=28RGWStatBucket_ObjStore_SWIFT
2017-03-30 15:48:50.473675 7ff4913c9700 2 req 7:0.000106:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:authorizing
2017-03-30 15:48:50.473682 7ff4913c9700 20 token_id=6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.473697 7ff4913c9700 20 found cached admin token
2017-03-30 15:48:50.473721 7ff4913c9700 20 sending request to http://os-identity:35357/v3/auth/tokens
2017-03-30 15:48:50.809584 7ff4913c9700 20 RGWPostHTTPData::receive_header parsing HTTP headers
2017-03-30 15:48:50.809613 7ff4913c9700 20 RGWPostHTTPData::receive_header: line=HTTP/1.1 200 OK
2017-03-30 15:48:50.809618 7ff4913c9700 20 RGWPostHTTPData::receive_header parsing HTTP headers
2017-03-30 15:48:50.809619 7ff4913c9700 20 RGWPostHTTPData::receive_header: line=Date: Thu, 30 Mar 2017 13:48:50 GMT
2017-03-30 15:48:50.809622 7ff4913c9700 20 RGWPostHTTPData::receive_header parsing HTTP headers
2017-03-30 15:48:50.809623 7ff4913c9700 20 RGWPostHTTPData::receive_header: line=Server: Apache/2.4.18 (Ubuntu)
2017-03-30 15:48:50.809624 7ff4913c9700 20 RGWPostHTTPData::receive_header parsing HTTP headers
2017-03-30 15:48:50.809625 7ff4913c9700 20 RGWPostHTTPData::receive_header: line=X-Subject-Token: 6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.809650 7ff4913c9700 20 RGWPostHTTPData::receive_header parsing HTTP headers
2017-03-30 15:48:50.809652 7ff4913c9700 20 RGWPostHTTPData::receive_header: line=Vary: X-Auth-Token
2017-03-30 15:48:50.809653 7ff4913c9700 20 RGWPostHTTPData::receive_header parsing HTTP headers
2017-03-30 15:48:50.809654 7ff4913c9700 20 RGWPostHTTPData::receive_header: line=X-Distribution: Ubuntu
2017-03-30 15:48:50.809655 7ff4913c9700 20 RGWPostHTTPData::receive_header parsing HTTP headers
2017-03-30 15:48:50.809655 7ff4913c9700 20 RGWPostHTTPData::receive_header: line=x-openstack-request-id: req-d7ff874f-979b-488f-af80-1a7fc59b39ef
2017-03-30 15:48:50.809657 7ff4913c9700 20 RGWPostHTTPData::receive_header parsing HTTP headers
2017-03-30 15:48:50.809657 7ff4913c9700 20 RGWPostHTTPData::receive_header: line=Content-Length: 11110
2017-03-30 15:48:50.809659 7ff4913c9700 20 RGWPostHTTPData::receive_header parsing HTTP headers
2017-03-30 15:48:50.809659 7ff4913c9700 20 RGWPostHTTPData::receive_header: line=Content-Type: application/json
2017-03-30 15:48:50.809660 7ff4913c9700 20 RGWPostHTTPData::receive_header parsing HTTP headers
2017-03-30 15:48:50.809660 7ff4913c9700 20 RGWPostHTTPData::receive_header: line=
2017-03-30 15:48:50.809762 7ff4913c9700 20 received response: {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "f54a928fc2e34275b92e962a5d99ea16", "name": "admin"}], "expires_at": "2017-03-30T14:48:48.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "f0f25423e84b4ad285c4a54ab15f7c1e", "name": "service"}, "catalog": [{"endpoints": [{"url": "https://os-cloud.mycompany.com:8774/v2.1/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "a704ddcb9a6841ada5f69012d2599538"}, {"url": "http://os-compute.mycompany.com:8774/v2.1/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "ab29ee06d19b41f8ad96718936dc7055"}, {"url": "http://os-compute.mycompany.com:8774/v2.1/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "f1ae48201784413daf661cf05a6862ca"}], "type": "compute", "id": "052d049154864ead86dba53d433c1811", "name": "nova"}, {"endpoints": [{"url": "http://os-objectstore.mycompany.com:7480/swift/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "0428c21ef85a4007887591e9f0b1d50c"}, {"url": "https://os-cloud.mycompany.com:7480/swift/v1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "10e153dc319d4810a591f50bac85a54c"}, {"url": "http://os-objectstore.mycompany.com:7480/swift/v1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "df057a0caf0f41e3ae161ee05e2e8666"}], "type": "object-store", "id": "07851e6bad204eff90aaae2a2b5be138", "name": "swift"}, {"endpoints": [{"url": "https://os-cloud.mycompany.com:5000/v3/", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "636a5eb1457b47f8aa67320cab8b5f42"}, {"url": "http://os-identity.mycompany.com:35357/v3/", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "8af09d32b63d47da8bacfbadd2449048"}, {"url": "http://os-identity.mycompany.com:5000/v3/", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "b2c59e69970c4013a7567205f819ec50"}], "type": "identity", "id": "1c658e572ad746d5bc9e5841a92fe2f4", "name": "keystone"}, {"endpoints": [{"url": "http://os-container.mycompany.com:9511/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "786d4f9e1dc54831b318ac3afb696ec5"}, {"url": "https://os-cloud.mycompany.com:9511/v1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "b245489e5624412d8e3ec141db28f3e6"}, {"url": "http://os-container.mycompany.com:9511/v1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "f74e4e716dcc485bbdb2dbdb7ea0777f"}], "type": "container-infra", "id": "29c5c0073d294c1babea01bdca14cffa", "name": "magnum"}, {"endpoints": [{"url": "https://os-cloud.mycompany.com:8042", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "107dd9bc06064c8d9d196bf6d0129a96"}, {"url": "http://os-alarming.mycompany.com:8042", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "1bed7da71e344c2287e3aa416366aafb"}, {"url": "http://os-alarming.mycompany.com:8042", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "765f04fc3aa34a9ca4f7ba04b6afa517"}], "type": "alarming", "id": "382871ee87c94d0b806a6f2bf7b0b8ce", "name": "aodh"}, {"endpoints": [{"url": "https://os-cloud.mycompany.com:9292", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2a25cfe4b39f4a4f88bb7915f50af43e"}, {"url": "http://os-image.mycompany.com:9292", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "5564a201640f47d68b65c23f00f7bbc7"}, {"url": "http://os-image.mycompany.com:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "fd18add466ce4d68867136a864862a70"}], "type": "image", "id": "44e4f6c64c7245fb825b1af541c21780", "name": "glance"}, {"endpoints": [{"url": "http://os-share.mycompany.com:8786/v1/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "334ea173f73f45778bba0cc64997b3e9"}, {"url": "https://os-cloud.mycompany.com:8786/v1/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "c9e5a7919f19473b9a7e3d7983b86efa"}, {"url": "http://os-share.mycompany.com:8786/v1/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "ea67e866ae7f4a6db8524da65adc7b6b"}], "type": "share", "id": "4d8f0e5c900f4cee9a8751dd7b0b986e", "name": "manila"}, {"endpoints": [{"url": "http://os-orchestration.mycompany.com:8004/v1/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "87c7412a464646968f5d419e7bbce751"}, {"url": "https://os-cloud.mycompany.com:8004/v1/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "8d0ebdcc7db3427087133eecaea6ae57"}, {"url": "http://os-orchestration.mycompany.com:8004/v1/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "a434831c8f2544d789e35ba453b8d306"}], "type": "orchestration", "id": "55c4442c44e34c8dbb7901edd187b3c0", "name": "heat"}, {"endpoints": [{"url": "https://os-cloud.mycompany.com:8778", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "6749a754bc2d477db23308ed5c4f851a"}, {"url": "http://os-placement.mycompany.com:8778", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "7b1af0ca9f4b407995a96c3bb1844c51"}, {"url": "http://os-placement.mycompany.com:8778", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "da0d353c21f945189128009f2e0865d4"}], "type": "placement", "id": "6b22c97db3c14bebae8e1985cc785c85", "name": "placement"}, {"endpoints": [{"url": "http://os-volume.mycompany.com:8776/v1/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "2a1be060261d41068364ebcc1d664a2a"}, {"url": "https://os-cloud.mycompany.com:8776/v1/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "6c08368fdaa047c0b2bc5cb8f242e491"}, {"url": "http://os-volume.mycompany.com:8776/v1/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "82400566a7d44a2480a35cd7b84f6361"}], "type": "volume", "id": "7f67a2b971b64c59ac55f2314ad70b6c", "name": "cinder"}, {"endpoints": [{"url": "http://os-cloudformation.mycompany.com:8000/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "7a798346eaf64b5983c830c5169e7444"}, {"url": "http://os-cloudformation.mycompany.com:8000/v1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "8347c241cf0f460b9ba47f0e9b2efdaa"}, {"url": "https://os-cloud.mycompany.com:8000/v1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "9ee60f706db04587bee9c4c6af0be5bd"}, {"url": "http://os-cloudformation.mycompany.com:8000/v1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "def1293c5461484b8c260c509ac54350"}, {"url": "http://os-cloudformation.mycompany.com:8000/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "e6eeba8e165c4885afd2618d30c691df"}, {"url": "https://os-cloud.mycompany.com:8000/v1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "e7ffd78a240e4335ad39ab85f7544fe2"}], "type": "cloudformation", "id": "9cbb3be9dbc243e8a132ef344a85446b", "name": "heat-cfn"}, {"endpoints": [{"url": "http://os-network.mycompany.com:9696", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "5e911f37a4df44e9b84ec14ebfe67725"}, {"url": "https://os-cloud.mycompany.com:9696", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "89b1d21cb92a4834bad2b40cccbdabff"}, {"url": "http://os-network.mycompany.com:9696", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "ff428d7135a84334be09803656b6883a"}], "type": "network", "id": "a4460b00be5f46048b07fc54b10d57d1", "name": "neutron"}, {"endpoints": [{"url": "http://os-telemetry.mycompany.com:8777", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "4a9de44c160a466583d93369f70f2b0a"}, {"url": "https://os-cloud.mycompany.com:8777", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "9b71d2bd1e6f4ecdaa0505f976925321"}, {"url": "http://os-telemetry.mycompany.com:8777", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "e270124a8c0640e790f1d1cfd375abef"}], "type": "metering", "id": "ad0b859bcddb400e875137e5e5162901", "name": "ceilometer"}, {"endpoints": [{"url": "http://os-share.mycompany.com:8786/v2/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "1b8a9685faeb4b8ba631ad3355ef7886"}, {"url": "https://os-cloud.mycompany.com:8786/v2/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "90c3ba5bd7d94dd886d083b603eeaa28"}, {"url": "http://os-share.mycompany.com:8786/v2/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "e7aa92f6ff954805ab01a4bbf66630c1"}], "type": "sharev2", "id": "e21b505c34a84f83adcb5ec3fa723c9d", "name": "manilav2"}, {"endpoints": [{"url": "http://os-volume.mycompany.com:8776/v2/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "205e8c108c07474d8259bd9a8ad050e5"}, {"url": "https://os-cloud.mycompany.com:8776/v2/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "5fc7fbc5ea7a4620913b0b943fcab1cd"}, {"url": "http://os-volume.mycompany.com:8776/v2/f0f25423e84b4ad285c4a54ab15f7c1e", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "ad4068e1ff2c4529b251555136185a7c"}], "type": "volumev2", "id": "ef259c40836e471e9db06cae21b25ba7", "name": "cinderv2"}, {"endpoints": [{"url": "http://os-securestore.mycompany.com:9311", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "6fb0ca53701840c08efb573feba3efd9"}, {"url": "http://os-securestore.mycompany.com:9311", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "c5bae7e8e9f94860a08d0bf6d6cb3b27"}, {"url": "https://os-cloud.mycompany.com:9311", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "d997a78da22c41e0834a7c74875849fe"}], "type": "key-manager", "id": "fac1886b3ded4fcd949c38aea03dd1f6", "name": "barbican"}], "user": {"password_expires_at": null, "domain": {"id": "default", "name": "Default"}, "id": "1acb940597e14beda7285aa06d6c4c62", "name": "ceilometer"}, "audit_ids": ["ciHYu7CcR-e1RcO4sfPylw"], "issued_at": "2017-03-30T13:48:48.000000Z"}}
2017-03-30 15:48:50.813977 7ff4913c9700 0 validated token: service:ceilometer expires: 1490885328
2017-03-30 15:48:50.814001 7ff4913c9700 20 updating user=f0f25423e84b4ad285c4a54ab15f7c1e
2017-03-30 15:48:50.814028 7ff4913c9700 20 get_system_obj_state: rctx=0x7ff4913c1170 obj=default.rgw.users.uid:f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e state=0x7ff4d008e678 s->prefetch_data=0
2017-03-30 15:48:50.814048 7ff4913c9700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e : type miss (requested=6, cached=0)
2017-03-30 15:48:50.814106 7ff4913c9700 1 -- 10.30.200.141:0/3968062828 --> 10.30.200.143:6800/2371 -- osd_op(client.2717284.0:2132 6.12fd808b f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e943) v7 -- ?+0 0x7ff4d008c520 con 0x7ff4d402e150
2017-03-30 15:48:50.815010 7ff4dd6f9700 1 -- 10.30.200.141:0/3968062828 <== osd.11 10.30.200.143:6800/2371 265 ==== osd_op_reply(2132 f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 227+0+0 (2177119573 0 0) 0x7ff4e0012330 con 0x7ff4d402e150
2017-03-30 15:48:50.815150 7ff4913c9700 10 cache put: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e info.flags=0
2017-03-30 15:48:50.815169 7ff4913c9700 10 moving default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e to cache LRU end
2017-03-30 15:48:50.815196 7ff4913c9700 20 get_system_obj_state: rctx=0x7ff4913c1170 obj=default.rgw.users.uid:f0f25423e84b4ad285c4a54ab15f7c1e state=0x7ff4d008e678 s->prefetch_data=0
2017-03-30 15:48:50.815214 7ff4913c9700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e : hit (requested=6, cached=7)
2017-03-30 15:48:50.815226 7ff4913c9700 20 get_system_obj_state: s->obj_tag was set empty
2017-03-30 15:48:50.815245 7ff4913c9700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e : hit (requested=1, cached=7)
2017-03-30 15:48:50.815271 7ff4913c9700 2 req 7:0.341701:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:normalizing buckets and tenants
2017-03-30 15:48:50.815289 7ff4913c9700 10 s->object=<NULL> s->bucket=AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.815298 7ff4913c9700 2 req 7:0.341729:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:init permissions
2017-03-30 15:48:50.815322 7ff4913c9700 20 get_system_obj_state: rctx=0x7ff4913c26c0 obj=default.rgw.data.root:AUTH_53eed6212bd54ff5acc4ad110e9899ac state=0x7ff4d008e678 s->prefetch_data=0
2017-03-30 15:48:50.815345 7ff4913c9700 10 cache get: name=default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac : type miss (requested=22, cached=0)
2017-03-30 15:48:50.815391 7ff4913c9700 1 -- 10.30.200.141:0/3968062828 --> 10.30.200.142:6808/2705 -- osd_op(client.2717284.0:2133 3.93c479fe AUTH_53eed6212bd54ff5acc4ad110e9899ac [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e943) v7 -- ?+0 0x7ff4d008dfc0 con 0x7ff4d4022240
2017-03-30 15:48:50.816209 7ff4ec290700 1 -- 10.30.200.141:0/3968062828 <== osd.4 10.30.200.142:6808/2705 72 ==== osd_op_reply(2133 AUTH_53eed6212bd54ff5acc4ad110e9899ac [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 241+0+0 (1282282941 0 0) 0x7ff4e0012330 con 0x7ff4d4022240
2017-03-30 15:48:50.816318 7ff4913c9700 10 cache put: name=default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac info.flags=0
2017-03-30 15:48:50.816364 7ff4913c9700 10 moving default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac to cache LRU end
2017-03-30 15:48:50.816416 7ff4913c9700 10 read_permissions on AUTH_53eed6212bd54ff5acc4ad110e9899ac(@[]) ret=-2002
2017-03-30 15:48:50.816461 7ff4913c9700 20 op->ERRORHANDLER: err_no=-2002 new_err_no=-2002
2017-03-30 15:48:50.816588 7ff4913c9700 2 req 7:0.343019:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:op status=0
2017-03-30 15:48:50.816637 7ff4913c9700 2 req 7:0.343066:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:http status=404
2017-03-30 15:48:50.816679 7ff4913c9700 1 ====== req done req=0x7ff4913c37e0 op status=0 http_status=404 ======
2017-03-30 15:48:50.816740 7ff4913c9700 20 process_request() returned -2002
2017-03-30 15:48:50.816833 7ff4913c9700 1 civetweb: 0x7ff4d0004c40: 10.30.200.101 - - [30/Mar/2017:15:48:50 +0200] "HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac HTTP/1.1" 404 0 - python-swiftclient-3.3.0
2017-03-30 15:48:50.824907 7ff48d9c0700 20 RGWEnv::set(): HTTP_HOST: os-objectstore.mycompany.com:7480
2017-03-30 15:48:50.824920 7ff48d9c0700 20 RGWEnv::set(): HTTP_ACCEPT_ENCODING: identity
2017-03-30 15:48:50.824922 7ff48d9c0700 20 RGWEnv::set(): HTTP_USER_AGENT: python-swiftclient-3.3.0
2017-03-30 15:48:50.824925 7ff48d9c0700 20 RGWEnv::set(): HTTP_X_AUTH_TOKEN: 6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.824927 7ff48d9c0700 20 RGWEnv::set(): HTTP_X_FORWARDED_PORT: 7480
2017-03-30 15:48:50.824928 7ff48d9c0700 20 RGWEnv::set(): REQUEST_METHOD: HEAD
2017-03-30 15:48:50.824929 7ff48d9c0700 20 RGWEnv::set(): REQUEST_URI: /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.824932 7ff48d9c0700 20 RGWEnv::set(): QUERY_STRING:
2017-03-30 15:48:50.824933 7ff48d9c0700 20 RGWEnv::set(): REMOTE_USER:
2017-03-30 15:48:50.824935 7ff48d9c0700 20 RGWEnv::set(): SCRIPT_URI: /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.824937 7ff48d9c0700 20 RGWEnv::set(): SERVER_PORT: 7480
2017-03-30 15:48:50.824938 7ff48d9c0700 20 HTTP_ACCEPT_ENCODING=identity
2017-03-30 15:48:50.824939 7ff48d9c0700 20 HTTP_HOST=os-objectstore.mycompany.com:7480
2017-03-30 15:48:50.824939 7ff48d9c0700 20 HTTP_USER_AGENT=python-swiftclient-3.3.0
2017-03-30 15:48:50.824940 7ff48d9c0700 20 HTTP_X_AUTH_TOKEN=6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.824940 7ff48d9c0700 20 HTTP_X_FORWARDED_PORT=7480
2017-03-30 15:48:50.824941 7ff48d9c0700 20 QUERY_STRING=
2017-03-30 15:48:50.824941 7ff48d9c0700 20 REMOTE_USER=
2017-03-30 15:48:50.824942 7ff48d9c0700 20 REQUEST_METHOD=HEAD
2017-03-30 15:48:50.824942 7ff48d9c0700 20 REQUEST_URI=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.824943 7ff48d9c0700 20 SCRIPT_URI=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.824943 7ff48d9c0700 20 SERVER_PORT=7480
2017-03-30 15:48:50.824944 7ff48d9c0700 1 ====== starting new request req=0x7ff48d9ba7e0 =====
2017-03-30 15:48:50.824958 7ff48d9c0700 2 req 8:0.000014::HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac::initializing for trans_id = tx000000000000000000008-0058dd0cc2-297664-default
2017-03-30 15:48:50.824967 7ff48d9c0700 10 rgw api priority: s3=5 s3website=4
2017-03-30 15:48:50.824968 7ff48d9c0700 10 host=os-objectstore.mycompany.com
2017-03-30 15:48:50.824970 7ff48d9c0700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2017-03-30 15:48:50.824972 7ff48d9c0700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.824997 7ff48d9c0700 10 ver=v1 first=AUTH_53eed6212bd54ff5acc4ad110e9899ac req=
2017-03-30 15:48:50.825001 7ff48d9c0700 10 handler=28RGWHandler_REST_Bucket_SWIFT
2017-03-30 15:48:50.825003 7ff48d9c0700 2 req 8:0.000058:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac::getting op 3
2017-03-30 15:48:50.825006 7ff48d9c0700 10 op=28RGWStatBucket_ObjStore_SWIFT
2017-03-30 15:48:50.825007 7ff48d9c0700 2 req 8:0.000062:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:authorizing
2017-03-30 15:48:50.825011 7ff48d9c0700 20 token_id=6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.825017 7ff48d9c0700 20 cached token.project.id=f0f25423e84b4ad285c4a54ab15f7c1e
2017-03-30 15:48:50.825019 7ff48d9c0700 20 updating user=f0f25423e84b4ad285c4a54ab15f7c1e
2017-03-30 15:48:50.825037 7ff48d9c0700 20 get_system_obj_state: rctx=0x7ff48d9b8170 obj=default.rgw.users.uid:f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e state=0x7ff4e004a5e8 s->prefetch_data=0
2017-03-30 15:48:50.825047 7ff48d9c0700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e : type miss (requested=6, cached=0)
2017-03-30 15:48:50.825087 7ff48d9c0700 1 -- 10.30.200.141:0/3968062828 --> 10.30.200.143:6800/2371 -- osd_op(client.2717284.0:2134 6.12fd808b f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e943) v7 -- ?+0 0x7ff4e004d020 con 0x7ff4d402e150
2017-03-30 15:48:50.825805 7ff4dd6f9700 1 -- 10.30.200.141:0/3968062828 <== osd.11 10.30.200.143:6800/2371 266 ==== osd_op_reply(2134 f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 227+0+0 (2177119573 0 0) 0x7ff4e004d1b0 con 0x7ff4d402e150
2017-03-30 15:48:50.825885 7ff48d9c0700 10 cache put: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e info.flags=0
2017-03-30 15:48:50.825896 7ff48d9c0700 10 moving default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e to cache LRU end
2017-03-30 15:48:50.825911 7ff48d9c0700 20 get_system_obj_state: rctx=0x7ff48d9b8170 obj=default.rgw.users.uid:f0f25423e84b4ad285c4a54ab15f7c1e state=0x7ff4e004a5e8 s->prefetch_data=0
2017-03-30 15:48:50.825918 7ff48d9c0700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e : hit (requested=6, cached=7)
2017-03-30 15:48:50.825924 7ff48d9c0700 20 get_system_obj_state: s->obj_tag was set empty
2017-03-30 15:48:50.825927 7ff48d9c0700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e : hit (requested=1, cached=7)
2017-03-30 15:48:50.825940 7ff48d9c0700 2 req 8:0.000995:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:normalizing buckets and tenants
2017-03-30 15:48:50.825943 7ff48d9c0700 10 s->object=<NULL> s->bucket=AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.825947 7ff48d9c0700 2 req 8:0.001002:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:init permissions
2017-03-30 15:48:50.825958 7ff48d9c0700 20 get_system_obj_state: rctx=0x7ff48d9b96c0 obj=default.rgw.data.root:AUTH_53eed6212bd54ff5acc4ad110e9899ac state=0x7ff4e004a4f8 s->prefetch_data=0
2017-03-30 15:48:50.825966 7ff48d9c0700 10 cache get: name=default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac : type miss (requested=22, cached=0)
2017-03-30 15:48:50.825996 7ff48d9c0700 1 -- 10.30.200.141:0/3968062828 --> 10.30.200.142:6808/2705 -- osd_op(client.2717284.0:2135 3.93c479fe AUTH_53eed6212bd54ff5acc4ad110e9899ac [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e943) v7 -- ?+0 0x7ff4e004f7a0 con 0x7ff4d4022240
2017-03-30 15:48:50.826900 7ff4ec290700 1 -- 10.30.200.141:0/3968062828 <== osd.4 10.30.200.142:6808/2705 73 ==== osd_op_reply(2135 AUTH_53eed6212bd54ff5acc4ad110e9899ac [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 241+0+0 (1282282941 0 0) 0x7ff4e004f900 con 0x7ff4d4022240
2017-03-30 15:48:50.826980 7ff48d9c0700 10 cache put: name=default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac info.flags=0
2017-03-30 15:48:50.826989 7ff48d9c0700 10 moving default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac to cache LRU end
2017-03-30 15:48:50.827002 7ff48d9c0700 10 read_permissions on AUTH_53eed6212bd54ff5acc4ad110e9899ac(@[]) ret=-2002
2017-03-30 15:48:50.827006 7ff48d9c0700 20 op->ERRORHANDLER: err_no=-2002 new_err_no=-2002
2017-03-30 15:48:50.827064 7ff48d9c0700 2 req 8:0.002119:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:op status=0
2017-03-30 15:48:50.827067 7ff48d9c0700 2 req 8:0.002123:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:http status=404
2017-03-30 15:48:50.827069 7ff48d9c0700 1 ====== req done req=0x7ff48d9ba7e0 op status=0 http_status=404 ======
2017-03-30 15:48:50.827077 7ff48d9c0700 20 process_request() returned -2002
2017-03-30 15:48:50.827107 7ff48d9c0700 1 civetweb: 0x7ff4e0014a20: 10.30.200.101 - - [30/Mar/2017:15:48:50 +0200] "HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac HTTP/1.1" 404 0 - python-swiftclient-3.3.0
2017-03-30 15:48:50.835237 7ff48eac3700 20 RGWEnv::set(): HTTP_HOST: os-objectstore.mycompany.com:7480
2017-03-30 15:48:50.835257 7ff48eac3700 20 RGWEnv::set(): HTTP_ACCEPT_ENCODING: identity
2017-03-30 15:48:50.835261 7ff48eac3700 20 RGWEnv::set(): HTTP_USER_AGENT: python-swiftclient-3.3.0
2017-03-30 15:48:50.835265 7ff48eac3700 20 RGWEnv::set(): HTTP_X_AUTH_TOKEN: 6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.835268 7ff48eac3700 20 RGWEnv::set(): HTTP_X_FORWARDED_PORT: 7480
2017-03-30 15:48:50.835270 7ff48eac3700 20 RGWEnv::set(): REQUEST_METHOD: HEAD
2017-03-30 15:48:50.835273 7ff48eac3700 20 RGWEnv::set(): REQUEST_URI: /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.835276 7ff48eac3700 20 RGWEnv::set(): QUERY_STRING:
2017-03-30 15:48:50.835278 7ff48eac3700 20 RGWEnv::set(): REMOTE_USER:
2017-03-30 15:48:50.835281 7ff48eac3700 20 RGWEnv::set(): SCRIPT_URI: /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.835285 7ff48eac3700 20 RGWEnv::set(): SERVER_PORT: 7480
2017-03-30 15:48:50.835287 7ff48eac3700 20 HTTP_ACCEPT_ENCODING=identity
2017-03-30 15:48:50.835289 7ff48eac3700 20 HTTP_HOST=os-objectstore.mycompany.com:7480
2017-03-30 15:48:50.835290 7ff48eac3700 20 HTTP_USER_AGENT=python-swiftclient-3.3.0
2017-03-30 15:48:50.835292 7ff48eac3700 20 HTTP_X_AUTH_TOKEN=6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.835294 7ff48eac3700 20 HTTP_X_FORWARDED_PORT=7480
2017-03-30 15:48:50.835295 7ff48eac3700 20 QUERY_STRING=
2017-03-30 15:48:50.835296 7ff48eac3700 20 REMOTE_USER=
2017-03-30 15:48:50.835297 7ff48eac3700 20 REQUEST_METHOD=HEAD
2017-03-30 15:48:50.835299 7ff48eac3700 20 REQUEST_URI=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.835300 7ff48eac3700 20 SCRIPT_URI=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.835301 7ff48eac3700 20 SERVER_PORT=7480
2017-03-30 15:48:50.835304 7ff48eac3700 1 ====== starting new request req=0x7ff48eabd7e0 =====
2017-03-30 15:48:50.835329 7ff48eac3700 2 req 9:0.000019::HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac::initializing for trans_id = tx000000000000000000009-0058dd0cc2-297664-default
2017-03-30 15:48:50.835340 7ff48eac3700 10 rgw api priority: s3=5 s3website=4
2017-03-30 15:48:50.835341 7ff48eac3700 10 host=os-objectstore.mycompany.com
2017-03-30 15:48:50.835343 7ff48eac3700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2017-03-30 15:48:50.835344 7ff48eac3700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.835369 7ff48eac3700 10 ver=v1 first=AUTH_53eed6212bd54ff5acc4ad110e9899ac req=
2017-03-30 15:48:50.835373 7ff48eac3700 10 handler=28RGWHandler_REST_Bucket_SWIFT
2017-03-30 15:48:50.835375 7ff48eac3700 2 req 9:0.000071:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac::getting op 3
2017-03-30 15:48:50.835378 7ff48eac3700 10 op=28RGWStatBucket_ObjStore_SWIFT
2017-03-30 15:48:50.835379 7ff48eac3700 2 req 9:0.000075:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:authorizing
2017-03-30 15:48:50.835386 7ff48eac3700 20 token_id=6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.835394 7ff48eac3700 20 cached token.project.id=f0f25423e84b4ad285c4a54ab15f7c1e
2017-03-30 15:48:50.835396 7ff48eac3700 20 updating user=f0f25423e84b4ad285c4a54ab15f7c1e
2017-03-30 15:48:50.835415 7ff48eac3700 20 get_system_obj_state: rctx=0x7ff48eabb170 obj=default.rgw.users.uid:f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e state=0x7ff4e803e8f8 s->prefetch_data=0
2017-03-30 15:48:50.835425 7ff48eac3700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e : type miss (requested=6, cached=0)
2017-03-30 15:48:50.835470 7ff48eac3700 1 -- 10.30.200.141:0/3968062828 --> 10.30.200.143:6800/2371 -- osd_op(client.2717284.0:2136 6.12fd808b f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e943) v7 -- ?+0 0x7ff4e8040770 con 0x7ff4d402e150
2017-03-30 15:48:50.836206 7ff4dd6f9700 1 -- 10.30.200.141:0/3968062828 <== osd.11 10.30.200.143:6800/2371 267 ==== osd_op_reply(2136 f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 227+0+0 (2177119573 0 0) 0x7ff4e004f900 con 0x7ff4d402e150
2017-03-30 15:48:50.836287 7ff48eac3700 10 cache put: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e info.flags=0
2017-03-30 15:48:50.836293 7ff48eac3700 10 moving default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e to cache LRU end
2017-03-30 15:48:50.836308 7ff48eac3700 20 get_system_obj_state: rctx=0x7ff48eabb170 obj=default.rgw.users.uid:f0f25423e84b4ad285c4a54ab15f7c1e state=0x7ff4e803e788 s->prefetch_data=0
2017-03-30 15:48:50.836314 7ff48eac3700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e : hit (requested=6, cached=7)
2017-03-30 15:48:50.836320 7ff48eac3700 20 get_system_obj_state: s->obj_tag was set empty
2017-03-30 15:48:50.836323 7ff48eac3700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e : hit (requested=1, cached=7)
2017-03-30 15:48:50.836342 7ff48eac3700 2 req 9:0.001039:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:normalizing buckets and tenants
2017-03-30 15:48:50.836346 7ff48eac3700 10 s->object=<NULL> s->bucket=AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.836349 7ff48eac3700 2 req 9:0.001046:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:init permissions
2017-03-30 15:48:50.836360 7ff48eac3700 20 get_system_obj_state: rctx=0x7ff48eabc6c0 obj=default.rgw.data.root:AUTH_53eed6212bd54ff5acc4ad110e9899ac state=0x7ff4e803e648 s->prefetch_data=0
2017-03-30 15:48:50.836364 7ff48eac3700 10 cache get: name=default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac : type miss (requested=22, cached=0)
2017-03-30 15:48:50.836400 7ff48eac3700 1 -- 10.30.200.141:0/3968062828 --> 10.30.200.142:6808/2705 -- osd_op(client.2717284.0:2137 3.93c479fe AUTH_53eed6212bd54ff5acc4ad110e9899ac [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e943) v7 -- ?+0 0x7ff4e8042a70 con 0x7ff4d4022240
2017-03-30 15:48:50.837037 7ff4ec290700 1 -- 10.30.200.141:0/3968062828 <== osd.4 10.30.200.142:6808/2705 74 ==== osd_op_reply(2137 AUTH_53eed6212bd54ff5acc4ad110e9899ac [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 241+0+0 (1282282941 0 0) 0x7ff4e004f910 con 0x7ff4d4022240
2017-03-30 15:48:50.837117 7ff48eac3700 10 cache put: name=default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac info.flags=0
2017-03-30 15:48:50.837126 7ff48eac3700 10 moving default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac to cache LRU end
2017-03-30 15:48:50.837139 7ff48eac3700 10 read_permissions on AUTH_53eed6212bd54ff5acc4ad110e9899ac(@[]) ret=-2002
2017-03-30 15:48:50.837142 7ff48eac3700 20 op->ERRORHANDLER: err_no=-2002 new_err_no=-2002
2017-03-30 15:48:50.837204 7ff48eac3700 2 req 9:0.001900:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:op status=0
2017-03-30 15:48:50.837207 7ff48eac3700 2 req 9:0.001904:swift:HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:stat_bucket:http status=404
2017-03-30 15:48:50.837209 7ff48eac3700 1 ====== req done req=0x7ff48eabd7e0 op status=0 http_status=404 ======
2017-03-30 15:48:50.837216 7ff48eac3700 20 process_request() returned -2002
2017-03-30 15:48:50.837244 7ff48eac3700 1 civetweb: 0x7ff4e80022f0: 10.30.200.101 - - [30/Mar/2017:15:48:50 +0200] "HEAD /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac HTTP/1.1" 404 0 - python-swiftclient-3.3.0
2017-03-30 15:48:50.844852 7ff4889b6700 20 RGWEnv::set(): HTTP_HOST: os-objectstore.mycompany.com:7480
2017-03-30 15:48:50.844867 7ff4889b6700 20 RGWEnv::set(): HTTP_USER_AGENT: python-swiftclient-3.3.0
2017-03-30 15:48:50.844869 7ff4889b6700 20 RGWEnv::set(): HTTP_ACCEPT_ENCODING: gzip
2017-03-30 15:48:50.844871 7ff4889b6700 20 RGWEnv::set(): HTTP_X_AUTH_TOKEN: 6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.844874 7ff4889b6700 20 RGWEnv::set(): HTTP_X_FORWARDED_PORT: 7480
2017-03-30 15:48:50.844877 7ff4889b6700 20 RGWEnv::set(): REQUEST_METHOD: GET
2017-03-30 15:48:50.844880 7ff4889b6700 20 RGWEnv::set(): REQUEST_URI: /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.844900 7ff4889b6700 20 RGWEnv::set(): QUERY_STRING: format=json
2017-03-30 15:48:50.844905 7ff4889b6700 20 RGWEnv::set(): REMOTE_USER:
2017-03-30 15:48:50.844907 7ff4889b6700 20 RGWEnv::set(): SCRIPT_URI: /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.844911 7ff4889b6700 20 RGWEnv::set(): SERVER_PORT: 7480
2017-03-30 15:48:50.844912 7ff4889b6700 20 HTTP_ACCEPT_ENCODING=gzip
2017-03-30 15:48:50.844914 7ff4889b6700 20 HTTP_HOST=os-objectstore.mycompany.com:7480
2017-03-30 15:48:50.844915 7ff4889b6700 20 HTTP_USER_AGENT=python-swiftclient-3.3.0
2017-03-30 15:48:50.844916 7ff4889b6700 20 HTTP_X_AUTH_TOKEN=6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.844916 7ff4889b6700 20 HTTP_X_FORWARDED_PORT=7480
2017-03-30 15:48:50.844917 7ff4889b6700 20 QUERY_STRING=format=json
2017-03-30 15:48:50.844918 7ff4889b6700 20 REMOTE_USER=
2017-03-30 15:48:50.844919 7ff4889b6700 20 REQUEST_METHOD=GET
2017-03-30 15:48:50.844920 7ff4889b6700 20 REQUEST_URI=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.844921 7ff4889b6700 20 SCRIPT_URI=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.844922 7ff4889b6700 20 SERVER_PORT=7480
2017-03-30 15:48:50.844925 7ff4889b6700 1 ====== starting new request req=0x7ff4889b07e0 =====
2017-03-30 15:48:50.844952 7ff4889b6700 2 req 10:0.000027::GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac::initializing for trans_id = tx00000000000000000000a-0058dd0cc2-297664-default
2017-03-30 15:48:50.844962 7ff4889b6700 10 rgw api priority: s3=5 s3website=4
2017-03-30 15:48:50.844964 7ff4889b6700 10 host=os-objectstore.mycompany.com
2017-03-30 15:48:50.844966 7ff4889b6700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2017-03-30 15:48:50.844968 7ff4889b6700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.844998 7ff4889b6700 10 ver=v1 first=AUTH_53eed6212bd54ff5acc4ad110e9899ac req=
2017-03-30 15:48:50.845004 7ff4889b6700 10 handler=28RGWHandler_REST_Bucket_SWIFT
2017-03-30 15:48:50.845018 7ff4889b6700 2 req 10:0.000082:swift:GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac::getting op 0
2017-03-30 15:48:50.845025 7ff4889b6700 10 op=28RGWListBucket_ObjStore_SWIFT
2017-03-30 15:48:50.845027 7ff4889b6700 2 req 10:0.000102:swift:GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:list_bucket:authorizing
2017-03-30 15:48:50.845032 7ff4889b6700 20 token_id=6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:50.845041 7ff4889b6700 20 cached token.project.id=f0f25423e84b4ad285c4a54ab15f7c1e
2017-03-30 15:48:50.845042 7ff4889b6700 20 updating user=f0f25423e84b4ad285c4a54ab15f7c1e
2017-03-30 15:48:50.845076 7ff4889b6700 20 get_system_obj_state: rctx=0x7ff4889ae170 obj=default.rgw.users.uid:f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e state=0x7ff4a8046ee8 s->prefetch_data=0
2017-03-30 15:48:50.845114 7ff4889b6700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e : type miss (requested=6, cached=0)
2017-03-30 15:48:50.845195 7ff4889b6700 1 -- 10.30.200.141:0/3968062828 --> 10.30.200.143:6800/2371 -- osd_op(client.2717284.0:2138 6.12fd808b f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e943) v7 -- ?+0 0x7ff4a8048e10 con 0x7ff4d402e150
2017-03-30 15:48:50.845797 7ff4dd6f9700 1 -- 10.30.200.141:0/3968062828 <== osd.11 10.30.200.143:6800/2371 268 ==== osd_op_reply(2138 f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 227+0+0 (2177119573 0 0) 0x7ff4e004f900 con 0x7ff4d402e150
2017-03-30 15:48:50.845870 7ff4889b6700 10 cache put: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e info.flags=0
2017-03-30 15:48:50.845877 7ff4889b6700 10 moving default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e to cache LRU end
2017-03-30 15:48:50.845897 7ff4889b6700 20 get_system_obj_state: rctx=0x7ff4889ae170 obj=default.rgw.users.uid:f0f25423e84b4ad285c4a54ab15f7c1e state=0x7ff4a8046d78 s->prefetch_data=0
2017-03-30 15:48:50.845908 7ff4889b6700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e : hit (requested=6, cached=7)
2017-03-30 15:48:50.845916 7ff4889b6700 20 get_system_obj_state: s->obj_tag was set empty
2017-03-30 15:48:50.845921 7ff4889b6700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e : hit (requested=1, cached=7)
2017-03-30 15:48:50.845952 7ff4889b6700 2 req 10:0.001026:swift:GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:list_bucket:normalizing buckets and tenants
2017-03-30 15:48:50.845968 7ff4889b6700 10 s->object=<NULL> s->bucket=AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:50.845972 7ff4889b6700 2 req 10:0.001047:swift:GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:list_bucket:init permissions
2017-03-30 15:48:50.845983 7ff4889b6700 20 get_system_obj_state: rctx=0x7ff4889af6c0 obj=default.rgw.data.root:AUTH_53eed6212bd54ff5acc4ad110e9899ac state=0x7ff4a8046d48 s->prefetch_data=0
2017-03-30 15:48:50.845988 7ff4889b6700 10 cache get: name=default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac : type miss (requested=22, cached=0)
2017-03-30 15:48:50.846025 7ff4889b6700 1 -- 10.30.200.141:0/3968062828 --> 10.30.200.142:6808/2705 -- osd_op(client.2717284.0:2139 3.93c479fe AUTH_53eed6212bd54ff5acc4ad110e9899ac [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e943) v7 -- ?+0 0x7ff4a804b110 con 0x7ff4d4022240
2017-03-30 15:48:50.846534 7ff4ec290700 1 -- 10.30.200.141:0/3968062828 <== osd.4 10.30.200.142:6808/2705 75 ==== osd_op_reply(2139 AUTH_53eed6212bd54ff5acc4ad110e9899ac [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 241+0+0 (1282282941 0 0) 0x7ff4e004f7b0 con 0x7ff4d4022240
2017-03-30 15:48:50.846606 7ff4889b6700 10 cache put: name=default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac info.flags=0
2017-03-30 15:48:50.846616 7ff4889b6700 10 moving default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac to cache LRU end
2017-03-30 15:48:50.846627 7ff4889b6700 10 read_permissions on AUTH_53eed6212bd54ff5acc4ad110e9899ac(@[]) ret=-2002
2017-03-30 15:48:50.846630 7ff4889b6700 20 op->ERRORHANDLER: err_no=-2002 new_err_no=-2002
2017-03-30 15:48:50.846697 7ff4889b6700 2 req 10:0.001772:swift:GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:list_bucket:op status=0
2017-03-30 15:48:50.846706 7ff4889b6700 2 req 10:0.001781:swift:GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:list_bucket:http status=404
2017-03-30 15:48:50.846709 7ff4889b6700 1 ====== req done req=0x7ff4889b07e0 op status=0 http_status=404 ======
2017-03-30 15:48:50.846717 7ff4889b6700 20 process_request() returned -2002
2017-03-30 15:48:50.846758 7ff4889b6700 1 civetweb: 0x7ff4a80041a0: 10.30.200.101 - - [30/Mar/2017:15:48:50 +0200] "GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac HTTP/1.1" 404 0 - python-swiftclient-3.3.0
2017-03-30 15:48:51.066316 7ff48e2c2700 20 RGWEnv::set(): HTTP_HOST: os-objectstore.mycompany.com:7480
2017-03-30 15:48:51.066337 7ff48e2c2700 20 RGWEnv::set(): HTTP_USER_AGENT: python-swiftclient-3.3.0
2017-03-30 15:48:51.066339 7ff48e2c2700 20 RGWEnv::set(): HTTP_ACCEPT_ENCODING: gzip
2017-03-30 15:48:51.066342 7ff48e2c2700 20 RGWEnv::set(): HTTP_X_AUTH_TOKEN: 6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:51.066345 7ff48e2c2700 20 RGWEnv::set(): HTTP_X_FORWARDED_PORT: 7480
2017-03-30 15:48:51.066347 7ff48e2c2700 20 RGWEnv::set(): REQUEST_METHOD: GET
2017-03-30 15:48:51.066349 7ff48e2c2700 20 RGWEnv::set(): REQUEST_URI: /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:51.066351 7ff48e2c2700 20 RGWEnv::set(): QUERY_STRING: format=json
2017-03-30 15:48:51.066353 7ff48e2c2700 20 RGWEnv::set(): REMOTE_USER:
2017-03-30 15:48:51.066355 7ff48e2c2700 20 RGWEnv::set(): SCRIPT_URI: /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:51.066357 7ff48e2c2700 20 RGWEnv::set(): SERVER_PORT: 7480
2017-03-30 15:48:51.066358 7ff48e2c2700 20 HTTP_ACCEPT_ENCODING=gzip
2017-03-30 15:48:51.066359 7ff48e2c2700 20 HTTP_HOST=os-objectstore.mycompany.com:7480
2017-03-30 15:48:51.066359 7ff48e2c2700 20 HTTP_USER_AGENT=python-swiftclient-3.3.0
2017-03-30 15:48:51.066360 7ff48e2c2700 20 HTTP_X_AUTH_TOKEN=6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:51.066361 7ff48e2c2700 20 HTTP_X_FORWARDED_PORT=7480
2017-03-30 15:48:51.066361 7ff48e2c2700 20 QUERY_STRING=format=json
2017-03-30 15:48:51.066362 7ff48e2c2700 20 REMOTE_USER=
2017-03-30 15:48:51.066363 7ff48e2c2700 20 REQUEST_METHOD=GET
2017-03-30 15:48:51.066363 7ff48e2c2700 20 REQUEST_URI=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:51.066364 7ff48e2c2700 20 SCRIPT_URI=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:51.066365 7ff48e2c2700 20 SERVER_PORT=7480
2017-03-30 15:48:51.066370 7ff48e2c2700 1 ====== starting new request req=0x7ff48e2bc7e0 =====
2017-03-30 15:48:51.066392 7ff48e2c2700 2 req 11:0.000025::GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac::initializing for trans_id = tx00000000000000000000b-0058dd0cc3-297664-default
2017-03-30 15:48:51.066403 7ff48e2c2700 10 rgw api priority: s3=5 s3website=4
2017-03-30 15:48:51.066405 7ff48e2c2700 10 host=os-objectstore.mycompany.com
2017-03-30 15:48:51.066407 7ff48e2c2700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2017-03-30 15:48:51.066409 7ff48e2c2700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:51.066439 7ff48e2c2700 10 ver=v1 first=AUTH_53eed6212bd54ff5acc4ad110e9899ac req=
2017-03-30 15:48:51.066444 7ff48e2c2700 10 handler=28RGWHandler_REST_Bucket_SWIFT
2017-03-30 15:48:51.066446 7ff48e2c2700 2 req 11:0.000079:swift:GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac::getting op 0
2017-03-30 15:48:51.066453 7ff48e2c2700 10 op=28RGWListBucket_ObjStore_SWIFT
2017-03-30 15:48:51.066454 7ff48e2c2700 2 req 11:0.000088:swift:GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:list_bucket:authorizing
2017-03-30 15:48:51.066460 7ff48e2c2700 20 token_id=6a7c4f5201f44358abd5d0f76b6f8770
2017-03-30 15:48:51.066469 7ff48e2c2700 20 cached token.project.id=f0f25423e84b4ad285c4a54ab15f7c1e
2017-03-30 15:48:51.066470 7ff48e2c2700 20 updating user=f0f25423e84b4ad285c4a54ab15f7c1e
2017-03-30 15:48:51.066493 7ff48e2c2700 20 get_system_obj_state: rctx=0x7ff48e2ba170 obj=default.rgw.users.uid:f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e state=0x55ec00063758 s->prefetch_data=0
2017-03-30 15:48:51.066508 7ff48e2c2700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e : type miss (requested=6, cached=0)
2017-03-30 15:48:51.066596 7ff48e2c2700 1 -- 10.30.200.141:0/3968062828 --> 10.30.200.143:6800/2371 -- osd_op(client.2717284.0:2140 6.12fd808b f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e943) v7 -- ?+0 0x55ec00065020 con 0x7ff4d402e150
2017-03-30 15:48:51.067487 7ff4dd6f9700 1 -- 10.30.200.141:0/3968062828 <== osd.11 10.30.200.143:6800/2371 269 ==== osd_op_reply(2140 f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 227+0+0 (2177119573 0 0) 0x7ff4e004adf0 con 0x7ff4d402e150
2017-03-30 15:48:51.067589 7ff48e2c2700 10 cache put: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e info.flags=0
2017-03-30 15:48:51.067595 7ff48e2c2700 10 moving default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e$f0f25423e84b4ad285c4a54ab15f7c1e to cache LRU end
2017-03-30 15:48:51.067614 7ff48e2c2700 20 get_system_obj_state: rctx=0x7ff48e2ba170 obj=default.rgw.users.uid:f0f25423e84b4ad285c4a54ab15f7c1e state=0x55ec00063758 s->prefetch_data=0
2017-03-30 15:48:51.067622 7ff48e2c2700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e : hit (requested=6, cached=7)
2017-03-30 15:48:51.067628 7ff48e2c2700 20 get_system_obj_state: s->obj_tag was set empty
2017-03-30 15:48:51.067631 7ff48e2c2700 10 cache get: name=default.rgw.users.uid+f0f25423e84b4ad285c4a54ab15f7c1e : hit (requested=1, cached=7)
2017-03-30 15:48:51.067654 7ff48e2c2700 2 req 11:0.001287:swift:GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:list_bucket:normalizing buckets and tenants
2017-03-30 15:48:51.067658 7ff48e2c2700 10 s->object=<NULL> s->bucket=AUTH_53eed6212bd54ff5acc4ad110e9899ac
2017-03-30 15:48:51.067662 7ff48e2c2700 2 req 11:0.001296:swift:GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:list_bucket:init permissions
2017-03-30 15:48:51.067674 7ff48e2c2700 20 get_system_obj_state: rctx=0x7ff48e2bb6c0 obj=default.rgw.data.root:AUTH_53eed6212bd54ff5acc4ad110e9899ac state=0x55ec00063758 s->prefetch_data=0
2017-03-30 15:48:51.067679 7ff48e2c2700 10 cache get: name=default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac : type miss (requested=22, cached=0)
2017-03-30 15:48:51.067720 7ff48e2c2700 1 -- 10.30.200.141:0/3968062828 --> 10.30.200.142:6808/2705 -- osd_op(client.2717284.0:2141 3.93c479fe AUTH_53eed6212bd54ff5acc4ad110e9899ac [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e943) v7 -- ?+0 0x55ec00067430 con 0x7ff4d4022240
2017-03-30 15:48:51.068355 7ff4ec290700 1 -- 10.30.200.141:0/3968062828 <== osd.4 10.30.200.142:6808/2705 76 ==== osd_op_reply(2141 AUTH_53eed6212bd54ff5acc4ad110e9899ac [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 241+0+0 (1282282941 0 0) 0x7ff4e004adf0 con 0x7ff4d4022240
2017-03-30 15:48:51.068439 7ff48e2c2700 10 cache put: name=default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac info.flags=0
2017-03-30 15:48:51.068447 7ff48e2c2700 10 moving default.rgw.data.root+AUTH_53eed6212bd54ff5acc4ad110e9899ac to cache LRU end
2017-03-30 15:48:51.068461 7ff48e2c2700 10 read_permissions on AUTH_53eed6212bd54ff5acc4ad110e9899ac(@[]) ret=-2002
2017-03-30 15:48:51.068464 7ff48e2c2700 20 op->ERRORHANDLER: err_no=-2002 new_err_no=-2002
2017-03-30 15:48:51.068538 7ff48e2c2700 2 req 11:0.002171:swift:GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:list_bucket:op status=0
2017-03-30 15:48:51.068541 7ff48e2c2700 2 req 11:0.002175:swift:GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac:list_bucket:http status=404
2017-03-30 15:48:51.068545 7ff48e2c2700 1 ====== req done req=0x7ff48e2bc7e0 op status=0 http_status=404 ======
2017-03-30 15:48:51.068553 7ff48e2c2700 20 process_request() returned -2002
2017-03-30 15:48:51.068585 7ff48e2c2700 1 civetweb: 0x55ec000459c0: 10.30.200.101 - - [30/Mar/2017:15:48:51 +0200] "GET /swift/v1/AUTH_53eed6212bd54ff5acc4ad110e9899ac HTTP/1.1" 404 0 - python-swiftclient-3.3.0
Have you got an idea what is going on here?
Updated by Albert Tu about 7 years ago
Try these workarounds
1. Set endpoint URL. But after changed it, public access to container will failed.
# Public interface
$ openstack endpoint set --url http://os-objectstore.mycompany.com:7480/swift/v1 {{ endpoint ID }}
2. Take a look at https://github.com/ceph/ceph/pull/14182
Updated by Jens Offenbach about 7 years ago
Thanks, but the swift endpoint is already available in service registry:
$ openstack endpoint list | grep swift
| 0428c21ef85a4007887591e9f0b1d50c | RegionOne | swift | object-store | True | admin | http://os-objectstore.mycompany.com:7480/swift/v1 |
| 10e153dc319d4810a591f50bac85a54c | RegionOne | swift | object-store | True | public | https://os-cloud.mycompany.com:7480/swift/v1 |
| df057a0caf0f41e3ae161ee05e2e8666 | RegionOne | swift | object-store | True | internal | http://os-objectstore.mycompany.com:7480/swift/v1 |
Ceilometer should use the internalURL.
ceilometer.conf:
[DEFAULT]
auth_strategy = keystone
debug = false
host = os-telemetry01
log_dir = /var/log/ceilometer
memcached_servers = os-memcache:11211
meter_dispatchers = database
transport_url = rabbit://ceilometer:secret@os-rabbit01:5672,ceilometer:secret@os-rabbit02:5672/openstack
[database]
event_connection = mysql+pymysql://ceilometer:secret@os-controller/ceilometer
metering_connection = mysql+pymysql://ceilometer:secret@os-controller/ceilometer
event_time_to_live = 604800
max_retries = -1
metering_time_to_live = 604800
[keystone_authtoken]
auth_type = password
auth_uri = http://os-identity:5000
auth_url = http://os-identity:35357
memcached_servers = os-memcache:11211
password = secret
project_domain_name = default
project_name = service
user_domain_name = default
username = ceilometer
[notification]
store_events = true
[oslo_concurrency]
lock_path = /var/lock/ceilometer
[oslo_messaging_notifications]
driver = messagingv2
[oslo_messaging_rabbit]
amqp_durable_queues = true
rabbit_ha_queues = true
rabbit_retry_backoff = 2
rabbit_retry_interval = 1
[oslo_middleware]
enable_proxy_headers_parsing = true
[service_credentials]
auth_type = password
auth_url = http://os-identity:35357/v3
interface = internalURL
password = secret
project_domain_name = default
project_name = service
region_name = RegionOne
user_domain_name = default
username = ceilometer
Updated by Albert Tu about 7 years ago
Try to add below setting to ceph.conf
rgw swift account in url = true
Updated by Jens Offenbach about 7 years ago
This modification only works in combination with modified Swift endpoints in Keystone's catalog:
| 0428c21ef85a4007887591e9f0b1d50c | RegionOne | swift | object-store | True | admin | http://os-objectstore.materna.com:7480/swift/v1/%(tenant_id)s |
| 10e153dc319d4810a591f50bac85a54c | RegionOne | swift | object-store | True | public | https://os-cloud.materna.com:7480/swift/v1/%(tenant_id)s |
| df057a0caf0f41e3ae161ee05e2e8666 | RegionOne | swift | object-store | True | internal | http://os-objectstore.materna.com:7480/swift/v1/%(tenant_id)s |
Is this the recommended approach?
Updated by Radoslaw Zarzynski about 7 years ago
The object store is fully functional, but the Ceilometer log contains the following entry which occurrs every 10 minutes
It looks like Ceilometer is trying to use account-in-URL schema. If object store's endpoints in Keystone have been set to not contain this part, then Ceilometer might have a bug. AFAIK OpenStack components should stick to the info they got from Keystone.
Updated by Radoslaw Zarzynski about 7 years ago
Jens Offenbach wrote:
This modification only works in combination with modified Swift endpoints in Keystone's catalog:
[...]Is this the recommended approach?
Yup, if you want to use rgw swift account in url = true, then endpoints is Keystone should be updated accordingly.
You might be also interested in rgw_keystone_implicit_tenants.
Updated by Donny Davis about 7 years ago
I had the same things happening, with both user accounts and with service accounts. This is how I fixed it, not sure if its correct but it works.
I updated my endpoints to reflect tenant ids as stated above
http://swift.example.com:7480/swift/v1/%(tenant_id)s public
http://127.0.0.1:7480/swift/v1/%(tenant_id)s admin
http://127.0.0.1:7480/swift/v1/%(tenant_id)s internal
and in ceph.conf I added the following to the rgw
rgw swift account in url = true
rgw keystone implicit tenants = false
I also has to delete and recreate my pools for it to work.
This also allows a user to view public containers created from the horizon UI.
This should be a documented process.
Actions