Project

General

Profile

Actions
Copy link

Bug #64028

open

Ceph dasbboard shows RGW REST API failed request with status code 403 (InvalidAccessKeyId)

Added by Morteza Bashsiz 4 months ago. Updated 3 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
v18.2.1
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

After upgrading ceph to 18.2.0 we faced with this issue https://tracker.ceph.com/issues/63698
Now after upgrading to 18.2.1 Object Gateway part of the ceph dashboard is not working and show error

Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"InvalidAccessKeyId","Message":"","RequestId":"tx00000da77a9856259c1' b'ab-0065a52386-1ed8cf0-host","HostId":"host"}')

[~]>$ ceph dashboard get-rgw-api-secret-key
{'realm': 'secret-key-string-sssssssssssssssssssssssss'}

[~]>$ ceph dashboard get-rgw-api-access-key
{'realm': 'access-key-string-aaaaaaaaaaa'}

[~]>$ ceph dashboard get-rgw-api-ssl-verify 
False

[~]>$ radosgw-admin user info --uid dashboard
{
    "user_id": "dashboard",
    "display_name": "Ceph Dashboard",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "dashboard",
            "access_key": "access-key-string-aaaaaaaaaaa",
            "secret_key": "secret-key-string-sssssssssssssssssssssssss" 
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "system": true,
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

ceph-mgr log

[dashboard ERROR rest_client] RGW REST API failed GET req status: 403
[dashboard ERROR rgw_client] RGW REST API failed request with status code 403
     (b'{"Code":"InvalidAccessKeyId","Message":"","RequestId":"tx00000da77a9856259c1'
      b'ab-0065a52386-1ed8cf0-host1","HostId":"1ed8cf0-host1-host"}')
     Traceback (most recent call last):
       File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 431, in __init__
         self.userid = self._get_user_id(self.admin_path) if self.got_keys_from_config \
       File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 542, in func_wrapper
         **kwargs)
       File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 466, in _get_user_id
         response = request()
       File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 325, in __call__
         data, raw_content, headers)
       File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 428, in do_request
         resp.content)
     dashboard.rest_client.RequestException: RGW REST API failed request with status code 403
     (b'{"Code":"InvalidAccessKeyId","Message":"","RequestId":"tx00000da77a9856259c1'
      b'ab-0065a52386-1ed8cf0-host1","HostId":"1ed8cf0-host1-host"}')

rgw log

====== starting new request req=0x7fcf96400660 =====
req 9769990003208590416 0.000000000s s3:get_obj No stored secret string, cache miss
req 9769990003208590416 0.019999748s s3:get_obj No stored secret string, cache miss
req 9769990003208590416 0.043999448s op->ERRORHANDLER: err_no=-2028 new_err_no=-2028
====== req done req=0x7fcf96400660 op status=0 http_status=403 latency=0.043999448s ======

Actions
Copy link
 #1

Updated by Morteza Bashsiz 3 months ago

When I enable the debug on mgr
I see following logs

Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/AUDIT_API_ENABLED
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config  AUDIT_API_ENABLED not found 
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_store get_store key: mgr/dashboard/jwt_token_block_list
Jan 25 11:07:57 hostname ceph-mgr[45241]: MonCommandCompletion::finish()
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr Gil Switched to new thread state 0x555ed1fd5600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr ~Gil Destroying new thread state 0x555ed1fd5600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr notify_all notify_all: notify_all command
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr notify_all queuing notify (command) to restful
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr Gil Switched to new thread state 0x555ed1825600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr ~Gil Destroying new thread state 0x555ed1825600
Jan 25 11:07:57 hostname ceph-mgr[45241]: MonCommandCompletion::finish()
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr Gil Switched to new thread state 0x555ed1fd5600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr ~Gil Destroying new thread state 0x555ed1fd5600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr notify_all notify_all: notify_all command
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr notify_all queuing notify (command) to restful
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr Gil Switched to new thread state 0x555ed1825600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr ~Gil Destroying new thread state 0x555ed1825600
Jan 25 11:07:57 hostname ceph-mgr[45241]: MonCommandCompletion::finish()
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr Gil Switched to new thread state 0x555ed1fd5600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr ~Gil Destroying new thread state 0x555ed1fd5600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr notify_all notify_all: notify_all command
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr notify_all queuing notify (command) to restful
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr Gil Switched to new thread state 0x555ed1825600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr ~Gil Destroying new thread state 0x555ed1825600
Jan 25 11:07:57 hostname ceph-mgr[45241]: MonCommandCompletion::finish()
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr Gil Switched to new thread state 0x555ed1fd5600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr ~Gil Destroying new thread state 0x555ed1fd5600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr notify_all notify_all: notify_all command
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr notify_all queuing notify (command) to restful
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/RGW_API_ACCESS_KEY
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr Gil Switched to new thread state 0x555ed1825600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr ~Gil Destroying new thread state 0x555ed1825600
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config get_typed_config RGW_API_ACCESS_KEY found
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/RGW_API_SECRET_KEY
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config get_typed_config RGW_API_SECRET_KEY found
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/RGW_API_ACCESS_KEY
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config get_typed_config RGW_API_ACCESS_KEY found
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/RGW_API_SECRET_KEY
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config get_typed_config RGW_API_SECRET_KEY found
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/RGW_API_ADMIN_RESOURCE
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config  RGW_API_ADMIN_RESOURCE not found 
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/RGW_API_SSL_VERIFY
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config get_typed_config RGW_API_SSL_VERIFY found
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/RGW_API_ACCESS_KEY
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config get_typed_config RGW_API_ACCESS_KEY found
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/RGW_API_ACCESS_KEY
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config get_typed_config RGW_API_ACCESS_KEY found
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/RGW_API_SECRET_KEY
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config get_typed_config RGW_API_SECRET_KEY found
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/RGW_API_SSL_VERIFY
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config get_typed_config RGW_API_SSL_VERIFY found
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/RGW_API_ADMIN_RESOURCE
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config  RGW_API_ADMIN_RESOURCE not found 
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/REST_REQUESTS_TIMEOUT
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config  REST_REQUESTS_TIMEOUT not found 
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_config  key: mgr/dashboard/REST_REQUESTS_TIMEOUT
Jan 25 11:07:57 hostname ceph-mgr[45241]: mgr get_typed_config  REST_REQUESTS_TIMEOUT not found
Actions
Copy link
 #2

Updated by Morteza Bashsiz 3 months ago

I found out following logs which maybe it is related to integration rgw with keystone same as https://tracker.ceph.com/issues/22632

s3:list_bucket No stored secret string, cache miss

and also I have checked the keystone logs. It seems that rgw wants to reach endpoint `/v3/s3tokens` from keystone which it returns 404

10.33.0.4 - - [26/Jan/2024:07:21:55 +0000] "POST /v3/s3tokens HTTP/1.1" 404 148 "-" "-"

And keystone uwsg log

Jan 26 07:46:31 ybk140927 keystone-wsgi-public[3942390]: 2024-01-26 07:46:31.821 54 WARNING keystone.server.flask.application [req-2e11c930-a397-4c49-ad9e-d7769d083e53 47f4602c00b746248389b8205d1e4a0f 2aedcdae9eea4e39b6a5764bbb2f43bb - default default] Could not find credential: 760dc155b6c16b5d8a29a4d198e5ed53212fefb0d5821a8eb5d328b70848724b.: keystone.exception.CredentialNotFound: Could not find credential: 760dc155b6c16b5d8a29a4d198e5ed53212fefb0d5821a8eb5d328b70848724b.
Actions
Copy link

Also available in: Atom PDF