Bug #62396: mgr/dashboard: after upgrade to reef - The Object Gateway Service is not configured - Dashboard - Ceph

Actions

Copy link

Bug #62396

closed

mgr/dashboard: after upgrade to reef - The Object Gateway Service is not configured

Added by Petr Bena 9 months ago. Updated 8 months ago.

Status:

Resolved

Priority:

Normal

Assignee:

Nizamudeen A

Category:

Component - RGW

Target version:

% Done:

Source:

Tags:

backport_processed

Backport:

reef, quincy, pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

53133

Crash signature (v1):

Crash signature (v2):

Description

Description of problem¶

After upgrade to Reef from Quincy Object Gateway section in Dashboard is broken:

The Object Gateway Service is not configured
Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx000005666bdef0ce' b'5934e-0064d4e476-2f83e-default","HostId":"2f83e-default-default"}')
Please consult the documentation on how to configure and enable the Object Gateway management functionality.

Link points to https://docs.ceph.com/en/latest/mgr/dashboard/#enabling-the-object-gateway-management-frontend which contains no useful information how to resolve this problem. Only information there is that by running set-rgw-credentials I can configure credentials, but that doesn't do anything, really:

ceph dashboard set-rgw-credentials
RGW credentials configured

via

radosgw-admin user list
[
"dashboard",

I can see that dashboard user exists (amongst many others).

Environment¶

ceph version string: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)
Platform (OS/distro/release): Ubuntu 22.04
Cluster details (nodes, monitors, OSDs): 4 physical nodes, 4 mons, 72 OSDs
Did it happen on a stable environment or after a migration/upgrade?: after migration to reef
Browser used (e.g.: Version 86.0.4240.198 (Official Build) (64-bit)):Firefox

How reproducible¶

N/A

Actual results¶

Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx000005666bdef0ce' b'5934e-0064d4e476-2f83e-default","HostId":"2f83e-default-default"}')

Expected results¶

Working dashboard

Additional info¶

I couldn't find any useful logs in mgr container, in fact it logs nothing related to RGW dashboard there

Related issues 3 (0 open — 3 closed)

Actions

Copy link

Updated by Petr Bena 9 months ago

I can find this in MGR logs

RGW REST API failed GET req status: 403
RGW REST API failed request with status code 403
(b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000e94834dee1f'
b'126e2-0064d612aa-2f83e-default","HostId":"2f83e-default-default"}')
Traceback (most recent call last):
File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 417, in init
self.userid = self._get_user_id(self.admin_path) if self.got_keys_from_config \
File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 542, in func_wrapper
**kwargs)
File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 452, in _get_user_id
response = request()
File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 325, in call
data, raw_content, headers)
File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 428, in do_request
resp.content)
dashboard.rest_client.RequestException: RGW REST API failed request with status code 403
(b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000e94834dee1f'
b'126e2-0064d612aa-2f83e-default","HostId":"2f83e-default-default"}')

Actions

Copy link

Updated by Petr Bena 9 months ago

So we figured out the problem was that since Reef dashboard started accessing the RGW using instance names, instead of FQDN that could be specified via custom variable in previous CEPH versions.

Since this variable can no longer be specified, a workaround was to modify the zone json config using

radosgw-admin zonegroup get

Then adding the short hostnames of instances into hostnames field and then committing the changed json back to RGW config

Actions

Copy link

Updated by Petr Bena 9 months ago

We found this solution in https://www.spinics.net/lists/ceph-users/msg76635.html

Actions

Copy link

Updated by brent s. 8 months ago

Directly related:
https://tracker.ceph.com/issues/59111

It would seem to me the simplest solution would just simply be to explicitly respect

rgw_dns_name

. Of course, a more thorough solution should/would be implemented, IMHO, via the following:

If
```
rgw_dns_name
```
is configured for a daemon, use that. If not...
If:
1. SSL/TLS (HTTPS) IS configured for the daemon: Attempt connection to the daemon's hostname (current behavior); if this fails (times out, does not resolve), continue to...
2. SSL/TLS (HTTPS) IS NOT configured for the daemon: Attempt connection to the daemon's IP address (previous behavior); if this fails, continue to...
If:
1. SSL/TLS (HTTPS) IS configured for the daemon: Attempt connection to the daemon's IP address
2. SSL/TLS (HTTPS) IS NOT configured for the daemon: Attempt connection to the daemon's hostname
Lastly, if none of the above succeed, return errors and consider RGW daemon as inaccessible.

This precise ordering allows for:

"automagic" configuration
explicitly overriding said automagic handling (especially important in the event of different network scoping)
SSL/TLS conditions, both with and without the IP address SAN'd on the certificate
No DNS configuration whatsoever (which, sure, is silly for an RGW, but best to not make assumptions of consumer infra/usage)

Actions

Copy link