Bug #58811
openmgr/dashboard: RGW broken on Pacific 16.2.11 after upgrade
0%
Description
Description of problem¶
We were running Proxmox Virtual Environment (PVE) with Ceph Pacific 16.2.9, which does not include/provide cephadm. We had however managed to setup the Ceph Dashboard together with the RGW module and could use that to administer S3 compatible buckets and user accounts. After upgrading from 16.2.9 to 16.2.11 accessing RGW no longer works via the Dashboard, yielding a 403 'SignatureDoesNotMatch' error.
When we subsequently upgraded to Quincy 17.2.5 functionality returned.
My concern is that there is a patch included in 16.2.11 which will eventually find its way in to Quincy, which will then cause this problem there as well...
I tried enabling debugging but it wouldn't log internal information when it should have been talking with the RADOS gateway, commands I used were:
```
ceph dashboard debug enable
ceph tell mgr config set debug_mgr 20
```
Possibly worth noting is that when previously running civetweb we could enable SSL verification when the Dashboard would interact with SSL enabled RGW, since running beast we need to disable SSL verify (RGW_API_SSL_VERIFY), otherwise this results in a 500 error code.
Configuration statements:
```
[admin@kvm7a ~]# radosgw-admin user info --uid=dashboard | grep _key.:
"access_key": "********************",
"secret_key": "****************************************"
[admin@kvm7a ~]# ceph config dump | grep -i -e rgw -e dash
mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY ************
mgr advanced mgr/dashboard/RGW_API_SECRET_KEY ********************************
mgr advanced mgr/dashboard/RGW_API_SSL_VERIFY false
mgr advanced mgr/dashboard/debug false
mgr advanced mgr/dashboard/server_addr 0.0.0.0
mgr advanced mgr/dashboard/ssl true
client.radosgw.kvm7a advanced rgw_dns_name kvm7a.redacted.com
client.radosgw.kvm7a basic rgw_frontends beast ssl_port=7480 ssl_certificate=/etc/pve/local/radosgw-ssl.pem
client.radosgw.kvm7b advanced rgw_dns_name kvm7b.redacted.com
client.radosgw.kvm7b basic rgw_frontends beast ssl_port=7480 ssl_certificate=/etc/pve/local/radosgw-ssl.pem
client.radosgw.kvm7e advanced rgw_dns_name kvm7e.redacted.com
client.radosgw.kvm7e basic rgw_frontends beast ssl_port=7480 ssl_certificate=/etc/pve/local/radosgw-ssl.pem
```
Environment¶
ceph versionstring: 16.2.11- Platform (OS/distro/release): Debian 11.6 (Proxmox Virtual Environment (aka PVE) 7.3)
- Cluster details (nodes, monitors, OSDs): Standard
- Did it happen on a stable environment or after a migration/upgrade?: After upgrading from Ceph Pacific 16.2.9 to 16.2.11, working again after upgrading to Quincy 17.2.5
- Browser used (e.g.:
Version 86.0.4240.198 (Official Build) (64-bit)): Chrome 109.0.5414.120
How reproducible¶
Steps:
Validated on two separate clusters, herewith is the section from the mgr log after enabling debugging with a debug level of 20. It enumerates variables but then doesn't actually log the RGW interactions which are failing:
```
2023-02-19T12:48:04.965+0200 7f24607fb700 10 mgr.server ms_handle_authentication ms_handle_authentication new session 0x5592faf37200 con 0x55930353e400 entity client.admin addr
2023-02-19T12:48:04.965+0200 7f24607fb700 10 mgr.server ms_handle_authentication session 0x5592faf37200 client.admin has caps allow * 'allow *'
2023-02-19T12:48:05.001+0200 7f24489c1700 10 module pg_autoscaler health checks:
2023-02-19T12:48:05.001+0200 7f2443b78700 20 mgr get_config key: mgr/rbd_support/mirror_snapshot_schedule
2023-02-19T12:48:05.001+0200 7f2443b78700 10 mgr get_typed_config mirror_snapshot_schedule not found
2023-02-19T12:48:05.025+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/AUDIT_API_ENABLED
2023-02-19T12:48:05.025+0200 7f24275c0700 10 mgr get_typed_config AUDIT_API_ENABLED not found
2023-02-19T12:48:05.025+0200 7f24275c0700 4 mgr get_store get_store key: mgr/dashboard/jwt_token_block_list
2023-02-19T12:48:05.025+0200 7f24275c0700 10 ceph_store_get jwt_token_block_list found: {"cd1a1a80-bca1-48ba-89fe-cd4d061a043b": 1676602596}
2023-02-19T12:48:05.025+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_ACCESS_KEY
2023-02-19T12:48:05.025+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_ACCESS_KEY found
2023-02-19T12:48:05.025+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_SECRET_KEY
2023-02-19T12:48:05.025+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_SECRET_KEY found
2023-02-19T12:48:05.025+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_ACCESS_KEY
2023-02-19T12:48:05.025+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_ACCESS_KEY found
2023-02-19T12:48:05.025+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_SECRET_KEY
2023-02-19T12:48:05.029+0200 7f2442375700 20 mgr get_config key: mgr/rbd_support/trash_purge_schedule
2023-02-19T12:48:05.029+0200 7f2442375700 10 mgr get_typed_config trash_purge_schedule not found
2023-02-19T12:48:05.029+0200 7f2443b78700 20 mgr get_config key: mgr/rbd_support/kvm7b/mirror_snapshot_schedule
2023-02-19T12:48:05.029+0200 7f2443b78700 20 mgr get_config key: mgr/rbd_support/mirror_snapshot_schedule
2023-02-19T12:48:05.029+0200 7f2443b78700 10 mgr get_typed_config [kvm7b/]mirror_snapshot_schedule not found
2023-02-19T12:48:05.029+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_SECRET_KEY found
2023-02-19T12:48:05.049+0200 7f2442375700 20 mgr get_config key: mgr/rbd_support/kvm7b/trash_purge_schedule
2023-02-19T12:48:05.049+0200 7f2442375700 20 mgr get_config key: mgr/rbd_support/trash_purge_schedule
2023-02-19T12:48:05.049+0200 7f2442375700 10 mgr get_typed_config [kvm7b/]trash_purge_schedule not found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_ADMIN_RESOURCE
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_ADMIN_RESOURCE found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_SSL_VERIFY
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_SSL_VERIFY found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_ACCESS_KEY
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_ACCESS_KEY found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_ACCESS_KEY
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_ACCESS_KEY found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_SECRET_KEY
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_SECRET_KEY found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_SSL_VERIFY
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_SSL_VERIFY found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_ADMIN_RESOURCE
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_ADMIN_RESOURCE found
2023-02-19T12:48:05.093+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/REST_REQUESTS_TIMEOUT
2023-02-19T12:48:05.093+0200 7f24275c0700 10 mgr get_typed_config REST_REQUESTS_TIMEOUT not found
2023-02-19T12:48:05.093+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/REST_REQUESTS_TIMEOUT
2023-02-19T12:48:05.093+0200 7f24275c0700 10 mgr get_typed_config REST_REQUESTS_TIMEOUT not found
2023-02-19T12:48:05.181+0200 7f244c288700 10 mgr.server tick
```
Additional info¶
```
[admin@kvm7a ~]# ceph dashboard feature status
Feature 'rbd': enabled
Feature 'mirroring': enabled
Feature 'iscsi': enabled
Feature 'cephfs': enabled
Feature 'rgw': enabled
Feature 'nfs': enabled
[admin@kvm7a ~]# ceph mgr services | jq .dashboard
"https://10.250.1.3:8443/"
```
Updated by Gilles Mocellin about 1 year ago
Hello,
I've got the same issue.
For me, it seems to be network related : I have a management network which is not the Ceph public Network.
The host names of nodes resolve on that management network.
It seems that between 16.2.10 and 16.2.11, the MGR tries to discover/detect/connect to radosgw instances by host name and not any more by IP (known by the MONs ?).
Updated by Gilles Mocellin about 1 year ago
In my config, I can see variables that could be used to connect to radosgw instances. They are still good, but perhaps not used :
Several :
mgr/dashboard/mynodename/server_addr W.X.Y.Z
Also :
mgr/dashboard/RGW_API_PORT 8080
mgr/dashboard/RGW_API_*
but no No RGW_API_HOST or similar.
Updated by Ilya Dryomov about 1 year ago
- Target version changed from v16.2.12 to v16.2.13
Updated by Patrick Preisler 10 months ago
I've recently upgraded from 17.2.5 to 17.2.6 and the Dashboard RGW Plugin shows that it is not configured. Following the Documentation I ran "ceph dashboard set-rgw-credentials" however this doesn't change anything. After doing some troubleshooting and not finding anything helpful, I went back to v17.2.5 and everything works again. To me it seems your concern about the patch included in 16.2.11 which breaks the RGW Plugin actually found its way to Quincy.