Bug #38428
mgr/dashboard: dashboard giving 401 unauthorized
0%
Description
Hello Team,
I have set up the ceph dashboard and its allowing me to login but immediately after login it log me out.
From logs I can see below
2019-02-22 13:44:33.919 7fbecd4fe700 0 mgr[dashboard] frontend error (https://192.168.1.142:8443/#/login): Http failure response for https://192.168.1.142:8443/api/summary: 401 Unauthorized
cephaio:/var/log/ceph # netstat -an| grep 8443
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN
cephaio:/usr/lib64/ceph/mgr/dashboard # radosgw-admin user info --uid ceph-admin
{
"user_id": "ceph-admin",
"display_name": "ceph admin",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "ceph-admin",
"access_key": "ANXQI47HUI7T00G1UQFC",
"secret_key": "EShhS8EcAFeEvBXDabzUV9JCaAJjvB93CsQD9Nqg"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"system": "true",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}
cephaio:/usr/lib64/ceph/mgr/dashboard # sudo ceph dashboard get-rgw-api-user-id
ceph-admin
cephaio:/usr/lib64/ceph/mgr/dashboard # ceph dashboard get-rgw-api-access-key
ANXQI47HUI7T00G1UQFC
cephaio:/usr/lib64/ceph/mgr/dashboard # ceph dashboard get-rgw-api-secret-key
EShhS8EcAFeEvBXDabzUV9JCaAJjvB93CsQD9Nqg
cephaio:/usr/lib64/ceph/mgr/dashboard # ceph dashboard get-rgw-api-ssl-verify
False
cephaio:/var/log/ceph # ceph dashboard ac-user-show ceph-admin
{"username": "ceph-admin", "password": "$2b$12$62HYc/RwUN4T/Sdp.z5SgexAlJghnRvAS8htab9aXO.pGsgh4CTNi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1550721565}
cephaio:/var/log/ceph #
cephaio:/var/log/ceph # ceph dashboard get-rgw-api-access-key
ANXQI47HUI7T00G1UQFC
cephaio:/var/log/ceph # ceph dashboard get-rgw-api-secret-key
EShhS8EcAFeEvBXDabzUV9JCaAJjvB93CsQD9Nqg
cephaio:/var/log/ceph # ceph dashboard get-rgw-api-host
cephaio
cephaio:/var/log/ceph # ceph dashboard get-rgw-api-port
8003
cephaio:/var/log/ceph # ceph mgr services
{
"dashboard": "https://0.0.0.0:8443/",
"restful": "https://cephaio:8003/"
}
cephaio:/var/log/ceph # ceph --version
ceph version 14.0.1-3975-g14d8f3cab5 (14d8f3cab592c4b3ac85000ccd8fc9fe03398428) nautilus (dev)
cephaio:/var/log/ceph #
cluster:
id: bfdfbe86-3ee9-417e-a3bd-0818694a18d0
health: HEALTH_WARN
application not enabled on 2 pool(s)
services:
mon: 1 daemons, quorum cephaio (age 2h)
mgr: cephaio(active, since 2m)
osd: 1 osds: 1 up (since 2h), 1 in (since 27h)
rgw: 1 daemon active
data:
pools: 6 pools, 34 pgs
objects: 191 objects, 5.2 KiB
usage: 45 MiB used, 979 MiB / 1 GiB avail
pgs: 34 active+clean
cephaio:/var/log/ceph #
Related issues
History
#1 Updated by Alfonso Martínez about 5 years ago
Tested Dashboard login with Nautilus 14.1.0 (2019-02-22, https://download.ceph.com/rpm-nautilus/el7/x86_64/):
Issue not reproducible (it works as expected).
Can you test this version?
#2 Updated by Lenz Grimmer about 5 years ago
- Status changed from New to Need More Info
Can you test if the provided RGW credentials actually work? I wonder if the dashboard reacts to the 401 error that is returned by RGW by any chance?
#3 Updated by Volker Theile about 5 years ago
The RGW REST API controller had a problem that might cause this issue. If the RGW Admin OPS API returned an 401 it was directly forwarded to the UI which forces a logout. This has been fixed ages ago with https://github.com/ceph/ceph/commit/ee5c9ad03c9d301d01716e56710fe24a67f0f869. Nowadays all errors raised by the RGW Admin OPS API are returned as 500 (see https://github.com/ceph/ceph/blob/master/src/pybind/mgr/dashboard/controllers/rgw.py#L103).
2019-02-22 13:44:33.919 7fbecd4fe700 0 mgr[dashboard] frontend error (https://192.168.1.142:8443/#/login): Http failure response for https://192.168.1.142:8443/api/summary: 401 Unauthorized
To me it looks like the /api/summary API call is responsible for this. But the controller code itself does not throw exceptions, so it might happen somewhere on the way when the REST call is processed.
#4 Updated by Volker Theile about 5 years ago
- Subject changed from ceph-dashboard giving 401 unauthorized to mgr/dashboard: dashboard giving 401 unauthorized
#5 Updated by Volker Theile about 5 years ago
- File Auswahl_002.png View added
Can you provide us a screenshot like this. Maybe this can help us to identify the problem.
#6 Updated by Lenz Grimmer about 5 years ago
Hi Lokesh, something in your report looks incorrect:
Lokesh Pashine wrote:
cephaio:/var/log/ceph # ceph dashboard get-rgw-api-host
cephaio
cephaio:/var/log/ceph # ceph dashboard get-rgw-api-port
8003cephaio:/var/log/ceph # ceph mgr services {
"dashboard": "https://0.0.0.0:8443/",
"restful": "https://cephaio:8003/"
}
This looks to me as if you've configured the dashboard to talk to the restful module on https://cephaio:8003/ is if it was the RGW API endpoint. Note that the restful module is not needed by the dashboard. If you want to manage RGW via the dashboard, make sure to use the correct hostname/TCP port to talk to the RGW admin ops API.
#7 Updated by Lokesh Pashine about 5 years ago
Lenz Grimmer wrote:
Hi Lokesh, something in your report looks incorrect:
Lokesh Pashine wrote:
cephaio:/var/log/ceph # ceph dashboard get-rgw-api-host
cephaio
cephaio:/var/log/ceph # ceph dashboard get-rgw-api-port
8003cephaio:/var/log/ceph # ceph mgr services {
"dashboard": "https://0.0.0.0:8443/",
"restful": "https://cephaio:8003/"
}This looks to me as if you've configured the dashboard to talk to the restful module on https://cephaio:8003/ is if it was the RGW API endpoint. Note that the restful module is not needed by the dashboard. If you want to manage RGW via the dashboard, make sure to use the correct hostname/TCP port to talk to the RGW admin ops API.
==============
Hi Lenz,
I also noticed the configuration missmatch and corrected it already but issue is still the same
cephaio:~ # ceph dashboard get-rgw-api-access-key
35S5T7IHEXPDRGZ012A6
cephaio:~ # ceph dashboard get-rgw-api-secret-key
mKkouV45flvszN5w4z7BAAL6EHO5375HhVQMyWj4
cephaio:~ # ceph dashboard get-rgw-api-host
cephaio
cephaio:~ # ceph dashboard get-rgw-api-port
7480
cephaio:~ # ceph dashboard get-rgw-api-ssl-verify
False
cephaio:~ # ceph dashboard get-rgw-api-user-id
radosadm
cephaio:~ # lsof -i:7480
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
radosgw 1559 ceph 40u IPv4 28900 0t0 TCP *:7480 (LISTEN)
cephaio:~ #
Please suggest what is wrong here and if I have to try upgrading the dashboard package?
#8 Updated by Lokesh Pashine about 5 years ago
- File ceph-dashboard-response.jpg View added
Volker Theile wrote:
Can you provide us a screenshot like this. Maybe this can help us to identify the problem.
============
Hi Volker,
As requested attached is the response code screenshot
#9 Updated by Lokesh Pashine about 5 years ago
! ceph-dashboard-response.jpg!
#10 Updated by Lokesh Pashine about 5 years ago
!"C:\Users\I355971\Desktop\Temp\ceph-dashboard-response.jpg!
#11 Updated by Lokesh Pashine about 5 years ago
Alfonso MH wrote:
Tested Dashboard login with Nautilus 14.1.0 (2019-02-22, https://download.ceph.com/rpm-nautilus/el7/x86_64/):
Issue not reproducible (it works as expected).
Can you test this version?
=========
Hello Alfonso,
DO I need to upgrade all the components or just by upgrading ceph-mgr will do?
#12 Updated by Nathan Cutler about 5 years ago
DO I need to upgrade all the components or just by upgrading ceph-mgr will do?
As a general rule, always upgrade all components (RPMs, DEBs). As a project, Ceph is rather monolithic and there are many interdependencies between the components.
In this specific case, it's possible the fix is contained within the ceph-mgr-dashboard package, in which case only that package would need to be upgraded.
#13 Updated by echo wen about 5 years ago
It looks like i have not only eth, but can i fix it
#14 Updated by echo wen about 5 years ago
迁 文 wrote:
It looks like i have not only eth, but how can i fix it
#15 Updated by echo wen about 5 years ago
迁 文 wrote:
#16 Updated by Volker Theile about 5 years ago
- Duplicated by Bug #38600: mgr/dashboard: dashboard giving 401 unauthorized. added
#17 Updated by 一帆 师 about 5 years ago
Lokesh Pashine wrote:
Hello Team,
I have set up the ceph dashboard and its allowing me to login but immediately after login it log me out.
From logs I can see below
2019-02-22 13:44:33.919 7fbecd4fe700 0 mgr[dashboard] frontend error (https://192.168.1.142:8443/#/login): Http failure response for https://192.168.1.142:8443/api/summary: 401 Unauthorizedcephaio:/var/log/ceph # netstat -an| grep 8443
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTENcephaio:/usr/lib64/ceph/mgr/dashboard # radosgw-admin user info --uid ceph-admin {
"user_id": "ceph-admin",
"display_name": "ceph admin",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [ {
"user": "ceph-admin",
"access_key": "ANXQI47HUI7T00G1UQFC",
"secret_key": "EShhS8EcAFeEvBXDabzUV9JCaAJjvB93CsQD9Nqg"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"system": "true",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}cephaio:/usr/lib64/ceph/mgr/dashboard # sudo ceph dashboard get-rgw-api-user-id
ceph-admincephaio:/usr/lib64/ceph/mgr/dashboard # ceph dashboard get-rgw-api-access-key
ANXQI47HUI7T00G1UQFCcephaio:/usr/lib64/ceph/mgr/dashboard # ceph dashboard get-rgw-api-secret-key
EShhS8EcAFeEvBXDabzUV9JCaAJjvB93CsQD9Nqgcephaio:/usr/lib64/ceph/mgr/dashboard # ceph dashboard get-rgw-api-ssl-verify
Falsecephaio:/var/log/ceph # ceph dashboard ac-user-show ceph-admin {"username": "ceph-admin", "password": "$2b$12$62HYc/RwUN4T/Sdp.z5SgexAlJghnRvAS8htab9aXO.pGsgh4CTNi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1550721565}
cephaio:/var/log/ceph #cephaio:/var/log/ceph # ceph dashboard get-rgw-api-access-key
ANXQI47HUI7T00G1UQFCcephaio:/var/log/ceph # ceph dashboard get-rgw-api-secret-key
EShhS8EcAFeEvBXDabzUV9JCaAJjvB93CsQD9Nqgcephaio:/var/log/ceph # ceph dashboard get-rgw-api-host
cephaio
cephaio:/var/log/ceph # ceph dashboard get-rgw-api-port
8003cephaio:/var/log/ceph # ceph mgr services {
"dashboard": "https://0.0.0.0:8443/",
"restful": "https://cephaio:8003/"
}cephaio:/var/log/ceph # ceph --version
ceph version 14.0.1-3975-g14d8f3cab5 (14d8f3cab592c4b3ac85000ccd8fc9fe03398428) nautilus (dev)
cephaio:/var/log/ceph #cluster:
id: bfdfbe86-3ee9-417e-a3bd-0818694a18d0
health: HEALTH_WARN
application not enabled on 2 pool(s)services:
mon: 1 daemons, quorum cephaio (age 2h)
mgr: cephaio(active, since 2m)
osd: 1 osds: 1 up (since 2h), 1 in (since 27h)
rgw: 1 daemon activedata:
pools: 6 pools, 34 pgs
objects: 191 objects, 5.2 KiB
usage: 45 MiB used, 979 MiB / 1 GiB avail
pgs: 34 active+cleancephaio:/var/log/ceph #
I resolved this bug by change my timezone from CST to EDT.
#18 Updated by Lenz Grimmer about 5 years ago
一帆 师 wrote:
I resolved this bug by change my timezone from CST to EDT.
Can you please share where you changed the time zone?
#19 Updated by Fareez Azhar about 5 years ago
Im on 14.2.0 changing timezone.. issue still there.. 401 api/summary
#20 Updated by 一帆 师 about 5 years ago
Fareez Azhar wrote:
Im on 14.2.0 changing timezone.. issue still there.. 401 api/summary
try more time zone, my computer's time zone is Asia/Shanghai, and my server's time need to be EDT.
so if your computer's time zone is not CST, maybe another time zone of your server different of EDT will work.
#21 Updated by Lenz Grimmer about 5 years ago
- Status changed from Need More Info to In Progress
- Target version set to v15.0.0
- Backport set to nautilus
- Pull request ID set to 27098
A patch for this has been proposed: https://github.com/ceph/ceph/pull/27098
#22 Updated by 一帆 师 about 5 years ago
Lenz Grimmer wrote:
A patch for this has been proposed: https://github.com/ceph/ceph/pull/27098
thks,I changed the code as "https://github.com/ceph/ceph/pull/27098",I works now.
#23 Updated by Lenz Grimmer about 5 years ago
- Status changed from In Progress to Pending Backport
- Affected Versions v14.0.0, v14.2.0 added
#24 Updated by Nathan Cutler about 5 years ago
- Copied to Backport #38871: nautilus: mgr/dashboard: dashboard giving 401 unauthorized added
#25 Updated by Lenz Grimmer about 5 years ago
- Status changed from Pending Backport to Resolved
#26 Updated by Lenz Grimmer almost 5 years ago
- Related to Bug #39300: mgr/dashboard: Can't login with a bigger time difference between user and server or make auth token work with UTC times only added
#27 Updated by Darrell Enns almost 5 years ago
- File screenshot.png View added
I am seeing this exact same behaviour on 14.2.1 (which includes the patch). Client and server are in the same time zone and clocks are in sync.
# ceph version ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
- cat /usr/share/ceph/mgr/dashboard/services/auth.py |head -n48|tail -n5
ttl = mgr.get_module_option('jwt_token_ttl', cls.JWT_TOKEN_TTL) ttl = int(ttl) now = int(time.time()) payload = { 'iss': 'ceph-dashboard',
#28 Updated by Lenz Grimmer almost 5 years ago
Can you please try the following:
- Manually apply the change posted at https://github.com/ceph/ceph/pull/27942/files to the file
/usr/share/ceph/mgr/dashboard/services/access_control.py
on all nodes running the dashboard - Restart the dashboard using
ceph mgr module disable dashboard; ceph mgr module enable dashboard
- Update the timestamp on admin's password:
ceph dashboard ac-user-set-password admin <same_password_as_before>
- Try to login again
#29 Updated by Darrell Enns almost 5 years ago
That patch is already applied in 14.2.1, and I have reset passwords many times already.
It looks like in my case it was caused by some sort of corruption of the config database. I began to suspect it when I noticed some odd behaviour when trying to set config options. I tried to change an option (mon_osd_down_out_subtree_limit) and it made a duplicate setting instead. I could remove the duplicate, but not the original. I ended up moving my config back into ceph.conf, using "ceph config reset" to revert the config db to the original version, and restarting all the mons. After that, I am now able to log in to the dashboard.
I'm still concerned about the config db situation though. Is anyone aware of any open tickets/known issues causing config corruption? If not, I'll create one. Is there some way to complete remove/recreate the config db?
#30 Updated by david sielert almost 5 years ago
+1 I have this same problem, upgrade from mimic , unable to login to dashboard
#31 Updated by david sielert almost 5 years ago
added comment here https://github.com/ceph/ceph/pull/27098 ..
the timezone fix from above breaks the login I was forced to manually revert it in /usr/share/ceph/mgr/dashboard/services/auth.py
#32 Updated by Lenz Grimmer almost 5 years ago
Looks like there is a different bug that causes these 401 errors than what this issue was initially about. See #40051 for a follow-up.
#33 Updated by Lenz Grimmer almost 5 years ago
- Related to Bug #40051: mgr/dashboard: Dashboard login page broken; summary returns 401 added
#34 Updated by Ernesto Puerta almost 3 years ago
- Project changed from mgr to Dashboard
- Category changed from 132 to General