Project

General

Profile

Bug #38428

mgr/dashboard: dashboard giving 401 unauthorized

Added by Lokesh Pashine 8 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
dashboard/general
Target version:
Start date:
02/22/2019
Due date:
% Done:

0%

Source:
Tags:
Backport:
nautilus
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

Hello Team,

I have set up the ceph dashboard and its allowing me to login but immediately after login it log me out.

From logs I can see below
2019-02-22 13:44:33.919 7fbecd4fe700 0 mgr[dashboard] frontend error (https://192.168.1.142:8443/#/login): Http failure response for https://192.168.1.142:8443/api/summary: 401 Unauthorized

cephaio:/var/log/ceph # netstat -an| grep 8443
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN

cephaio:/usr/lib64/ceph/mgr/dashboard # radosgw-admin user info --uid ceph-admin {
"user_id": "ceph-admin",
"display_name": "ceph admin",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [ {
"user": "ceph-admin",
"access_key": "ANXQI47HUI7T00G1UQFC",
"secret_key": "EShhS8EcAFeEvBXDabzUV9JCaAJjvB93CsQD9Nqg"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"system": "true",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}

cephaio:/usr/lib64/ceph/mgr/dashboard # sudo ceph dashboard get-rgw-api-user-id
ceph-admin

cephaio:/usr/lib64/ceph/mgr/dashboard # ceph dashboard get-rgw-api-access-key
ANXQI47HUI7T00G1UQFC

cephaio:/usr/lib64/ceph/mgr/dashboard # ceph dashboard get-rgw-api-secret-key
EShhS8EcAFeEvBXDabzUV9JCaAJjvB93CsQD9Nqg

cephaio:/usr/lib64/ceph/mgr/dashboard # ceph dashboard get-rgw-api-ssl-verify
False

cephaio:/var/log/ceph # ceph dashboard ac-user-show ceph-admin {"username": "ceph-admin", "password": "$2b$12$62HYc/RwUN4T/Sdp.z5SgexAlJghnRvAS8htab9aXO.pGsgh4CTNi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1550721565}
cephaio:/var/log/ceph #

cephaio:/var/log/ceph # ceph dashboard get-rgw-api-access-key
ANXQI47HUI7T00G1UQFC

cephaio:/var/log/ceph # ceph dashboard get-rgw-api-secret-key
EShhS8EcAFeEvBXDabzUV9JCaAJjvB93CsQD9Nqg

cephaio:/var/log/ceph # ceph dashboard get-rgw-api-host

cephaio

cephaio:/var/log/ceph # ceph dashboard get-rgw-api-port
8003

cephaio:/var/log/ceph # ceph mgr services {
"dashboard": "https://0.0.0.0:8443/",
"restful": "https://cephaio:8003/"
}

cephaio:/var/log/ceph # ceph --version
ceph version 14.0.1-3975-g14d8f3cab5 (14d8f3cab592c4b3ac85000ccd8fc9fe03398428) nautilus (dev)
cephaio:/var/log/ceph #

cluster:
id: bfdfbe86-3ee9-417e-a3bd-0818694a18d0
health: HEALTH_WARN
application not enabled on 2 pool(s)
services:
mon: 1 daemons, quorum cephaio (age 2h)
mgr: cephaio(active, since 2m)
osd: 1 osds: 1 up (since 2h), 1 in (since 27h)
rgw: 1 daemon active
data:
pools: 6 pools, 34 pgs
objects: 191 objects, 5.2 KiB
usage: 45 MiB used, 979 MiB / 1 GiB avail
pgs: 34 active+clean

cephaio:/var/log/ceph #

Auswahl_002.png View (66 KB) Volker Theile, 02/28/2019 11:27 AM

ceph-dashboard-response.jpg View (214 KB) Lokesh Pashine, 02/28/2019 03:43 PM

screenshot.png View (15.1 KB) Darrell Enns, 05/09/2019 10:02 PM


Related issues

Related to mgr - Bug #39300: mgr/dashboard: Can't login with a bigger time difference between user and server or make auth token work with UTC times only Resolved 04/15/2019
Related to mgr - Bug #40051: mgr/dashboard: Dashboard login page broken; summary returns 401 Duplicate 05/28/2019
Duplicated by mgr - Bug #38600: mgr/dashboard: dashboard giving 401 unauthorized. Duplicate 03/06/2019
Copied to mgr - Backport #38871: nautilus: mgr/dashboard: dashboard giving 401 unauthorized Resolved

History

#1 Updated by Alfonso MH 8 months ago

Tested Dashboard login with Nautilus 14.1.0 (2019-02-22, https://download.ceph.com/rpm-nautilus/el7/x86_64/):

Issue not reproducible (it works as expected).

Can you test this version?

#2 Updated by Lenz Grimmer 8 months ago

  • Status changed from New to Need More Info

Can you test if the provided RGW credentials actually work? I wonder if the dashboard reacts to the 401 error that is returned by RGW by any chance?

#3 Updated by Volker Theile 8 months ago

The RGW REST API controller had a problem that might cause this issue. If the RGW Admin OPS API returned an 401 it was directly forwarded to the UI which forces a logout. This has been fixed ages ago with https://github.com/ceph/ceph/commit/ee5c9ad03c9d301d01716e56710fe24a67f0f869. Nowadays all errors raised by the RGW Admin OPS API are returned as 500 (see https://github.com/ceph/ceph/blob/master/src/pybind/mgr/dashboard/controllers/rgw.py#L103).

2019-02-22 13:44:33.919 7fbecd4fe700 0 mgr[dashboard] frontend error (https://192.168.1.142:8443/#/login): Http failure response for https://192.168.1.142:8443/api/summary: 401 Unauthorized

To me it looks like the /api/summary API call is responsible for this. But the controller code itself does not throw exceptions, so it might happen somewhere on the way when the REST call is processed.

#4 Updated by Volker Theile 8 months ago

  • Subject changed from ceph-dashboard giving 401 unauthorized to mgr/dashboard: dashboard giving 401 unauthorized

#5 Updated by Volker Theile 8 months ago

Can you provide us a screenshot like this. Maybe this can help us to identify the problem.

#6 Updated by Lenz Grimmer 8 months ago

Hi Lokesh, something in your report looks incorrect:

Lokesh Pashine wrote:

cephaio:/var/log/ceph # ceph dashboard get-rgw-api-host

cephaio

cephaio:/var/log/ceph # ceph dashboard get-rgw-api-port
8003

cephaio:/var/log/ceph # ceph mgr services {
"dashboard": "https://0.0.0.0:8443/",
"restful": "https://cephaio:8003/"
}

This looks to me as if you've configured the dashboard to talk to the restful module on https://cephaio:8003/ is if it was the RGW API endpoint. Note that the restful module is not needed by the dashboard. If you want to manage RGW via the dashboard, make sure to use the correct hostname/TCP port to talk to the RGW admin ops API.

#7 Updated by Lokesh Pashine 8 months ago

Lenz Grimmer wrote:

Hi Lokesh, something in your report looks incorrect:

Lokesh Pashine wrote:

cephaio:/var/log/ceph # ceph dashboard get-rgw-api-host

cephaio

cephaio:/var/log/ceph # ceph dashboard get-rgw-api-port
8003

cephaio:/var/log/ceph # ceph mgr services {
"dashboard": "https://0.0.0.0:8443/",
"restful": "https://cephaio:8003/"
}

This looks to me as if you've configured the dashboard to talk to the restful module on https://cephaio:8003/ is if it was the RGW API endpoint. Note that the restful module is not needed by the dashboard. If you want to manage RGW via the dashboard, make sure to use the correct hostname/TCP port to talk to the RGW admin ops API.

==============

Hi Lenz,

I also noticed the configuration missmatch and corrected it already but issue is still the same

cephaio:~ # ceph dashboard get-rgw-api-access-key
35S5T7IHEXPDRGZ012A6
cephaio:~ # ceph dashboard get-rgw-api-secret-key
mKkouV45flvszN5w4z7BAAL6EHO5375HhVQMyWj4
cephaio:~ # ceph dashboard get-rgw-api-host
cephaio
cephaio:~ # ceph dashboard get-rgw-api-port
7480
cephaio:~ # ceph dashboard get-rgw-api-ssl-verify
False
cephaio:~ # ceph dashboard get-rgw-api-user-id
radosadm
cephaio:~ # lsof -i:7480
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
radosgw 1559 ceph 40u IPv4 28900 0t0 TCP *:7480 (LISTEN)
cephaio:~ #

Please suggest what is wrong here and if I have to try upgrading the dashboard package?

#8 Updated by Lokesh Pashine 8 months ago

Volker Theile wrote:

Can you provide us a screenshot like this. Maybe this can help us to identify the problem.

============

Hi Volker,

As requested attached is the response code screenshot

#9 Updated by Lokesh Pashine 8 months ago

! ceph-dashboard-response.jpg!

#10 Updated by Lokesh Pashine 8 months ago

!"C:\Users\I355971\Desktop\Temp\ceph-dashboard-response.jpg!

#11 Updated by Lokesh Pashine 8 months ago

Alfonso MH wrote:

Tested Dashboard login with Nautilus 14.1.0 (2019-02-22, https://download.ceph.com/rpm-nautilus/el7/x86_64/):

Issue not reproducible (it works as expected).

Can you test this version?

=========
Hello Alfonso,

DO I need to upgrade all the components or just by upgrading ceph-mgr will do?

#12 Updated by Nathan Cutler 8 months ago

DO I need to upgrade all the components or just by upgrading ceph-mgr will do?

As a general rule, always upgrade all components (RPMs, DEBs). As a project, Ceph is rather monolithic and there are many interdependencies between the components.

In this specific case, it's possible the fix is contained within the ceph-mgr-dashboard package, in which case only that package would need to be upgraded.

#13 Updated by echo wen 8 months ago

It looks like i have not only eth, but can i fix it

#14 Updated by echo wen 8 months ago

迁 文 wrote:

It looks like i have not only eth, but how can i fix it

#15 Updated by echo wen 8 months ago

迁 文 wrote:

#16 Updated by Volker Theile 8 months ago

  • Duplicated by Bug #38600: mgr/dashboard: dashboard giving 401 unauthorized. added

#17 Updated by 一帆 师 7 months ago

Lokesh Pashine wrote:

Hello Team,

I have set up the ceph dashboard and its allowing me to login but immediately after login it log me out.

From logs I can see below
2019-02-22 13:44:33.919 7fbecd4fe700 0 mgr[dashboard] frontend error (https://192.168.1.142:8443/#/login): Http failure response for https://192.168.1.142:8443/api/summary: 401 Unauthorized

cephaio:/var/log/ceph # netstat -an| grep 8443
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN

cephaio:/usr/lib64/ceph/mgr/dashboard # radosgw-admin user info --uid ceph-admin {
"user_id": "ceph-admin",
"display_name": "ceph admin",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [ {
"user": "ceph-admin",
"access_key": "ANXQI47HUI7T00G1UQFC",
"secret_key": "EShhS8EcAFeEvBXDabzUV9JCaAJjvB93CsQD9Nqg"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"system": "true",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}

cephaio:/usr/lib64/ceph/mgr/dashboard # sudo ceph dashboard get-rgw-api-user-id
ceph-admin

cephaio:/usr/lib64/ceph/mgr/dashboard # ceph dashboard get-rgw-api-access-key
ANXQI47HUI7T00G1UQFC

cephaio:/usr/lib64/ceph/mgr/dashboard # ceph dashboard get-rgw-api-secret-key
EShhS8EcAFeEvBXDabzUV9JCaAJjvB93CsQD9Nqg

cephaio:/usr/lib64/ceph/mgr/dashboard # ceph dashboard get-rgw-api-ssl-verify
False

cephaio:/var/log/ceph # ceph dashboard ac-user-show ceph-admin {"username": "ceph-admin", "password": "$2b$12$62HYc/RwUN4T/Sdp.z5SgexAlJghnRvAS8htab9aXO.pGsgh4CTNi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1550721565}
cephaio:/var/log/ceph #

cephaio:/var/log/ceph # ceph dashboard get-rgw-api-access-key
ANXQI47HUI7T00G1UQFC

cephaio:/var/log/ceph # ceph dashboard get-rgw-api-secret-key
EShhS8EcAFeEvBXDabzUV9JCaAJjvB93CsQD9Nqg

cephaio:/var/log/ceph # ceph dashboard get-rgw-api-host

cephaio

cephaio:/var/log/ceph # ceph dashboard get-rgw-api-port
8003

cephaio:/var/log/ceph # ceph mgr services {
"dashboard": "https://0.0.0.0:8443/",
"restful": "https://cephaio:8003/"
}

cephaio:/var/log/ceph # ceph --version
ceph version 14.0.1-3975-g14d8f3cab5 (14d8f3cab592c4b3ac85000ccd8fc9fe03398428) nautilus (dev)
cephaio:/var/log/ceph #

cluster:
id: bfdfbe86-3ee9-417e-a3bd-0818694a18d0
health: HEALTH_WARN
application not enabled on 2 pool(s)

services:
mon: 1 daemons, quorum cephaio (age 2h)
mgr: cephaio(active, since 2m)
osd: 1 osds: 1 up (since 2h), 1 in (since 27h)
rgw: 1 daemon active

data:
pools: 6 pools, 34 pgs
objects: 191 objects, 5.2 KiB
usage: 45 MiB used, 979 MiB / 1 GiB avail
pgs: 34 active+clean

cephaio:/var/log/ceph #

I resolved this bug by change my timezone from CST to EDT.

#18 Updated by Lenz Grimmer 7 months ago

一帆 师 wrote:

I resolved this bug by change my timezone from CST to EDT.

Can you please share where you changed the time zone?

#19 Updated by Fareez Azhar 7 months ago

Im on 14.2.0 changing timezone.. issue still there.. 401 api/summary

#20 Updated by 一帆 师 7 months ago

Fareez Azhar wrote:

Im on 14.2.0 changing timezone.. issue still there.. 401 api/summary

try more time zone, my computer's time zone is Asia/Shanghai, and my server's time need to be EDT.

so if your computer's time zone is not CST, maybe another time zone of your server different of EDT will work.

#21 Updated by Lenz Grimmer 7 months ago

  • Status changed from Need More Info to In Progress
  • Target version set to v15.0.0
  • Backport set to nautilus
  • Pull request ID set to 27098

A patch for this has been proposed: https://github.com/ceph/ceph/pull/27098

#22 Updated by 一帆 师 7 months ago

Lenz Grimmer wrote:

A patch for this has been proposed: https://github.com/ceph/ceph/pull/27098

thks,I changed the code as "https://github.com/ceph/ceph/pull/27098",I works now.

#23 Updated by Lenz Grimmer 7 months ago

  • Status changed from In Progress to Pending Backport
  • Affected Versions v14.0.0, v14.2.0 added

#24 Updated by Nathan Cutler 7 months ago

  • Copied to Backport #38871: nautilus: mgr/dashboard: dashboard giving 401 unauthorized added

#25 Updated by Lenz Grimmer 7 months ago

  • Status changed from Pending Backport to Resolved

#26 Updated by Lenz Grimmer 6 months ago

  • Related to Bug #39300: mgr/dashboard: Can't login with a bigger time difference between user and server or make auth token work with UTC times only added

#27 Updated by Darrell Enns 5 months ago

I am seeing this exact same behaviour on 14.2.1 (which includes the patch). Client and server are in the same time zone and clocks are in sync.

# ceph version
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
  1. cat /usr/share/ceph/mgr/dashboard/services/auth.py |head -n48|tail -n5
            ttl = mgr.get_module_option('jwt_token_ttl', cls.JWT_TOKEN_TTL)
            ttl = int(ttl)
            now = int(time.time())
            payload = {
                'iss': 'ceph-dashboard',
    

#28 Updated by Lenz Grimmer 5 months ago

Can you please try the following:

  1. Manually apply the change posted at https://github.com/ceph/ceph/pull/27942/files to the file /usr/share/ceph/mgr/dashboard/services/access_control.py on all nodes running the dashboard
  2. Restart the dashboard using ceph mgr module disable dashboard; ceph mgr module enable dashboard
  3. Update the timestamp on admin's password: ceph dashboard ac-user-set-password admin <same_password_as_before>
  4. Try to login again

#29 Updated by Darrell Enns 5 months ago

That patch is already applied in 14.2.1, and I have reset passwords many times already.

It looks like in my case it was caused by some sort of corruption of the config database. I began to suspect it when I noticed some odd behaviour when trying to set config options. I tried to change an option (mon_osd_down_out_subtree_limit) and it made a duplicate setting instead. I could remove the duplicate, but not the original. I ended up moving my config back into ceph.conf, using "ceph config reset" to revert the config db to the original version, and restarting all the mons. After that, I am now able to log in to the dashboard.

I'm still concerned about the config db situation though. Is anyone aware of any open tickets/known issues causing config corruption? If not, I'll create one. Is there some way to complete remove/recreate the config db?

#30 Updated by david sielert 5 months ago

+1 I have this same problem, upgrade from mimic , unable to login to dashboard

#31 Updated by david sielert 5 months ago

added comment here https://github.com/ceph/ceph/pull/27098 ..
the timezone fix from above breaks the login I was forced to manually revert it in /usr/share/ceph/mgr/dashboard/services/auth.py

#32 Updated by Lenz Grimmer 5 months ago

Looks like there is a different bug that causes these 401 errors than what this issue was initially about. See #40051 for a follow-up.

#33 Updated by Lenz Grimmer 5 months ago

  • Related to Bug #40051: mgr/dashboard: Dashboard login page broken; summary returns 401 added

Also available in: Atom PDF