Project

General

Profile

Actions

Bug #20950

closed

key mismatch for mgr after upgrade from jewel to luminous(dev)

Added by Vasu Kulkarni over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

1. Setup cluster using ceph-deploy using jewel release
2. ceph-deploy upgrade to luminous ( dev=luminous to pickup latest dev and this should have the fix for auth create for mgr - https://github.com/ceph/ceph/pull/16395
mentioned in tracker http://tracker.ceph.com/issues/20848 )

[ubuntu@smithi022 cd]$ sudo ceph daemon mon.smithi022 version  
{"version":"12.1.2-469-ge29c598","release":"luminous","release_type":"rc"}

3. run mgr create to create mgr, but in the logs I see following as i create mgr

[ubuntu@smithi022 ~]$ sudo tail -f /var/log/ceph/ceph-mon.smithi022.log
2017-08-08 23:20:55.939986 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e26: no daemons active
2017-08-08 23:21:00.955907 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e27: no daemons active
2017-08-08 23:21:05.989035 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e28: no daemons active
2017-08-08 23:21:10.979647 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e29: no daemons active
2017-08-08 23:21:15.995454 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e30: no daemons active
2017-08-08 23:21:21.011366 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e31: no daemons active
2017-08-08 23:21:26.018917 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e32: no daemons active
2017-08-08 23:21:31.034912 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e33: no daemons active
2017-08-08 23:21:36.050718 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e34: no daemons active
2017-08-08 23:21:41.066701 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e35: no daemons active
2017-08-08 23:21:46.082633 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e36: no daemons active
2017-08-08 23:21:50.530084 7f965ead8700  0 mon.smithi022@0(leader).data_health(5) update_stats avail 93% total 916 GB, used 9432 MB, avail 860 GB
2017-08-08 23:21:51.098663 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e37: no daemons active
2017-08-08 23:21:56.139458 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e38: no daemons active
2017-08-08 23:21:58.234998 7f965c2d3700  0 cephx server client.bootstrap-mgr:  unexpected key: req.key=5dc3ed0bd2defac3 expected_key=32830f9952c9e52
2017-08-08 23:22:01.147021 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e39: no daemons active
2017-08-08 23:22:06.162826 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e40: no daemons active
2017-08-08 23:22:11.200540 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e41: no daemons active
2017-08-08 23:22:16.194801 7f96612dd700  0 log_channel(cluster) log [DBG] : mgrmap e42: no daemons active
^C[ubuntu@smithi022 ~]$  sudo cat /var/lib/ceph/bootstrap-mgr/ceph.keyring
[client.bootstrap-mgr]
        key = AQDKK31ZYcZNKRAANhRFer5Fr0McbuW/QHla1w==
[ubuntu@smithi022 ~]$ sudo ceph auth list
installed auth entries:

osd.0
        key: AQB1LH1ZRBtTIRAAd4PkIVg4FAAwMaJJ3dKaAA==
        caps: [mgr] allow profile osd
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.1
        key: AQCELH1Z2QXFNxAApvrhJD60mxRpN2FhhNrvHg==
        caps: [mgr] allow profile osd
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.2
        key: AQCPLH1ZDt0FOBAA4TZrHGJIlDRl+0BF2Sy+MQ==
        caps: [mgr] allow profile osd
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.3
        key: AQCaLH1ZRhyyOBAAH4XK0+sEUu+lqba+SwEhJg==
        caps: [mgr] allow profile osd
        caps: [mon] allow profile osd
        caps: [osd] allow *
client.admin
        key: AQDIK31ZLGS9GhAA5R2+1pXEWZNR29VYhIjs1A==
        caps: [mds] allow *
        caps: [mgr] allow *
        caps: [mon] allow *
        caps: [osd] allow *
client.bootstrap-mds
        key: AQDJK31ZRWkhDBAAmwRH6NsAw8Bf/VcrfRLIDw==
        caps: [mgr] allow r
        caps: [mon] allow profile bootstrap-mds
client.bootstrap-mgr
        key: AQDaRopZckmUIxAASSuQsbDmggTPrzPIt1NXJg==
        caps: [mon] allow profile bootstrap-mgr

I also have a teuthology test that fails here for 4 node: http://pulpito.ceph.com/vasu-2017-08-08_18:34:23-upgrade-master-distro-basic-vps/
recreated this on single node smithi022 as well, Feel free to login to that node. I believe auth create workaround is not needed here due to latest sha.


Related issues 1 (0 open1 closed)

Copied to mgr - Backport #22034: luminous: key mismatch for mgr after upgrade from jewel to luminous(dev) ResolvedShinobu KinjoActions
Actions

Also available in: Atom PDF