Actions
Bug #20950
closedkey mismatch for mgr after upgrade from jewel to luminous(dev)
% Done:
0%
Source:
Tags:
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
1. Setup cluster using ceph-deploy using jewel release
2. ceph-deploy upgrade to luminous ( dev=luminous to pickup latest dev and this should have the fix for auth create for mgr - https://github.com/ceph/ceph/pull/16395
mentioned in tracker http://tracker.ceph.com/issues/20848 )
[ubuntu@smithi022 cd]$ sudo ceph daemon mon.smithi022 version {"version":"12.1.2-469-ge29c598","release":"luminous","release_type":"rc"}
3. run mgr create to create mgr, but in the logs I see following as i create mgr
[ubuntu@smithi022 ~]$ sudo tail -f /var/log/ceph/ceph-mon.smithi022.log 2017-08-08 23:20:55.939986 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e26: no daemons active 2017-08-08 23:21:00.955907 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e27: no daemons active 2017-08-08 23:21:05.989035 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e28: no daemons active 2017-08-08 23:21:10.979647 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e29: no daemons active 2017-08-08 23:21:15.995454 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e30: no daemons active 2017-08-08 23:21:21.011366 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e31: no daemons active 2017-08-08 23:21:26.018917 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e32: no daemons active 2017-08-08 23:21:31.034912 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e33: no daemons active 2017-08-08 23:21:36.050718 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e34: no daemons active 2017-08-08 23:21:41.066701 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e35: no daemons active 2017-08-08 23:21:46.082633 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e36: no daemons active 2017-08-08 23:21:50.530084 7f965ead8700 0 mon.smithi022@0(leader).data_health(5) update_stats avail 93% total 916 GB, used 9432 MB, avail 860 GB 2017-08-08 23:21:51.098663 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e37: no daemons active 2017-08-08 23:21:56.139458 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e38: no daemons active 2017-08-08 23:21:58.234998 7f965c2d3700 0 cephx server client.bootstrap-mgr: unexpected key: req.key=5dc3ed0bd2defac3 expected_key=32830f9952c9e52 2017-08-08 23:22:01.147021 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e39: no daemons active 2017-08-08 23:22:06.162826 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e40: no daemons active 2017-08-08 23:22:11.200540 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e41: no daemons active 2017-08-08 23:22:16.194801 7f96612dd700 0 log_channel(cluster) log [DBG] : mgrmap e42: no daemons active ^C[ubuntu@smithi022 ~]$ sudo cat /var/lib/ceph/bootstrap-mgr/ceph.keyring
[client.bootstrap-mgr] key = AQDKK31ZYcZNKRAANhRFer5Fr0McbuW/QHla1w== [ubuntu@smithi022 ~]$ sudo ceph auth list installed auth entries: osd.0 key: AQB1LH1ZRBtTIRAAd4PkIVg4FAAwMaJJ3dKaAA== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * osd.1 key: AQCELH1Z2QXFNxAApvrhJD60mxRpN2FhhNrvHg== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * osd.2 key: AQCPLH1ZDt0FOBAA4TZrHGJIlDRl+0BF2Sy+MQ== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * osd.3 key: AQCaLH1ZRhyyOBAAH4XK0+sEUu+lqba+SwEhJg== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * client.admin key: AQDIK31ZLGS9GhAA5R2+1pXEWZNR29VYhIjs1A== caps: [mds] allow * caps: [mgr] allow * caps: [mon] allow * caps: [osd] allow * client.bootstrap-mds key: AQDJK31ZRWkhDBAAmwRH6NsAw8Bf/VcrfRLIDw== caps: [mgr] allow r caps: [mon] allow profile bootstrap-mds client.bootstrap-mgr key: AQDaRopZckmUIxAASSuQsbDmggTPrzPIt1NXJg== caps: [mon] allow profile bootstrap-mgr
I also have a teuthology test that fails here for 4 node: http://pulpito.ceph.com/vasu-2017-08-08_18:34:23-upgrade-master-distro-basic-vps/
recreated this on single node smithi022 as well, Feel free to login to that node. I believe auth create workaround is not needed here due to latest sha.
Actions