Project

General

Profile

Actions
Copy link

Bug #21122

closed

Kraken to luminous upgrade: Error EINVAL: key for mgr.vpm037 exists but cap mds does not match

Added by Vasu Kulkarni over 6 years ago. Updated about 6 years ago.

Status:
Won't Fix
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Steps:

1. Installed stable version of kraken using ceph-deploy and setup a 4 node cluster
2. To workaround issue http://tracker.ceph.com/issues/20950 , key is pushed to disk as below (and the jewel -> luminous upgrade test works fine )

2017-08-24T21:44:45.156 INFO:teuthology.orchestra.run.vpm037:Running: 'sudo ceph auth get client.bootstrap-mgr | sudo tee /var/lib/ceph/bootstrap-mgr/ceph.keyring'
2017-08-24T21:44:45.457 INFO:teuthology.orchestra.run.vpm037.stderr:exported keyring for client.bootstrap-mgr
2017-08-24T21:44:45.471 INFO:teuthology.orchestra.run.vpm037.stdout:[client.bootstrap-mgr]
2017-08-24T21:44:45.474 INFO:teuthology.orchestra.run.vpm037.stdout:    key = AQDMSJ9ZBSlOMxAA1Fje8hHXRaitROQK8vnTHw==
2017-08-24T21:44:45.477 INFO:teuthology.orchestra.run.vpm037.stdout:    caps mon = "allow profile bootstrap-mgr" 
2017-08-24T21:44:45.481 INFO:teuthology.orchestra.run.vpm001:Running: 'sudo ceph auth get client.bootstrap-mgr | sudo tee /var/lib/ceph/bootstrap-mgr/ceph.keyring'
2017-08-24T21:44:45.806 INFO:teuthology.orchestra.run.vpm001.stderr:exported keyring for client.bootstrap-mgr
2017-08-24T21:44:45.854 INFO:teuthology.orchestra.run.vpm001.stdout:[client.bootstrap-mgr]
2017-08-24T21:44:45.858 INFO:teuthology.orchestra.run.vpm001.stdout:    key = AQDMSJ9ZBSlOMxAA1Fje8hHXRaitROQK8vnTHw==
2017-08-24T21:44:45.860 INFO:teuthology.orchestra.run.vpm001.stdout:    caps mon = "allow profile bootstrap-mgr" 
2017-08-24T21:44:45.872 INFO:teuthology.orchestra.run.vpm125:Running: 'sudo ceph auth get client.bootstrap-mgr | sudo tee /var/lib/ceph/bootstrap-mgr/ceph.keyring'
2017-08-24T21:44:46.252 INFO:teuthology.orchestra.run.vpm125.stderr:exported keyring for client.bootstrap-mgr
2017-08-24T21:44:46.259 INFO:teuthology.orchestra.run.vpm125.stdout:[client.bootstrap-mgr]
2017-08-24T21:44:46.305 INFO:teuthology.orchestra.run.vpm125.stdout:    key = AQDMSJ9ZBSlOMxAA1Fje8hHXRaitROQK8vnTHw==
2017-08-24T21:44:46.308 INFO:teuthology.orchestra.run.vpm125.stdout:    caps mon = "allow profile bootstrap-mgr"

3. ceph-deploy mgr create is then run to create the mgr, at this stage it fails with below error

2017-08-24T21:44:55.428 INFO:teuthology.orchestra.run.vpm037:Running: 'cd /home/ubuntu/cephtest/ceph-deploy && ./ceph-deploy mgr create vpm037 vpm001'
2017-08-24T21:44:55.725 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ubuntu/.cephdeploy.conf
2017-08-24T21:44:55.729 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ] Invoked (1.5.38): ./ceph-deploy mgr create vpm037 vpm001
2017-08-24T21:44:55.731 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ] ceph-deploy options:
2017-08-24T21:44:55.734 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  username                      : None
2017-08-24T21:44:55.736 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  verbose                       : False
2017-08-24T21:44:55.739 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  mgr                           : [('vpm037', 'vpm037'), ('vpm001', 'vpm001')]
2017-08-24T21:44:55.753 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
2017-08-24T21:44:55.756 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  subcommand                    : create
2017-08-24T21:44:55.758 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  quiet                         : False
2017-08-24T21:44:55.760 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f0fd3c31cb0>
2017-08-24T21:44:55.763 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  cluster                       : ceph
2017-08-24T21:44:55.765 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  func                          : <function mgr at 0x7f0fd44b0c08>
2017-08-24T21:44:55.768 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
2017-08-24T21:44:55.785 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  default_release               : False
2017-08-24T21:44:55.787 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts vpm037:vpm037 vpm001:vpm001
2017-08-24T21:44:55.792 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][DEBUG ] connection detected need for sudo
2017-08-24T21:44:55.798 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][DEBUG ] connected to host: vpm037
2017-08-24T21:44:55.804 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][DEBUG ] detect platform information from remote host
2017-08-24T21:44:55.840 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][DEBUG ] detect machine type
2017-08-24T21:44:56.021 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.mgr][INFO  ] Distro info: Ubuntu 16.04 xenial
2017-08-24T21:44:56.033 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.mgr][DEBUG ] remote host will use systemd
2017-08-24T21:44:56.044 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to vpm037
2017-08-24T21:44:56.047 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
2017-08-24T21:44:57.254 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][DEBUG ] create path if it doesn't exist
2017-08-24T21:44:58.196 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.vpm037 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-vpm037/keyring
2017-08-24T21:44:58.349 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][ERROR ] Error EINVAL: key for mgr.vpm037 exists but cap mds does not match

Full logs: http://qa-proxy.ceph.com/teuthology/vasu-2017-08-24_20:59:02-upgrade-master-distro-basic-vps/1560586/teuthology.log

Actions
Copy link

Also available in: Atom PDF