Bug #21122: Kraken to luminous upgrade: Error EINVAL: key for mgr.vpm037 exists but cap mds does not match - mgr - Ceph

Actions

Copy link

Bug #21122

closed

Kraken to luminous upgrade: Error EINVAL: key for mgr.vpm037 exists but cap mds does not match

Added by Vasu Kulkarni over 6 years ago. Updated about 6 years ago.

Status:

Won't Fix

Priority:

High

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Steps:

1. Installed stable version of kraken using ceph-deploy and setup a 4 node cluster
2. To workaround issue http://tracker.ceph.com/issues/20950 , key is pushed to disk as below (and the jewel -> luminous upgrade test works fine )

2017-08-24T21:44:45.156 INFO:teuthology.orchestra.run.vpm037:Running: 'sudo ceph auth get client.bootstrap-mgr | sudo tee /var/lib/ceph/bootstrap-mgr/ceph.keyring'
2017-08-24T21:44:45.457 INFO:teuthology.orchestra.run.vpm037.stderr:exported keyring for client.bootstrap-mgr
2017-08-24T21:44:45.471 INFO:teuthology.orchestra.run.vpm037.stdout:[client.bootstrap-mgr]
2017-08-24T21:44:45.474 INFO:teuthology.orchestra.run.vpm037.stdout:    key = AQDMSJ9ZBSlOMxAA1Fje8hHXRaitROQK8vnTHw==
2017-08-24T21:44:45.477 INFO:teuthology.orchestra.run.vpm037.stdout:    caps mon = "allow profile bootstrap-mgr" 
2017-08-24T21:44:45.481 INFO:teuthology.orchestra.run.vpm001:Running: 'sudo ceph auth get client.bootstrap-mgr | sudo tee /var/lib/ceph/bootstrap-mgr/ceph.keyring'
2017-08-24T21:44:45.806 INFO:teuthology.orchestra.run.vpm001.stderr:exported keyring for client.bootstrap-mgr
2017-08-24T21:44:45.854 INFO:teuthology.orchestra.run.vpm001.stdout:[client.bootstrap-mgr]
2017-08-24T21:44:45.858 INFO:teuthology.orchestra.run.vpm001.stdout:    key = AQDMSJ9ZBSlOMxAA1Fje8hHXRaitROQK8vnTHw==
2017-08-24T21:44:45.860 INFO:teuthology.orchestra.run.vpm001.stdout:    caps mon = "allow profile bootstrap-mgr" 
2017-08-24T21:44:45.872 INFO:teuthology.orchestra.run.vpm125:Running: 'sudo ceph auth get client.bootstrap-mgr | sudo tee /var/lib/ceph/bootstrap-mgr/ceph.keyring'
2017-08-24T21:44:46.252 INFO:teuthology.orchestra.run.vpm125.stderr:exported keyring for client.bootstrap-mgr
2017-08-24T21:44:46.259 INFO:teuthology.orchestra.run.vpm125.stdout:[client.bootstrap-mgr]
2017-08-24T21:44:46.305 INFO:teuthology.orchestra.run.vpm125.stdout:    key = AQDMSJ9ZBSlOMxAA1Fje8hHXRaitROQK8vnTHw==
2017-08-24T21:44:46.308 INFO:teuthology.orchestra.run.vpm125.stdout:    caps mon = "allow profile bootstrap-mgr"

3. ceph-deploy mgr create is then run to create the mgr, at this stage it fails with below error

2017-08-24T21:44:55.428 INFO:teuthology.orchestra.run.vpm037:Running: 'cd /home/ubuntu/cephtest/ceph-deploy && ./ceph-deploy mgr create vpm037 vpm001'
2017-08-24T21:44:55.725 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ubuntu/.cephdeploy.conf
2017-08-24T21:44:55.729 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ] Invoked (1.5.38): ./ceph-deploy mgr create vpm037 vpm001
2017-08-24T21:44:55.731 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ] ceph-deploy options:
2017-08-24T21:44:55.734 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  username                      : None
2017-08-24T21:44:55.736 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  verbose                       : False
2017-08-24T21:44:55.739 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  mgr                           : [('vpm037', 'vpm037'), ('vpm001', 'vpm001')]
2017-08-24T21:44:55.753 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
2017-08-24T21:44:55.756 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  subcommand                    : create
2017-08-24T21:44:55.758 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  quiet                         : False
2017-08-24T21:44:55.760 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f0fd3c31cb0>
2017-08-24T21:44:55.763 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  cluster                       : ceph
2017-08-24T21:44:55.765 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  func                          : <function mgr at 0x7f0fd44b0c08>
2017-08-24T21:44:55.768 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
2017-08-24T21:44:55.785 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.cli][INFO  ]  default_release               : False
2017-08-24T21:44:55.787 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts vpm037:vpm037 vpm001:vpm001
2017-08-24T21:44:55.792 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][DEBUG ] connection detected need for sudo
2017-08-24T21:44:55.798 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][DEBUG ] connected to host: vpm037
2017-08-24T21:44:55.804 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][DEBUG ] detect platform information from remote host
2017-08-24T21:44:55.840 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][DEBUG ] detect machine type
2017-08-24T21:44:56.021 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.mgr][INFO  ] Distro info: Ubuntu 16.04 xenial
2017-08-24T21:44:56.033 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.mgr][DEBUG ] remote host will use systemd
2017-08-24T21:44:56.044 INFO:teuthology.orchestra.run.vpm037.stderr:[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to vpm037
2017-08-24T21:44:56.047 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
2017-08-24T21:44:57.254 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][DEBUG ] create path if it doesn't exist
2017-08-24T21:44:58.196 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.vpm037 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-vpm037/keyring
2017-08-24T21:44:58.349 INFO:teuthology.orchestra.run.vpm037.stderr:[vpm037][ERROR ] Error EINVAL: key for mgr.vpm037 exists but cap mds does not match

Full logs: http://qa-proxy.ceph.com/teuthology/vasu-2017-08-24_20:59:02-upgrade-master-distro-basic-vps/1560586/teuthology.log

Actions

Copy link

Updated by Nicolas Drufin over 6 years ago

I have the same error when I migrate my cluster and create mgr

Actions

Copy link

Updated by John Spray over 6 years ago

Status changed from New to Closed

This kind of looks like the mgr daemon with that name just already existed?

I'm going to close this as the bootstrap-mgr issue has been fixed elsewhere, so I suspect the sequence of actions (including workarounds) that led to this might not exist any more.

Actions

Copy link

Updated by Vasu Kulkarni over 6 years ago

Status changed from Closed to New

Hi John,

The issue from Kraken(Stable) -> Luminous(dev) is still not fixed, I have removed the workaround of recreating the mgr key which was required before but not in the recent dev. The same steps works for Jewel(Stable) -> Luminous(dev) but not for Kraken to Luminous

http://qa-proxy.ceph.com/teuthology/vasu-2017-11-21_20:56:29-upgrade-master-distro-basic-vps/1874418/teuthology.log


2017-11-21T21:26:12.938 INFO:teuthology.orchestra.run.vpm019:Running: 'cd /home/ubuntu/cephtest/ceph-deploy && ./ceph-deploy mgr create vpm019 vpm181'
2017-11-21T21:26:13.179 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ubuntu/.cephdeploy.conf
2017-11-21T21:26:13.180 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ] Invoked (1.5.39): ./ceph-deploy mgr create vpm019 vpm181
2017-11-21T21:26:13.180 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ] ceph-deploy options:
2017-11-21T21:26:13.181 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ]  username                      : None
2017-11-21T21:26:13.181 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ]  verbose                       : False
2017-11-21T21:26:13.181 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ]  mgr                           : [('vpm019', 'vpm019'), ('vpm181', 'vpm181')]
2017-11-21T21:26:13.210 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
2017-11-21T21:26:13.210 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ]  subcommand                    : create
2017-11-21T21:26:13.210 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ]  quiet                         : False
2017-11-21T21:26:13.210 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f87c38dccb0>
2017-11-21T21:26:13.210 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ]  cluster                       : ceph
2017-11-21T21:26:13.210 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ]  func                          : <function mgr at 0x7f87c415cd70>
2017-11-21T21:26:13.210 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
2017-11-21T21:26:13.210 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.cli][INFO  ]  default_release               : False
2017-11-21T21:26:13.211 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts vpm019:vpm019 vpm181:vpm181
2017-11-21T21:26:13.221 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm019][DEBUG ] connection detected need for sudo
2017-11-21T21:26:13.244 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm019][DEBUG ] connected to host: vpm019
2017-11-21T21:26:13.245 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm019][DEBUG ] detect platform information from remote host
2017-11-21T21:26:13.300 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm019][DEBUG ] detect machine type
2017-11-21T21:26:13.300 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.mgr][INFO  ] Distro info: Ubuntu 16.04 xenial
2017-11-21T21:26:13.301 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.mgr][DEBUG ] remote host will use systemd
2017-11-21T21:26:13.301 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to vpm019
2017-11-21T21:26:13.301 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm019][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
2017-11-21T21:26:13.301 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm019][DEBUG ] create path if it doesn't exist
2017-11-21T21:26:13.306 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm019][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.vpm019 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-vpm019/keyring
2017-11-21T21:26:13.622 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm019][ERROR ] Error EINVAL: key for mgr.vpm019 exists but cap mds does not match
2017-11-21T21:26:13.623 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm019][ERROR ] exit code from command was: 22
2017-11-21T21:26:13.623 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.mgr][ERROR ] could not create mgr
2017-11-21T21:26:13.749 INFO:teuthology.orchestra.run.vpm019.stderr:Warning: Permanently added 'vpm181,172.21.2.181' (ECDSA) to the list of known hosts.
2017-11-21T21:26:13.964 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm181][DEBUG ] connection detected need for sudo
2017-11-21T21:26:14.092 INFO:teuthology.orchestra.run.vpm019.stderr:Warning: Permanently added 'vpm181,172.21.2.181' (ECDSA) to the list of known hosts.
2017-11-21T21:26:14.310 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm181][DEBUG ] connected to host: vpm181
2017-11-21T21:26:14.310 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm181][DEBUG ] detect platform information from remote host
2017-11-21T21:26:14.362 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm181][DEBUG ] detect machine type
2017-11-21T21:26:14.372 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.mgr][INFO  ] Distro info: Ubuntu 16.04 xenial
2017-11-21T21:26:14.372 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.mgr][DEBUG ] remote host will use systemd
2017-11-21T21:26:14.372 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to vpm181
2017-11-21T21:26:14.372 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm181][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
2017-11-21T21:26:14.373 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm181][DEBUG ] create path if it doesn't exist
2017-11-21T21:26:14.376 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm181][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.vpm181 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-vpm181/keyring
2017-11-21T21:26:14.695 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm181][ERROR ] Error EINVAL: key for mgr.vpm181 exists but cap mds does not match
2017-11-21T21:26:14.695 INFO:teuthology.orchestra.run.vpm019.stderr:[vpm181][ERROR ] exit code from command was: 22
2017-11-21T21:26:14.696 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy.mgr][ERROR ] could not create mgr
2017-11-21T21:26:14.696 INFO:teuthology.orchestra.run.vpm019.stderr:[ceph_deploy][ERROR ] GenericError: Failed to create 2 MGRs

Actions

Copy link