Bug #45296: cephadm: daemon add mon failure: orchestrator._interface.OrchestratorValidationError: ('name %s already in use', 'b') - Orchestrator - Ceph

Actions

Copy link

Bug #45296

closed

cephadm: daemon add mon failure: orchestrator._interface.OrchestratorValidationError: ('name %s already in use', 'b')

Added by Brad Hubbard almost 4 years ago. Updated almost 4 years ago.

Status:

Duplicate

Priority:

High

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

/a/teuthology-2020-04-26_07:01:02-rados-master-distro-basic-smithi/4985864/

2020-04-26T20:45:25.540 INFO:tasks.cephadm:Adding mon.b on smithi124
2020-04-26T20:45:25.540 INFO:teuthology.orchestra.run.smithi124:> true
2020-04-26T20:45:25.661 INFO:teuthology.orchestra.run.smithi124:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph-ci/ceph:28c7ba68ae17612394a47291f42995d4c893a440 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 708e071e-87fe-11ea-a068-001a4aab830c -- ceph orch daemon add mon smithi124:172.21.15.124=b
2020-04-26T20:45:25.990 INFO:ceph.mon.a.smithi162.stdout:Apr 26 20:45:25 smithi162 bash[8955]: cluster 2020-04-26T20:45:24.413976+0000 mgr.y (mgr.14143) 47 : cluster [DBG] pgmap v41: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-04-26T20:45:25.991 INFO:ceph.mon.a.smithi162.stdout:Apr 26 20:45:25 smithi162 bash[8955]: audit 2020-04-26T20:45:24.997072+0000 mon.a (mon.0) 134 : audit [INF] from='client.? 172.21.15.162:0/3819341904' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished
2020-04-26T20:45:25.991 INFO:ceph.mon.a.smithi162.stdout:Apr 26 20:45:25 smithi162 bash[8955]: cluster 2020-04-26T20:45:24.997190+0000 mon.a (mon.0) 135 : cluster [DBG] osdmap e6: 0 total, 0 up, 0 in
2020-04-26T20:45:27.991 INFO:ceph.mon.a.smithi162.stdout:Apr 26 20:45:27 smithi162 bash[8955]: cluster 2020-04-26T20:45:26.414483+0000 mgr.y (mgr.14143) 48 : cluster [DBG] pgmap v43: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-04-26T20:45:27.992 INFO:ceph.mon.a.smithi162.stdout:Apr 26 20:45:27 smithi162 bash[8955]: audit 2020-04-26T20:45:27.892160+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14143 172.21.15.162:0/2473917240' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch
2020-04-26T20:45:27.992 INFO:ceph.mon.a.smithi162.stdout:Apr 26 20:45:27 smithi162 bash[8955]: audit 2020-04-26T20:45:27.893958+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14143 172.21.15.162:0/2473917240' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
2020-04-26T20:45:27.993 INFO:ceph.mon.a.smithi162.stdout:Apr 26 20:45:27 smithi162 bash[8955]: audit 2020-04-26T20:45:27.896020+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14143 172.21.15.162:0/2473917240' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.b", "key": "container_image"}]: dispatch
2020-04-26T20:45:29.000 INFO:ceph.mon.a.smithi162.stdout:Apr 26 20:45:29 smithi162 bash[8955]: audit 2020-04-26T20:45:27.888339+0000 mgr.y (mgr.14143) 49 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi124:172.21.15.124=b", "target": ["mon-mgr", ""]}]: dispatch
2020-04-26T20:45:29.001 INFO:ceph.mon.a.smithi162.stdout:Apr 26 20:45:29 smithi162 bash[8955]: cephadm 2020-04-26T20:45:27.895403+0000 mgr.y (mgr.14143) 50 : cephadm [INF] Deploying daemon mon.b on smithi124
2020-04-26T20:45:29.998 INFO:ceph.mon.a.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[8955]: cluster 2020-04-26T20:45:28.415024+0000 mgr.y (mgr.14143) 51 : cluster [DBG] pgmap v44: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-04-26T20:45:29.998 INFO:ceph.mon.a.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[8955]: audit 2020-04-26T20:45:28.690935+0000 mgr.y (mgr.14143) 52 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi124:172.21.15.124=b", "target": ["mon-mgr", ""]}]: dispatch
2020-04-26T20:45:30.898 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: debug 2020-04-26T20:45:30.900+0000 7faf0dade700 -1 Remote method threw exception: Traceback (most recent call last):
2020-04-26T20:45:30.899 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2699, in add_mon
2020-04-26T20:45:30.899 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: return self._add_daemon('mon', spec, self._create_mon)
2020-04-26T20:45:30.900 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2611, in _add_daemon
2020-04-26T20:45:30.900 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: create_func, config_func)
2020-04-26T20:45:30.901 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2626, in _create_daemons
2020-04-26T20:45:30.901 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: spec.service_id, name)
2020-04-26T20:45:30.901 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1224, in get_unique_name
2020-04-26T20:45:30.902 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: raise orchestrator.OrchestratorValidationError('name %s already in use', forcename)
2020-04-26T20:45:30.902 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: orchestrator._interface.OrchestratorValidationError: ('name %s already in use', 'b')
2020-04-26T20:45:30.903 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: debug 2020-04-26T20:45:30.900+0000 7faf0dade700 -1 mgr handle_command module 'orchestrator' command handler threw exception: Remote method threw exception: Traceback (most recent call last):
2020-04-26T20:45:30.903 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2699, in add_mon
2020-04-26T20:45:30.903 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: return self._add_daemon('mon', spec, self._create_mon)
2020-04-26T20:45:30.904 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2611, in _add_daemon
2020-04-26T20:45:30.905 INFO:teuthology.orchestra.run.smithi124.stderr:Error EINVAL: Traceback (most recent call last):
2020-04-26T20:45:30.905 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 1157, in _handle_command
2020-04-26T20:45:30.906 INFO:teuthology.orchestra.run.smithi124.stderr:    return self.handle_command(inbuf, cmd)
2020-04-26T20:45:30.906 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 110, in handle_command
2020-04-26T20:45:30.906 INFO:teuthology.orchestra.run.smithi124.stderr:    return dispatch[cmd['prefix']].call(self, cmd, inbuf)
2020-04-26T20:45:30.906 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 308, in call
2020-04-26T20:45:30.906 INFO:teuthology.orchestra.run.smithi124.stderr:    return self.func(mgr, **kwargs)
2020-04-26T20:45:30.907 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 72, in <lambda>
2020-04-26T20:45:30.907 INFO:teuthology.orchestra.run.smithi124.stderr:    wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)
2020-04-26T20:45:30.907 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 63, in wrapper
2020-04-26T20:45:30.907 INFO:teuthology.orchestra.run.smithi124.stderr:    return func(*args, **kwargs)
2020-04-26T20:45:30.908 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/orchestrator/module.py", line 658, in _daemon_add_misc
2020-04-26T20:45:30.908 INFO:teuthology.orchestra.run.smithi124.stderr:    completion = self.add_mon(spec)
2020-04-26T20:45:30.908 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1542, in inner
2020-04-26T20:45:30.908 INFO:teuthology.orchestra.run.smithi124.stderr:    completion = self._oremote(method_name, args, kwargs)
2020-04-26T20:45:30.909 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1614, in _oremote
2020-04-26T20:45:30.909 INFO:teuthology.orchestra.run.smithi124.stderr:    return mgr.remote(o, meth, *args, **kwargs)
2020-04-26T20:45:30.909 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 1519, in remote
2020-04-26T20:45:30.909 INFO:teuthology.orchestra.run.smithi124.stderr:    args, kwargs)
2020-04-26T20:45:30.909 INFO:teuthology.orchestra.run.smithi124.stderr:RuntimeError: Remote method threw exception: Traceback (most recent call last):
2020-04-26T20:45:30.910 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/cephadm/module.py", line 2699, in add_mon
2020-04-26T20:45:30.910 INFO:teuthology.orchestra.run.smithi124.stderr:    return self._add_daemon('mon', spec, self._create_mon)
2020-04-26T20:45:30.910 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/cephadm/module.py", line 2611, in _add_daemon
2020-04-26T20:45:30.910 INFO:teuthology.orchestra.run.smithi124.stderr:    create_func, config_func)
2020-04-26T20:45:30.911 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/cephadm/module.py", line 2626, in _create_daemons
2020-04-26T20:45:30.911 INFO:teuthology.orchestra.run.smithi124.stderr:    spec.service_id, name)
2020-04-26T20:45:30.911 INFO:teuthology.orchestra.run.smithi124.stderr:  File "/usr/share/ceph/mgr/cephadm/module.py", line 1224, in get_unique_name
2020-04-26T20:45:30.911 INFO:teuthology.orchestra.run.smithi124.stderr:    raise orchestrator.OrchestratorValidationError('name %s already in use', forcename)
2020-04-26T20:45:30.912 INFO:teuthology.orchestra.run.smithi124.stderr:orchestrator._interface.OrchestratorValidationError: ('name %s already in use', 'b')
2020-04-26T20:45:30.912 INFO:teuthology.orchestra.run.smithi124.stderr:
2020-04-26T20:45:30.912 INFO:teuthology.orchestra.run.smithi124.stderr:
2020-04-26T20:45:30.912 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: create_func, config_func)
2020-04-26T20:45:30.913 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2626, in _create_daemons
2020-04-26T20:45:30.913 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: spec.service_id, name)
2020-04-26T20:45:30.913 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1224, in get_unique_name
2020-04-26T20:45:30.913 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: raise orchestrator.OrchestratorValidationError('name %s already in use', forcename)
2020-04-26T20:45:30.914 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: orchestrator._interface.OrchestratorValidationError: ('name %s already in use', 'b')
2020-04-26T20:45:30.914 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: debug 2020-04-26T20:45:30.903+0000 7faf0dade700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
2020-04-26T20:45:30.914 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/mgr_module.py", line 1157, in _handle_command
2020-04-26T20:45:30.917 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: return self.handle_command(inbuf, cmd)
2020-04-26T20:45:30.917 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 110, in handle_command
2020-04-26T20:45:30.917 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: return dispatch[cmd['prefix']].call(self, cmd, inbuf)
2020-04-26T20:45:30.918 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/mgr_module.py", line 308, in call
2020-04-26T20:45:30.918 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: return self.func(mgr, **kwargs)
2020-04-26T20:45:30.918 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 72, in <lambda>
2020-04-26T20:45:30.918 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)
2020-04-26T20:45:30.919 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 63, in wrapper
2020-04-26T20:45:30.919 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: return func(*args, **kwargs)
2020-04-26T20:45:30.919 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/orchestrator/module.py", line 658, in _daemon_add_misc
2020-04-26T20:45:30.919 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: completion = self.add_mon(spec)
2020-04-26T20:45:30.920 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1542, in inner
2020-04-26T20:45:30.920 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: completion = self._oremote(method_name, args, kwargs)
2020-04-26T20:45:30.920 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1614, in _oremote
2020-04-26T20:45:30.920 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: return mgr.remote(o, meth, *args, **kwargs)
2020-04-26T20:45:30.921 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/mgr_module.py", line 1519, in remote
2020-04-26T20:45:30.921 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: args, kwargs)
2020-04-26T20:45:30.921 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: RuntimeError: Remote method threw exception: Traceback (most recent call last):
2020-04-26T20:45:30.921 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2699, in add_mon
2020-04-26T20:45:30.921 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: return self._add_daemon('mon', spec, self._create_mon)
2020-04-26T20:45:30.922 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2611, in _add_daemon
2020-04-26T20:45:30.922 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: create_func, config_func)
2020-04-26T20:45:30.922 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2626, in _create_daemons
2020-04-26T20:45:30.922 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: spec.service_id, name)
2020-04-26T20:45:30.923 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1224, in get_unique_name
2020-04-26T20:45:30.923 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: raise orchestrator.OrchestratorValidationError('name %s already in use', forcename)
2020-04-26T20:45:30.923 INFO:ceph.mgr.y.smithi162.stdout:Apr 26 20:45:30 smithi162 bash[9201]: orchestrator._interface.OrchestratorValidationError: ('name %s already in use', 'b')

Related issues 1 (0 open — 1 closed)