Actions
Bug #45465
closedcephadm: `ceph orch restart osd` has the potential to break your cluster
Added by Sebastian Wagner almost 4 years ago. Updated about 3 years ago.
Status:
Resolved
Priority:
High
Assignee:
-
Category:
cephadm
Target version:
-
% Done:
0%
Source:
Tags:
ux
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Description
Multiple bugs here:
- the cephadm implementation doesn't check anything. (ceph osd ok-to-stop...., HEALTH_ERR)
- we're doing O(hosts) networks calls, rendering cephadm completely unresponsive (even ceph orch ps will not respond)
- no parallel vs sequential. parallel might be dangerous. sequential is safe.
- --force if HEALTH_ERR. cause users might actually need to restart something in order to recover from it.
Updated by Sebastian Wagner almost 4 years ago
And we need to call
cephadm.upgrade.CephadmUpgrade#_wait_for_ok_to_stop
Updated by Sebastian Wagner almost 4 years ago
Also we have to make this an asynchronous operation. See https://docs.ceph.com/docs/master/dev/cephadm/#note-regarding-network-calls-from-cli-handlers
Right now, this command has the potential to make cephadm completely unresponsive
Updated by Sebastian Wagner almost 4 years ago
- Related to Bug #46813: `ceph orch * --refresh` is broken added
Updated by Sebastian Wagner over 3 years ago
depends on https://github.com/ceph/ceph/pull/36753
Updated by Sebastian Wagner over 3 years ago
2020-08-27T14:07:08.067376+0000 mgr.vm-00.atmtmv [INF] Redeploy service mgr
2020-08-27T14:07:08.076180+0000 mgr.vm-00.atmtmv [INF] Failing over to other MGR
2020-08-27T14:07:08.085401+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'auth get' -> 0 in 0.006s
2020-08-27T14:07:08.087295+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.002s
2020-08-27T14:07:08.087497+0000 mgr.vm-00.atmtmv [INF] Deploying daemon mgr.vm-02.qckgdm on vm-02
2020-08-27T14:07:08.087793+0000 mgr.vm-00.atmtmv [DBG] Have connection to vm-02
2020-08-27T14:07:08.089249+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'auth get' -> 0 in 0.010s
2020-08-27T14:07:08.091176+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config get' -> 0 in 0.001s
2020-08-27T14:07:08.091394+0000 mgr.vm-00.atmtmv [DBG] mgr.vm-02.qckgdm container image docker.io/amk3798/ceph:async
2020-08-27T14:07:08.091531+0000 mgr.vm-00.atmtmv [DBG] args: --image docker.io/amk3798/ceph:async deploy --fsid 7e4b7de8-e869-11ea-85bb-525400a8a9dd --name mgr.vm-02.qckgdm --config-json -
2020-08-27T14:07:08.091716+0000 mgr.vm-00.atmtmv [DBG] stdin: {"config": "# minimal ceph.conf for 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n[global]\n\tfsid = 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n\tmon_host = [v2:192.168.122.162:3300/0,v1:192.168.122.162:6789/0] [v2:192.168.122.93:3300/0,v1:192.168.122.93:6789/0] [v2:192.168.122.80:3300/0,v1:192.168.122.80:6789/0]\n", "keyring": "[mgr.vm-02.qckgdm]\n\tkey = AQAZt0dfWeLvFRAARLBl3YW1WiDo1yrgGKMnmQ==\n\tcaps mds = \"allow *\"\n\tcaps mon = \"profile mgr\"\n\tcaps osd = \"allow *\"\n"}
2020-08-27T14:07:08.097401+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.008s
2020-08-27T14:07:08.097597+0000 mgr.vm-00.atmtmv [INF] Deploying daemon mgr.vm-01.arttvi on vm-01
2020-08-27T14:07:08.098001+0000 mgr.vm-00.atmtmv [DBG] Have connection to vm-01
2020-08-27T14:07:08.100103+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config get' -> 0 in 0.002s
2020-08-27T14:07:08.100262+0000 mgr.vm-00.atmtmv [DBG] mgr.vm-01.arttvi container image docker.io/amk3798/ceph:async
2020-08-27T14:07:08.100381+0000 mgr.vm-00.atmtmv [DBG] args: --image docker.io/amk3798/ceph:async deploy --fsid 7e4b7de8-e869-11ea-85bb-525400a8a9dd --name mgr.vm-01.arttvi --config-json -
2020-08-27T14:07:08.100486+0000 mgr.vm-00.atmtmv [DBG] stdin: {"config": "# minimal ceph.conf for 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n[global]\n\tfsid = 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n\tmon_host = [v2:192.168.122.162:3300/0,v1:192.168.122.162:6789/0] [v2:192.168.122.93:3300/0,v1:192.168.122.93:6789/0] [v2:192.168.122.80:3300/0,v1:192.168.122.80:6789/0]\n", "keyring": "[mgr.vm-01.arttvi]\n\tkey = AQBuu0dfpjP4EBAAL4tpDJ8crZD7TxvY3MMywg==\n\tcaps mds = \"allow *\"\n\tcaps mon = \"profile mgr\"\n\tcaps osd = \"allow *\"\n"}
2020-08-27T14:07:13.401287+0000 mgr.vm-01.arttvi [DBG] setting log level based on debug_mgr: WARNING (1/5)
2020-08-27T14:07:13.405330+0000 mgr.vm-01.arttvi [DBG] mgr option ssh_config_file = None
2020-08-27T14:07:13.405472+0000 mgr.vm-01.arttvi [DBG] mgr option device_cache_timeout = 1800
2020-08-27T14:07:13.405608+0000 mgr.vm-01.arttvi [DBG] mgr option daemon_cache_timeout = 600
2020-08-27T14:07:13.405873+0000 mgr.vm-01.arttvi [DBG] mgr option host_check_interval = 600
2020-08-27T14:07:13.406067+0000 mgr.vm-01.arttvi [DBG] mgr option mode = root
2020-08-27T14:07:13.408608+0000 mgr.vm-01.arttvi [DBG] mgr option container_image_base = docker.io/ceph/ceph
2020-08-27T14:07:13.408769+0000 mgr.vm-01.arttvi [DBG] mgr option container_image_prometheus = prom/prometheus:v2.18.1
2020-08-27T14:07:13.408925+0000 mgr.vm-01.arttvi [DBG] mgr option container_image_grafana = ceph/ceph-grafana:6.6.2
2020-08-27T14:07:13.409069+0000 mgr.vm-01.arttvi [DBG] mgr option container_image_alertmanager = prom/alertmanager:v0.20.0
2020-08-27T14:07:13.409212+0000 mgr.vm-01.arttvi [DBG] mgr option container_image_node_exporter = prom/node-exporter:v0.18.1
2020-08-27T14:07:13.409355+0000 mgr.vm-01.arttvi [DBG] mgr option warn_on_stray_hosts = True
2020-08-27T14:07:13.409486+0000 mgr.vm-01.arttvi [DBG] mgr option warn_on_stray_daemons = True
2020-08-27T14:07:13.409611+0000 mgr.vm-01.arttvi [DBG] mgr option warn_on_failed_host_check = True
2020-08-27T14:07:13.409767+0000 mgr.vm-01.arttvi [DBG] mgr option log_to_cluster = True
2020-08-27T14:07:13.409896+0000 mgr.vm-01.arttvi [DBG] mgr option allow_ptrace = False
2020-08-27T14:07:13.410038+0000 mgr.vm-01.arttvi [DBG] mgr option prometheus_alerts_path = /etc/prometheus/ceph/ceph_default_alerts.yml
2020-08-27T14:07:13.410171+0000 mgr.vm-01.arttvi [DBG] mgr option migration_current = 2
2020-08-27T14:07:13.410296+0000 mgr.vm-01.arttvi [DBG] mgr option config_dashboard = True
2020-08-27T14:07:13.410421+0000 mgr.vm-01.arttvi [DBG] mgr option manage_etc_ceph_ceph_conf = False
2020-08-27T14:07:13.410547+0000 mgr.vm-01.arttvi [DBG] mgr option registry_url = None
2020-08-27T14:07:13.410696+0000 mgr.vm-01.arttvi [DBG] mgr option registry_username = None
2020-08-27T14:07:13.410845+0000 mgr.vm-01.arttvi [DBG] mgr option registry_password = None
2020-08-27T14:07:13.410984+0000 mgr.vm-01.arttvi [DBG] mgr option log_level =
2020-08-27T14:07:13.411109+0000 mgr.vm-01.arttvi [DBG] mgr option log_to_file = False
2020-08-27T14:07:13.411240+0000 mgr.vm-01.arttvi [DBG] mgr option log_to_cluster_level = debug
2020-08-27T14:07:13.417060+0000 mgr.vm-01.arttvi [DBG] Loaded inventory {'vm-00': {'hostname': 'vm-00', 'addr': 'vm-00', 'labels': [], 'status': ''}, 'vm-01': {'hostname': 'vm-01', 'addr': 'vm-01', 'labels': [], 'status': ''}, 'vm-02': {'hostname': 'vm-02', 'addr': 'vm-02', 'labels': [], 'status': ''}}
2020-08-27T14:07:13.420013+0000 mgr.vm-01.arttvi [DBG] HostCache.load: host vm-00 has 5 daemons, 3 devices, 3 networks
2020-08-27T14:07:13.421079+0000 mgr.vm-01.arttvi [DBG] HostCache.load: host vm-01 has 6 daemons, 3 devices, 3 networks
2020-08-27T14:07:13.421919+0000 mgr.vm-01.arttvi [DBG] HostCache.load: host vm-02 has 4 daemons, 3 devices, 3 networks
2020-08-27T14:07:13.422405+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for alertmanager
2020-08-27T14:07:13.422692+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for crash
2020-08-27T14:07:13.422963+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for grafana
2020-08-27T14:07:13.423217+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for mgr
2020-08-27T14:07:13.423468+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for mon
2020-08-27T14:07:13.423738+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for node-exporter
2020-08-27T14:07:13.424019+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for prometheus
2020-08-27T14:07:13.601579+0000 mgr.vm-01.arttvi [DBG] serve starting
2020-08-27T14:07:13.602966+0000 mgr.vm-01.arttvi [DBG] refreshing hosts and daemons
2020-08-27T14:07:13.603386+0000 mgr.vm-01.arttvi [DBG] refreshing vm-00 daemons
2020-08-27T14:07:13.603556+0000 mgr.vm-01.arttvi [DBG] Opening connection to root@vm-00 with ssh options '-F /tmp/cephadm-conf-fxilxrpq -i /tmp/cephadm-identity-k2p957q4'
2020-08-27T14:07:13.604083+0000 mgr.vm-01.arttvi [DBG] refreshing vm-01 daemons
2020-08-27T14:07:13.604228+0000 mgr.vm-01.arttvi [DBG] Opening connection to root@vm-01 with ssh options '-F /tmp/cephadm-conf-fxilxrpq -i /tmp/cephadm-identity-k2p957q4'
2020-08-27T14:07:13.606838+0000 mgr.vm-01.arttvi [DBG] refreshing vm-02 daemons
2020-08-27T14:07:13.607020+0000 mgr.vm-01.arttvi [DBG] Opening connection to root@vm-02 with ssh options '-F /tmp/cephadm-conf-fxilxrpq -i /tmp/cephadm-identity-k2p957q4'
2020-08-27T14:07:14.082284+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config get' -> 0 in 0.004s
2020-08-27T14:07:14.082390+0000 mgr.vm-01.arttvi [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:14.082465+0000 mgr.vm-01.arttvi [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:14.083680+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config get' -> 0 in 0.008s
2020-08-27T14:07:14.083800+0000 mgr.vm-01.arttvi [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:14.083890+0000 mgr.vm-01.arttvi [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:14.178552+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config get' -> 0 in 0.036s
2020-08-27T14:07:14.178759+0000 mgr.vm-01.arttvi [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:14.178878+0000 mgr.vm-01.arttvi [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:14.498854+0000 mgr.vm-01.arttvi [DBG] process: completions=<CephadmCompletion>[
<lambda>(...),
]
2020-08-27T14:07:14.499071+0000 mgr.vm-01.arttvi [INF] Redeploy service mgr
2020-08-27T14:07:14.510561+0000 mgr.vm-01.arttvi [INF] Failing over to other MGR
2020-08-27T14:07:14.565848+0000 mgr.vm-01.arttvi [DBG] mon_command: 'auth get' -> 0 in 0.065s
2020-08-27T14:07:14.566190+0000 mgr.vm-01.arttvi [DBG] mon_command: 'auth get' -> 0 in 0.055s
2020-08-27T14:07:14.568530+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.002s
2020-08-27T14:07:14.568770+0000 mgr.vm-01.arttvi [INF] Deploying daemon mgr.vm-00.atmtmv on vm-00
2020-08-27T14:07:14.569052+0000 mgr.vm-01.arttvi [DBG] Have connection to vm-00
2020-08-27T14:07:14.569446+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.003s
2020-08-27T14:07:14.569589+0000 mgr.vm-01.arttvi [INF] Deploying daemon mgr.vm-02.qckgdm on vm-02
2020-08-27T14:07:14.569796+0000 mgr.vm-01.arttvi [DBG] Have connection to vm-02
2020-08-27T14:07:14.572406+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config get' -> 0 in 0.002s
2020-08-27T14:07:14.572566+0000 mgr.vm-01.arttvi [DBG] mgr.vm-02.qckgdm container image docker.io/amk3798/ceph:async
2020-08-27T14:07:14.572741+0000 mgr.vm-01.arttvi [DBG] args: --image docker.io/amk3798/ceph:async deploy --fsid 7e4b7de8-e869-11ea-85bb-525400a8a9dd --name mgr.vm-02.qckgdm --config-json -
2020-08-27T14:07:14.572853+0000 mgr.vm-01.arttvi [DBG] stdin: {"config": "# minimal ceph.conf for 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n[global]\n\tfsid = 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n\tmon_host = [v2:192.168.122.162:3300/0,v1:192.168.122.162:6789/0] [v2:192.168.122.93:3300/0,v1:192.168.122.93:6789/0] [v2:192.168.122.80:3300/0,v1:192.168.122.80:6789/0]\n", "keyring": "[mgr.vm-02.qckgdm]\n\tkey = AQAZt0dfWeLvFRAARLBl3YW1WiDo1yrgGKMnmQ==\n\tcaps mds = \"allow *\"\n\tcaps mon = \"profile mgr\"\n\tcaps osd = \"allow *\"\n"}
2020-08-27T14:07:14.575564+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config get' -> 0 in 0.005s
2020-08-27T14:07:14.575771+0000 mgr.vm-01.arttvi [DBG] mgr.vm-00.atmtmv container image docker.io/amk3798/ceph:async
2020-08-27T14:07:14.575898+0000 mgr.vm-01.arttvi [DBG] args: --image docker.io/amk3798/ceph:async deploy --fsid 7e4b7de8-e869-11ea-85bb-525400a8a9dd --name mgr.vm-00.atmtmv --config-json -
2020-08-27T14:07:14.576037+0000 mgr.vm-01.arttvi [DBG] stdin: {"config": "# minimal ceph.conf for 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n[global]\n\tfsid = 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n\tmon_host = [v2:192.168.122.162:3300/0,v1:192.168.122.162:6789/0] [v2:192.168.122.93:3300/0,v1:192.168.122.93:6789/0] [v2:192.168.122.80:3300/0,v1:192.168.122.80:6789/0]\n", "keyring": "[mgr.vm-00.atmtmv]\n\tkey = AQDitUdf2gKWMhAAiqPAkPwfDuwfMgtWXeAnjw==\n\tcaps mds = \"allow *\"\n\tcaps mon = \"profile mgr\"\n\tcaps osd = \"allow *\"\n"}
2020-08-27T14:07:25.527152+0000 mgr.vm-00.atmtmv [DBG] setting log level based on debug_mgr: WARNING (1/5)
2020-08-27T14:07:25.535205+0000 mgr.vm-00.atmtmv [DBG] mgr option ssh_config_file = None
2020-08-27T14:07:25.535291+0000 mgr.vm-00.atmtmv [DBG] mgr option device_cache_timeout = 1800
2020-08-27T14:07:25.535357+0000 mgr.vm-00.atmtmv [DBG] mgr option daemon_cache_timeout = 600
2020-08-27T14:07:25.535414+0000 mgr.vm-00.atmtmv [DBG] mgr option host_check_interval = 600
2020-08-27T14:07:25.535475+0000 mgr.vm-00.atmtmv [DBG] mgr option mode = root
2020-08-27T14:07:25.535531+0000 mgr.vm-00.atmtmv [DBG] mgr option container_image_base = docker.io/ceph/ceph
2020-08-27T14:07:25.535585+0000 mgr.vm-00.atmtmv [DBG] mgr option container_image_prometheus = prom/prometheus:v2.18.1
2020-08-27T14:07:25.535659+0000 mgr.vm-00.atmtmv [DBG] mgr option container_image_grafana = ceph/ceph-grafana:6.6.2
2020-08-27T14:07:25.535713+0000 mgr.vm-00.atmtmv [DBG] mgr option container_image_alertmanager = prom/alertmanager:v0.20.0
2020-08-27T14:07:25.535770+0000 mgr.vm-00.atmtmv [DBG] mgr option container_image_node_exporter = prom/node-exporter:v0.18.1
2020-08-27T14:07:25.535823+0000 mgr.vm-00.atmtmv [DBG] mgr option warn_on_stray_hosts = True
2020-08-27T14:07:25.535877+0000 mgr.vm-00.atmtmv [DBG] mgr option warn_on_stray_daemons = True
2020-08-27T14:07:25.535968+0000 mgr.vm-00.atmtmv [DBG] mgr option warn_on_failed_host_check = True
2020-08-27T14:07:25.536040+0000 mgr.vm-00.atmtmv [DBG] mgr option log_to_cluster = True
2020-08-27T14:07:25.536100+0000 mgr.vm-00.atmtmv [DBG] mgr option allow_ptrace = False
2020-08-27T14:07:25.536160+0000 mgr.vm-00.atmtmv [DBG] mgr option prometheus_alerts_path = /etc/prometheus/ceph/ceph_default_alerts.yml
2020-08-27T14:07:25.536219+0000 mgr.vm-00.atmtmv [DBG] mgr option migration_current = 2
2020-08-27T14:07:25.536276+0000 mgr.vm-00.atmtmv [DBG] mgr option config_dashboard = True
2020-08-27T14:07:25.536335+0000 mgr.vm-00.atmtmv [DBG] mgr option manage_etc_ceph_ceph_conf = False
2020-08-27T14:07:25.536394+0000 mgr.vm-00.atmtmv [DBG] mgr option registry_url = None
2020-08-27T14:07:25.536448+0000 mgr.vm-00.atmtmv [DBG] mgr option registry_username = None
2020-08-27T14:07:25.536505+0000 mgr.vm-00.atmtmv [DBG] mgr option registry_password = None
2020-08-27T14:07:25.536561+0000 mgr.vm-00.atmtmv [DBG] mgr option log_level =
2020-08-27T14:07:25.536638+0000 mgr.vm-00.atmtmv [DBG] mgr option log_to_file = False
2020-08-27T14:07:25.536698+0000 mgr.vm-00.atmtmv [DBG] mgr option log_to_cluster_level = debug
2020-08-27T14:07:25.542334+0000 mgr.vm-00.atmtmv [DBG] Loaded inventory {'vm-00': {'hostname': 'vm-00', 'addr': 'vm-00', 'labels': [], 'status': ''}, 'vm-01': {'hostname': 'vm-01', 'addr': 'vm-01', 'labels': [], 'status': ''}, 'vm-02': {'hostname': 'vm-02', 'addr': 'vm-02', 'labels': [], 'status': ''}}
2020-08-27T14:07:25.546029+0000 mgr.vm-00.atmtmv [DBG] HostCache.load: host vm-00 has 5 daemons, 3 devices, 3 networks
2020-08-27T14:07:25.546845+0000 mgr.vm-00.atmtmv [DBG] HostCache.load: host vm-01 has 6 daemons, 3 devices, 3 networks
2020-08-27T14:07:25.547439+0000 mgr.vm-00.atmtmv [DBG] HostCache.load: host vm-02 has 4 daemons, 3 devices, 3 networks
2020-08-27T14:07:25.547794+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for alertmanager
2020-08-27T14:07:25.547942+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for crash
2020-08-27T14:07:25.548092+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for grafana
2020-08-27T14:07:25.548219+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for mgr
2020-08-27T14:07:25.548344+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for mon
2020-08-27T14:07:25.548507+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for node-exporter
2020-08-27T14:07:25.548658+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for prometheus
2020-08-27T14:07:25.713317+0000 mgr.vm-00.atmtmv [DBG] serve starting
2020-08-27T14:07:25.713448+0000 mgr.vm-00.atmtmv [DBG] refreshing hosts and daemons
2020-08-27T14:07:25.718122+0000 mgr.vm-00.atmtmv [DBG] refreshing vm-00 daemons
2020-08-27T14:07:25.718316+0000 mgr.vm-00.atmtmv [DBG] Opening connection to root@vm-00 with ssh options '-F /tmp/cephadm-conf-x6j8kka8 -i /tmp/cephadm-identity-g62o7rrm'
2020-08-27T14:07:25.745762+0000 mgr.vm-00.atmtmv [DBG] refreshing vm-01 daemons
2020-08-27T14:07:25.745963+0000 mgr.vm-00.atmtmv [DBG] Opening connection to root@vm-01 with ssh options '-F /tmp/cephadm-conf-x6j8kka8 -i /tmp/cephadm-identity-g62o7rrm'
2020-08-27T14:07:25.761825+0000 mgr.vm-00.atmtmv [DBG] refreshing vm-02 daemons
2020-08-27T14:07:25.762037+0000 mgr.vm-00.atmtmv [DBG] Opening connection to root@vm-02 with ssh options '-F /tmp/cephadm-conf-x6j8kka8 -i /tmp/cephadm-identity-g62o7rrm'
2020-08-27T14:07:26.216718+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config get' -> 0 in 0.020s
2020-08-27T14:07:26.216871+0000 mgr.vm-00.atmtmv [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:26.217435+0000 mgr.vm-00.atmtmv [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:26.218146+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config get' -> 0 in 0.006s
2020-08-27T14:07:26.219360+0000 mgr.vm-00.atmtmv [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:26.219452+0000 mgr.vm-00.atmtmv [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:26.220631+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config get' -> 0 in 0.006s
2020-08-27T14:07:26.220798+0000 mgr.vm-00.atmtmv [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:26.220938+0000 mgr.vm-00.atmtmv [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:26.579174+0000 mgr.vm-00.atmtmv [DBG] process: completions=<CephadmCompletion>[
<lambda>(...),
]
2020-08-27T14:07:26.579401+0000 mgr.vm-00.atmtmv [INF] Redeploy service mgr
2020-08-27T14:07:26.588150+0000 mgr.vm-00.atmtmv [INF] Failing over to other MGR
2020-08-27T14:07:26.592671+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'auth get' -> 0 in 0.004s
2020-08-27T14:07:26.593058+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'auth get' -> 0 in 0.004s
2020-08-27T14:07:26.594837+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.002s
2020-08-27T14:07:26.595039+0000 mgr.vm-00.atmtmv [INF] Deploying daemon mgr.vm-01.arttvi on vm-01
2020-08-27T14:07:26.595257+0000 mgr.vm-00.atmtmv [DBG] Have connection to vm-01
2020-08-27T14:07:26.595580+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.003s
2020-08-27T14:07:26.595728+0000 mgr.vm-00.atmtmv [INF] Deploying daemon mgr.vm-02.qckgdm on vm-02
2020-08-27T14:07:26.595909+0000 mgr.vm-00.atmtmv [DBG] Have connection to vm-02
2020-08-27T14:07:26.597864+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config get' -> 0 in 0.002s
2020-08-27T14:07:26.598022+0000 mgr.vm-00.atmtmv [DBG] mgr.vm-02.qckgdm container image docker.io/amk3798/ceph:async
2020-08-27T14:07:26.598144+0000 mgr.vm-00.atmtmv [DBG] args: --image docker.io/amk3798/ceph:async deploy --fsid 7e4b7de8-e869-11ea-85bb-525400a8a9dd --name mgr.vm-02.qckgdm --config-json -
2020-08-27T14:07:26.598237+0000 mgr.vm-00.atmtmv [DBG] stdin: {"config": "# minimal ceph.conf for 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n[global]\n\tfsid = 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n\tmon_host = [v2:192.168.122.162:3300/0,v1:192.168.122.162:6789/0] [v2:192.168.122.93:3300/0,v1:192.168.122.93:6789/0] [v2:192.168.122.80:3300/0,v1:192.168.122.80:6789/0]\n", "keyring": "[mgr.vm-02.qckgdm]\n\tkey = AQAZt0dfWeLvFRAARLBl3YW1WiDo1yrgGKMnmQ==\n\tcaps mds = \"allow *\"\n\tcaps mon = \"profile mgr\"\n\tcaps osd = \"allow *\"\n"}
2020-08-27T14:07:26.599183+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config get' -> 0 in 0.004s
2020-08-27T14:07:26.599317+0000 mgr.vm-00.atmtmv [DBG] mgr.vm-01.arttvi container image docker.io/amk3798/ceph:async
2020-08-27T14:07:26.599425+0000 mgr.vm-00.atmtmv [DBG] args: --image docker.io/amk3798/ceph:async deploy --fsid 7e4b7de8-e869-11ea-85bb-525400a8a9dd --name mgr.vm-01.arttvi --config-json -
2020-08-27T14:07:26.599512+0000 mgr.vm-00.atmtmv [DBG] stdin: {"config": "# minimal ceph.conf for 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n[global]\n\tfsid = 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n\tmon_host = [v2:192.168.122.162:3300/0,v1:192.168.122.162:6789/0] [v2:192.168.122.93:3300/0,v1:192.168.122.93:6789/0] [v2:192.168.122.80:3300/0,v1:192.168.122.80:6789/0]\n", "keyring": "[mgr.vm-01.arttvi]\n\tkey = AQBuu0dfpjP4EBAAL4tpDJ8crZD7TxvY3MMywg==\n\tcaps mds = \"allow *\"\n\tcaps mon = \"profile mgr\"\n\tcaps osd = \"allow *\"\n"}
2020-08-27T14:07:32.592726+0000 mgr.vm-02.qckgdm [DBG] setting log level based on debug_mgr: WARNING (1/5)
2020-08-27T14:07:32.595824+0000 mgr.vm-02.qckgdm [DBG] mgr option ssh_config_file = None
2020-08-27T14:07:32.595919+0000 mgr.vm-02.qckgdm [DBG] mgr option device_cache_timeout = 1800
2020-08-27T14:07:32.595996+0000 mgr.vm-02.qckgdm [DBG] mgr option daemon_cache_timeout = 600
2020-08-27T14:07:32.596079+0000 mgr.vm-02.qckgdm [DBG] mgr option host_check_interval = 600
2020-08-27T14:07:32.596168+0000 mgr.vm-02.qckgdm [DBG] mgr option mode = root
2020-08-27T14:07:32.596291+0000 mgr.vm-02.qckgdm [DBG] mgr option container_image_base = docker.io/ceph/ceph
2020-08-27T14:07:32.596771+0000 mgr.vm-02.qckgdm [DBG] mgr option container_image_prometheus = prom/prometheus:v2.18.1
2020-08-27T14:07:32.596872+0000 mgr.vm-02.qckgdm [DBG] mgr option container_image_grafana = ceph/ceph-grafana:6.6.2
2020-08-27T14:07:32.596935+0000 mgr.vm-02.qckgdm [DBG] mgr option container_image_alertmanager = prom/alertmanager:v0.20.0
2020-08-27T14:07:32.596995+0000 mgr.vm-02.qckgdm [DBG] mgr option container_image_node_exporter = prom/node-exporter:v0.18.1
2020-08-27T14:07:32.597054+0000 mgr.vm-02.qckgdm [DBG] mgr option warn_on_stray_hosts = True
2020-08-27T14:07:32.597118+0000 mgr.vm-02.qckgdm [DBG] mgr option warn_on_stray_daemons = True
2020-08-27T14:07:32.597241+0000 mgr.vm-02.qckgdm [DBG] mgr option warn_on_failed_host_check = True
2020-08-27T14:07:32.597340+0000 mgr.vm-02.qckgdm [DBG] mgr option log_to_cluster = True
2020-08-27T14:07:32.597441+0000 mgr.vm-02.qckgdm [DBG] mgr option allow_ptrace = False
2020-08-27T14:07:32.597528+0000 mgr.vm-02.qckgdm [DBG] mgr option prometheus_alerts_path = /etc/prometheus/ceph/ceph_default_alerts.yml
2020-08-27T14:07:32.597607+0000 mgr.vm-02.qckgdm [DBG] mgr option migration_current = 2
2020-08-27T14:07:32.597673+0000 mgr.vm-02.qckgdm [DBG] mgr option config_dashboard = True
2020-08-27T14:07:32.597737+0000 mgr.vm-02.qckgdm [DBG] mgr option manage_etc_ceph_ceph_conf = False
2020-08-27T14:07:32.597802+0000 mgr.vm-02.qckgdm [DBG] mgr option registry_url = None
2020-08-27T14:07:32.597867+0000 mgr.vm-02.qckgdm [DBG] mgr option registry_username = None
2020-08-27T14:07:32.597945+0000 mgr.vm-02.qckgdm [DBG] mgr option registry_password = None
2020-08-27T14:07:32.598013+0000 mgr.vm-02.qckgdm [DBG] mgr option log_level =
2020-08-27T14:07:32.598079+0000 mgr.vm-02.qckgdm [DBG] mgr option log_to_file = False
2020-08-27T14:07:32.598147+0000 mgr.vm-02.qckgdm [DBG] mgr option log_to_cluster_level = debug
2020-08-27T14:07:32.606979+0000 mgr.vm-02.qckgdm [DBG] Loaded inventory {'vm-00': {'hostname': 'vm-00', 'addr': 'vm-00', 'labels': [], 'status': ''}, 'vm-01': {'hostname': 'vm-01', 'addr': 'vm-01', 'labels': [], 'status': ''}, 'vm-02': {'hostname': 'vm-02', 'addr': 'vm-02', 'labels': [], 'status': ''}}
2020-08-27T14:07:32.610211+0000 mgr.vm-02.qckgdm [DBG] HostCache.load: host vm-00 has 5 daemons, 3 devices, 3 networks
2020-08-27T14:07:32.611131+0000 mgr.vm-02.qckgdm [DBG] HostCache.load: host vm-01 has 6 daemons, 3 devices, 3 networks
2020-08-27T14:07:32.612032+0000 mgr.vm-02.qckgdm [DBG] HostCache.load: host vm-02 has 4 daemons, 3 devices, 3 networks
2020-08-27T14:07:32.612605+0000 mgr.vm-02.qckgdm [DBG] SpecStore: loaded spec for alertmanager
2020-08-27T14:07:32.612882+0000 mgr.vm-02.qckgdm [DBG] SpecStore: loaded spec for crash
2020-08-27T14:07:32.613146+0000 mgr.vm-02.qckgdm [DBG] SpecStore: loaded spec for grafana
2020-08-27T14:07:32.613430+0000 mgr.vm-02.qckgdm [DBG] SpecStore: loaded spec for mgr
2020-08-27T14:07:32.613707+0000 mgr.vm-02.qckgdm [DBG] SpecStore: loaded spec for mon
2020-08-27T14:07:32.613965+0000 mgr.vm-02.qckgdm [DBG] SpecStore: loaded spec for node-exporter
2020-08-27T14:07:32.614222+0000 mgr.vm-02.qckgdm [DBG] SpecStore: loaded spec for prometheus
2020-08-27T14:07:32.805405+0000 mgr.vm-02.qckgdm [DBG] serve starting
2020-08-27T14:07:32.805486+0000 mgr.vm-02.qckgdm [DBG] refreshing hosts and daemons
2020-08-27T14:07:32.805986+0000 mgr.vm-02.qckgdm [DBG] refreshing vm-00 daemons
2020-08-27T14:07:32.806120+0000 mgr.vm-02.qckgdm [DBG] Opening connection to root@vm-00 with ssh options '-F /tmp/cephadm-conf-3g6skorl -i /tmp/cephadm-identity-iwiqllgm'
2020-08-27T14:07:32.807407+0000 mgr.vm-02.qckgdm [DBG] refreshing vm-01 daemons
2020-08-27T14:07:32.807545+0000 mgr.vm-02.qckgdm [DBG] Opening connection to root@vm-01 with ssh options '-F /tmp/cephadm-conf-3g6skorl -i /tmp/cephadm-identity-iwiqllgm'
2020-08-27T14:07:32.808876+0000 mgr.vm-02.qckgdm [DBG] refreshing vm-02 daemons
2020-08-27T14:07:32.809004+0000 mgr.vm-02.qckgdm [DBG] Opening connection to root@vm-02 with ssh options '-F /tmp/cephadm-conf-3g6skorl -i /tmp/cephadm-identity-iwiqllgm'
2020-08-27T14:07:33.263978+0000 mgr.vm-02.qckgdm [DBG] mon_command: 'config get' -> 0 in 0.026s
2020-08-27T14:07:33.264207+0000 mgr.vm-02.qckgdm [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:33.267189+0000 mgr.vm-02.qckgdm [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:33.269199+0000 mgr.vm-02.qckgdm [DBG] mon_command: 'config get' -> 0 in 0.006s
2020-08-27T14:07:33.269376+0000 mgr.vm-02.qckgdm [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:33.269514+0000 mgr.vm-02.qckgdm [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:33.325709+0000 mgr.vm-02.qckgdm [DBG] mon_command: 'config get' -> 0 in 0.023s
2020-08-27T14:07:33.325815+0000 mgr.vm-02.qckgdm [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:33.325885+0000 mgr.vm-02.qckgdm [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:33.695671+0000 mgr.vm-02.qckgdm [DBG] process: completions=<CephadmCompletion>[
<lambda>(...),
]
2020-08-27T14:07:33.695943+0000 mgr.vm-02.qckgdm [INF] Redeploy service mgr
2020-08-27T14:07:33.712250+0000 mgr.vm-02.qckgdm [INF] Failing over to other MGR
2020-08-27T14:07:33.748208+0000 mgr.vm-02.qckgdm [DBG] mon_command: 'auth get' -> 0 in 0.049s
2020-08-27T14:07:33.748664+0000 mgr.vm-02.qckgdm [DBG] mon_command: 'auth get' -> 0 in 0.049s
2020-08-27T14:07:33.809308+0000 mgr.vm-02.qckgdm [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.060s
2020-08-27T14:07:33.809715+0000 mgr.vm-02.qckgdm [INF] Deploying daemon mgr.vm-01.arttvi on vm-01
2020-08-27T14:07:33.809981+0000 mgr.vm-02.qckgdm [DBG] Have connection to vm-01
2020-08-27T14:07:33.810419+0000 mgr.vm-02.qckgdm [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.062s
2020-08-27T14:07:33.810600+0000 mgr.vm-02.qckgdm [INF] Deploying daemon mgr.vm-00.atmtmv on vm-00
2020-08-27T14:07:33.810787+0000 mgr.vm-02.qckgdm [DBG] Have connection to vm-00
2020-08-27T14:07:33.850054+0000 mgr.vm-02.qckgdm [DBG] mon_command: 'config get' -> 0 in 0.039s
2020-08-27T14:07:33.850217+0000 mgr.vm-02.qckgdm [DBG] mgr.vm-00.atmtmv container image docker.io/amk3798/ceph:async
2020-08-27T14:07:33.850413+0000 mgr.vm-02.qckgdm [DBG] args: --image docker.io/amk3798/ceph:async deploy --fsid 7e4b7de8-e869-11ea-85bb-525400a8a9dd --name mgr.vm-00.atmtmv --config-json -
2020-08-27T14:07:33.850556+0000 mgr.vm-02.qckgdm [DBG] stdin: {"config": "# minimal ceph.conf for 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n[global]\n\tfsid = 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n\tmon_host = [v2:192.168.122.162:3300/0,v1:192.168.122.162:6789/0] [v2:192.168.122.93:3300/0,v1:192.168.122.93:6789/0] [v2:192.168.122.80:3300/0,v1:192.168.122.80:6789/0]\n", "keyring": "[mgr.vm-00.atmtmv]\n\tkey = AQDitUdf2gKWMhAAiqPAkPwfDuwfMgtWXeAnjw==\n\tcaps mds = \"allow *\"\n\tcaps mon = \"profile mgr\"\n\tcaps osd = \"allow *\"\n"}
2020-08-27T14:07:33.851771+0000 mgr.vm-02.qckgdm [DBG] mon_command: 'config get' -> 0 in 0.042s
2020-08-27T14:07:33.851901+0000 mgr.vm-02.qckgdm [DBG] mgr.vm-01.arttvi container image docker.io/amk3798/ceph:async
2020-08-27T14:07:33.851999+0000 mgr.vm-02.qckgdm [DBG] args: --image docker.io/amk3798/ceph:async deploy --fsid 7e4b7de8-e869-11ea-85bb-525400a8a9dd --name mgr.vm-01.arttvi --config-json -
2020-08-27T14:07:33.852080+0000 mgr.vm-02.qckgdm [DBG] stdin: {"config": "# minimal ceph.conf for 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n[global]\n\tfsid = 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n\tmon_host = [v2:192.168.122.162:3300/0,v1:192.168.122.162:6789/0] [v2:192.168.122.93:3300/0,v1:192.168.122.93:6789/0] [v2:192.168.122.80:3300/0,v1:192.168.122.80:6789/0]\n", "keyring": "[mgr.vm-01.arttvi]\n\tkey = AQBuu0dfpjP4EBAAL4tpDJ8crZD7TxvY3MMywg==\n\tcaps mds = \"allow *\"\n\tcaps mon = \"profile mgr\"\n\tcaps osd = \"allow *\"\n"}
2020-08-27T14:07:47.737438+0000 mgr.vm-01.arttvi [DBG] setting log level based on debug_mgr: WARNING (1/5)
2020-08-27T14:07:47.744369+0000 mgr.vm-01.arttvi [DBG] mgr option ssh_config_file = None
2020-08-27T14:07:47.747415+0000 mgr.vm-01.arttvi [DBG] mgr option device_cache_timeout = 1800
2020-08-27T14:07:47.747507+0000 mgr.vm-01.arttvi [DBG] mgr option daemon_cache_timeout = 600
2020-08-27T14:07:47.747567+0000 mgr.vm-01.arttvi [DBG] mgr option host_check_interval = 600
2020-08-27T14:07:47.747621+0000 mgr.vm-01.arttvi [DBG] mgr option mode = root
2020-08-27T14:07:47.747704+0000 mgr.vm-01.arttvi [DBG] mgr option container_image_base = docker.io/ceph/ceph
2020-08-27T14:07:47.747759+0000 mgr.vm-01.arttvi [DBG] mgr option container_image_prometheus = prom/prometheus:v2.18.1
2020-08-27T14:07:47.747814+0000 mgr.vm-01.arttvi [DBG] mgr option container_image_grafana = ceph/ceph-grafana:6.6.2
2020-08-27T14:07:47.747864+0000 mgr.vm-01.arttvi [DBG] mgr option container_image_alertmanager = prom/alertmanager:v0.20.0
2020-08-27T14:07:47.747929+0000 mgr.vm-01.arttvi [DBG] mgr option container_image_node_exporter = prom/node-exporter:v0.18.1
2020-08-27T14:07:47.747984+0000 mgr.vm-01.arttvi [DBG] mgr option warn_on_stray_hosts = True
2020-08-27T14:07:47.748036+0000 mgr.vm-01.arttvi [DBG] mgr option warn_on_stray_daemons = True
2020-08-27T14:07:47.748091+0000 mgr.vm-01.arttvi [DBG] mgr option warn_on_failed_host_check = True
2020-08-27T14:07:47.748143+0000 mgr.vm-01.arttvi [DBG] mgr option log_to_cluster = True
2020-08-27T14:07:47.748195+0000 mgr.vm-01.arttvi [DBG] mgr option allow_ptrace = False
2020-08-27T14:07:47.748245+0000 mgr.vm-01.arttvi [DBG] mgr option prometheus_alerts_path = /etc/prometheus/ceph/ceph_default_alerts.yml
2020-08-27T14:07:47.748303+0000 mgr.vm-01.arttvi [DBG] mgr option migration_current = 2
2020-08-27T14:07:47.748383+0000 mgr.vm-01.arttvi [DBG] mgr option config_dashboard = True
2020-08-27T14:07:47.748437+0000 mgr.vm-01.arttvi [DBG] mgr option manage_etc_ceph_ceph_conf = False
2020-08-27T14:07:47.748489+0000 mgr.vm-01.arttvi [DBG] mgr option registry_url = None
2020-08-27T14:07:47.748540+0000 mgr.vm-01.arttvi [DBG] mgr option registry_username = None
2020-08-27T14:07:47.748592+0000 mgr.vm-01.arttvi [DBG] mgr option registry_password = None
2020-08-27T14:07:47.748660+0000 mgr.vm-01.arttvi [DBG] mgr option log_level =
2020-08-27T14:07:47.748715+0000 mgr.vm-01.arttvi [DBG] mgr option log_to_file = False
2020-08-27T14:07:47.748770+0000 mgr.vm-01.arttvi [DBG] mgr option log_to_cluster_level = debug
2020-08-27T14:07:47.754024+0000 mgr.vm-01.arttvi [DBG] Loaded inventory {'vm-00': {'hostname': 'vm-00', 'addr': 'vm-00', 'labels': [], 'status': ''}, 'vm-01': {'hostname': 'vm-01', 'addr': 'vm-01', 'labels': [], 'status': ''}, 'vm-02': {'hostname': 'vm-02', 'addr': 'vm-02', 'labels': [], 'status': ''}}
2020-08-27T14:07:47.757550+0000 mgr.vm-01.arttvi [DBG] HostCache.load: host vm-00 has 5 daemons, 3 devices, 3 networks
2020-08-27T14:07:47.758590+0000 mgr.vm-01.arttvi [DBG] HostCache.load: host vm-01 has 6 daemons, 3 devices, 3 networks
2020-08-27T14:07:47.759381+0000 mgr.vm-01.arttvi [DBG] HostCache.load: host vm-02 has 4 daemons, 3 devices, 3 networks
2020-08-27T14:07:47.759800+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for alertmanager
2020-08-27T14:07:47.760020+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for crash
2020-08-27T14:07:47.760156+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for grafana
2020-08-27T14:07:47.760280+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for mgr
2020-08-27T14:07:47.760395+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for mon
2020-08-27T14:07:47.760511+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for node-exporter
2020-08-27T14:07:47.760644+0000 mgr.vm-01.arttvi [DBG] SpecStore: loaded spec for prometheus
2020-08-27T14:07:47.935569+0000 mgr.vm-01.arttvi [DBG] serve starting
2020-08-27T14:07:47.935726+0000 mgr.vm-01.arttvi [DBG] refreshing hosts and daemons
2020-08-27T14:07:47.936394+0000 mgr.vm-01.arttvi [DBG] refreshing vm-00 daemons
2020-08-27T14:07:47.936517+0000 mgr.vm-01.arttvi [DBG] Opening connection to root@vm-00 with ssh options '-F /tmp/cephadm-conf-m0p3r7ge -i /tmp/cephadm-identity-kbowm9hx'
2020-08-27T14:07:47.946116+0000 mgr.vm-01.arttvi [DBG] refreshing vm-01 daemons
2020-08-27T14:07:47.946340+0000 mgr.vm-01.arttvi [DBG] Opening connection to root@vm-01 with ssh options '-F /tmp/cephadm-conf-m0p3r7ge -i /tmp/cephadm-identity-kbowm9hx'
2020-08-27T14:07:47.948168+0000 mgr.vm-01.arttvi [DBG] refreshing vm-02 daemons
2020-08-27T14:07:47.949059+0000 mgr.vm-01.arttvi [DBG] Opening connection to root@vm-02 with ssh options '-F /tmp/cephadm-conf-m0p3r7ge -i /tmp/cephadm-identity-kbowm9hx'
2020-08-27T14:07:48.416077+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config get' -> 0 in 0.012s
2020-08-27T14:07:48.416229+0000 mgr.vm-01.arttvi [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:48.416336+0000 mgr.vm-01.arttvi [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:48.417225+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config get' -> 0 in 0.015s
2020-08-27T14:07:48.417356+0000 mgr.vm-01.arttvi [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:48.417499+0000 mgr.vm-01.arttvi [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:48.427711+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config get' -> 0 in 0.003s
2020-08-27T14:07:48.428121+0000 mgr.vm-01.arttvi [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:48.428250+0000 mgr.vm-01.arttvi [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:48.902340+0000 mgr.vm-01.arttvi [DBG] process: completions=<CephadmCompletion>[
<lambda>(...),
]
2020-08-27T14:07:48.902790+0000 mgr.vm-01.arttvi [INF] Redeploy service mgr
2020-08-27T14:07:48.918875+0000 mgr.vm-01.arttvi [INF] Failing over to other MGR
2020-08-27T14:07:48.919177+0000 mgr.vm-01.arttvi [DBG] mon_command: 'auth get' -> 0 in 0.014s
2020-08-27T14:07:48.921841+0000 mgr.vm-01.arttvi [DBG] mon_command: 'auth get' -> 0 in 0.004s
2020-08-27T14:07:48.922238+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.003s
2020-08-27T14:07:48.922408+0000 mgr.vm-01.arttvi [INF] Deploying daemon mgr.vm-00.atmtmv on vm-00
2020-08-27T14:07:48.922654+0000 mgr.vm-01.arttvi [DBG] Have connection to vm-00
2020-08-27T14:07:48.925156+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.003s
2020-08-27T14:07:48.925312+0000 mgr.vm-01.arttvi [INF] Deploying daemon mgr.vm-02.qckgdm on vm-02
2020-08-27T14:07:48.925529+0000 mgr.vm-01.arttvi [DBG] Have connection to vm-02
2020-08-27T14:07:48.925951+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config get' -> 0 in 0.003s
2020-08-27T14:07:48.926077+0000 mgr.vm-01.arttvi [DBG] mgr.vm-00.atmtmv container image docker.io/amk3798/ceph:async
2020-08-27T14:07:48.926196+0000 mgr.vm-01.arttvi [DBG] args: --image docker.io/amk3798/ceph:async deploy --fsid 7e4b7de8-e869-11ea-85bb-525400a8a9dd --name mgr.vm-00.atmtmv --config-json -
2020-08-27T14:07:48.926287+0000 mgr.vm-01.arttvi [DBG] stdin: {"config": "# minimal ceph.conf for 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n[global]\n\tfsid = 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n\tmon_host = [v2:192.168.122.162:3300/0,v1:192.168.122.162:6789/0] [v2:192.168.122.93:3300/0,v1:192.168.122.93:6789/0] [v2:192.168.122.80:3300/0,v1:192.168.122.80:6789/0]\n", "keyring": "[mgr.vm-00.atmtmv]\n\tkey = AQDitUdf2gKWMhAAiqPAkPwfDuwfMgtWXeAnjw==\n\tcaps mds = \"allow *\"\n\tcaps mon = \"profile mgr\"\n\tcaps osd = \"allow *\"\n"}
2020-08-27T14:07:48.931024+0000 mgr.vm-01.arttvi [DBG] mon_command: 'config get' -> 0 in 0.005s
2020-08-27T14:07:48.931195+0000 mgr.vm-01.arttvi [DBG] mgr.vm-02.qckgdm container image docker.io/amk3798/ceph:async
2020-08-27T14:07:48.931321+0000 mgr.vm-01.arttvi [DBG] args: --image docker.io/amk3798/ceph:async deploy --fsid 7e4b7de8-e869-11ea-85bb-525400a8a9dd --name mgr.vm-02.qckgdm --config-json -
2020-08-27T14:07:48.931440+0000 mgr.vm-01.arttvi [DBG] stdin: {"config": "# minimal ceph.conf for 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n[global]\n\tfsid = 7e4b7de8-e869-11ea-85bb-525400a8a9dd\n\tmon_host = [v2:192.168.122.162:3300/0,v1:192.168.122.162:6789/0] [v2:192.168.122.93:3300/0,v1:192.168.122.93:6789/0] [v2:192.168.122.80:3300/0,v1:192.168.122.80:6789/0]\n", "keyring": "[mgr.vm-02.qckgdm]\n\tkey = AQAZt0dfWeLvFRAARLBl3YW1WiDo1yrgGKMnmQ==\n\tcaps mds = \"allow *\"\n\tcaps mon = \"profile mgr\"\n\tcaps osd = \"allow *\"\n"}
2020-08-27T14:07:56.891501+0000 mgr.vm-00.atmtmv [DBG] setting log level based on debug_mgr: WARNING (1/5)
2020-08-27T14:07:56.896831+0000 mgr.vm-00.atmtmv [DBG] mgr option ssh_config_file = None
2020-08-27T14:07:56.897488+0000 mgr.vm-00.atmtmv [DBG] mgr option device_cache_timeout = 1800
2020-08-27T14:07:56.897671+0000 mgr.vm-00.atmtmv [DBG] mgr option daemon_cache_timeout = 600
2020-08-27T14:07:56.897802+0000 mgr.vm-00.atmtmv [DBG] mgr option host_check_interval = 600
2020-08-27T14:07:56.897909+0000 mgr.vm-00.atmtmv [DBG] mgr option mode = root
2020-08-27T14:07:56.898054+0000 mgr.vm-00.atmtmv [DBG] mgr option container_image_base = docker.io/ceph/ceph
2020-08-27T14:07:56.898178+0000 mgr.vm-00.atmtmv [DBG] mgr option container_image_prometheus = prom/prometheus:v2.18.1
2020-08-27T14:07:56.903412+0000 mgr.vm-00.atmtmv [DBG] mgr option container_image_grafana = ceph/ceph-grafana:6.6.2
2020-08-27T14:07:56.903547+0000 mgr.vm-00.atmtmv [DBG] mgr option container_image_alertmanager = prom/alertmanager:v0.20.0
2020-08-27T14:07:56.903684+0000 mgr.vm-00.atmtmv [DBG] mgr option container_image_node_exporter = prom/node-exporter:v0.18.1
2020-08-27T14:07:56.903799+0000 mgr.vm-00.atmtmv [DBG] mgr option warn_on_stray_hosts = True
2020-08-27T14:07:56.903928+0000 mgr.vm-00.atmtmv [DBG] mgr option warn_on_stray_daemons = True
2020-08-27T14:07:56.904053+0000 mgr.vm-00.atmtmv [DBG] mgr option warn_on_failed_host_check = True
2020-08-27T14:07:56.904156+0000 mgr.vm-00.atmtmv [DBG] mgr option log_to_cluster = True
2020-08-27T14:07:56.904256+0000 mgr.vm-00.atmtmv [DBG] mgr option allow_ptrace = False
2020-08-27T14:07:56.904352+0000 mgr.vm-00.atmtmv [DBG] mgr option prometheus_alerts_path = /etc/prometheus/ceph/ceph_default_alerts.yml
2020-08-27T14:07:56.904459+0000 mgr.vm-00.atmtmv [DBG] mgr option migration_current = 2
2020-08-27T14:07:56.904563+0000 mgr.vm-00.atmtmv [DBG] mgr option config_dashboard = True
2020-08-27T14:07:56.905873+0000 mgr.vm-00.atmtmv [DBG] mgr option manage_etc_ceph_ceph_conf = False
2020-08-27T14:07:56.906118+0000 mgr.vm-00.atmtmv [DBG] mgr option registry_url = None
2020-08-27T14:07:56.906248+0000 mgr.vm-00.atmtmv [DBG] mgr option registry_username = None
2020-08-27T14:07:56.906356+0000 mgr.vm-00.atmtmv [DBG] mgr option registry_password = None
2020-08-27T14:07:56.906480+0000 mgr.vm-00.atmtmv [DBG] mgr option log_level =
2020-08-27T14:07:56.906583+0000 mgr.vm-00.atmtmv [DBG] mgr option log_to_file = False
2020-08-27T14:07:56.906713+0000 mgr.vm-00.atmtmv [DBG] mgr option log_to_cluster_level = debug
2020-08-27T14:07:56.916312+0000 mgr.vm-00.atmtmv [DBG] Loaded inventory {'vm-00': {'hostname': 'vm-00', 'addr': 'vm-00', 'labels': [], 'status': ''}, 'vm-01': {'hostname': 'vm-01', 'addr': 'vm-01', 'labels': [], 'status': ''}, 'vm-02': {'hostname': 'vm-02', 'addr': 'vm-02', 'labels': [], 'status': ''}}
2020-08-27T14:07:56.921188+0000 mgr.vm-00.atmtmv [DBG] HostCache.load: host vm-00 has 5 daemons, 3 devices, 3 networks
2020-08-27T14:07:56.923398+0000 mgr.vm-00.atmtmv [DBG] HostCache.load: host vm-01 has 6 daemons, 3 devices, 3 networks
2020-08-27T14:07:56.924689+0000 mgr.vm-00.atmtmv [DBG] HostCache.load: host vm-02 has 4 daemons, 3 devices, 3 networks
2020-08-27T14:07:56.925316+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for alertmanager
2020-08-27T14:07:56.925573+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for crash
2020-08-27T14:07:56.925833+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for grafana
2020-08-27T14:07:56.926071+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for mgr
2020-08-27T14:07:56.926288+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for mon
2020-08-27T14:07:56.926496+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for node-exporter
2020-08-27T14:07:56.927029+0000 mgr.vm-00.atmtmv [DBG] SpecStore: loaded spec for prometheus
2020-08-27T14:07:57.368469+0000 mgr.vm-00.atmtmv [DBG] serve starting
2020-08-27T14:07:57.368692+0000 mgr.vm-00.atmtmv [DBG] refreshing hosts and daemons
2020-08-27T14:07:57.369447+0000 mgr.vm-00.atmtmv [DBG] refreshing vm-00 daemons
2020-08-27T14:07:57.369654+0000 mgr.vm-00.atmtmv [DBG] Opening connection to root@vm-00 with ssh options '-F /tmp/cephadm-conf-1a9yrzyx -i /tmp/cephadm-identity-rc4ji9cz'
2020-08-27T14:07:57.374021+0000 mgr.vm-00.atmtmv [DBG] refreshing vm-01 daemons
2020-08-27T14:07:57.374767+0000 mgr.vm-00.atmtmv [DBG] Opening connection to root@vm-01 with ssh options '-F /tmp/cephadm-conf-1a9yrzyx -i /tmp/cephadm-identity-rc4ji9cz'
2020-08-27T14:07:57.384125+0000 mgr.vm-00.atmtmv [DBG] refreshing vm-02 daemons
2020-08-27T14:07:57.384290+0000 mgr.vm-00.atmtmv [DBG] Opening connection to root@vm-02 with ssh options '-F /tmp/cephadm-conf-1a9yrzyx -i /tmp/cephadm-identity-rc4ji9cz'
2020-08-27T14:07:57.914311+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config get' -> 0 in 0.021s
2020-08-27T14:07:57.915737+0000 mgr.vm-00.atmtmv [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:57.915893+0000 mgr.vm-00.atmtmv [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:57.945377+0000 mgr.vm-00.atmtmv [DBG] mon_command: 'config get' -> 0 in 0.030s
2020-08-27T14:07:57.950913+0000 mgr.vm-00.atmtmv [DBG] mon container image docker.io/amk3798/ceph:async
2020-08-27T14:07:57.951135+0000 mgr.vm-00.atmtmv [DBG] args: --image docker.io/amk3798/ceph:async ls
2020-08-27T14:07:57.992326+0000 mgr.vm-00.atmtmv [DBG] process: completions=<CephadmCompletion>[
<lambda>(...),
]
2020-08-27T14:07:57.997792+0000 mgr.vm-00.atmtmv [INF] Redeploy service mgr
2020-08-27T14:07:58.005139+0000 mgr.vm-00.atmtmv [INF] Failing over to other MGR
Updated by Sebastian Wagner over 3 years ago
- Related to Bug #47332: repo_digest: Follow up's added
Updated by Sebastian Wagner over 3 years ago
- Status changed from New to Fix Under Review
- Pull request ID set to 37139
Updated by Sebastian Wagner over 3 years ago
- Pull request ID changed from 37139 to 38766
Updated by Sebastian Wagner about 3 years ago
- Status changed from Fix Under Review to Resolved
Actions