Ceph » Orchestrator

only seen on ubuntu 18.04

cephadm reports name already in use for mon.c

2021-09-23T22:00:17.992 INFO:tasks.cephadm:Adding mon.c on smithi163
2021-09-23T22:00:17.992 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c02a53453e418d89a20f49c1eaae7fb050b79d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31509b24-1cb9-11ec-8c25-001a4aab830c -- ceph orch daemon add mon 'smithi163:[v1:172.21.15.163:6789]=c'
2021-09-23T22:00:18.446 INFO:journalctl@ceph.mon.b.smithi076.stdout:Sep 23 22:00:18 smithi076 bash[12703]: cluster 2021-09-23T22:00:17.018366+0000 mgr.y (mgr.14143) 61 : cluster [DBG] pgmap v46: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2021-09-23T22:00:18.446 INFO:journalctl@ceph.mon.b.smithi076.stdout:Sep 23 22:00:18 smithi076 bash[12703]: audit 2021-09-23T22:00:17.357130+0000 mon.a (mon.0) 173 : audit [DBG] from='client.? v1:172.21.15.76:0/2164750106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch
2021-09-23T22:00:18.446 INFO:journalctl@ceph.mon.b.smithi076.stdout:Sep 23 22:00:18 smithi076 bash[12703]: audit 2021-09-23T22:00:18.086933+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch
2021-09-23T22:00:18.475 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:18 smithi074 bash[12738]: cluster 2021-09-23T22:00:17.018366+0000 mgr.y (mgr.14143) 61 : cluster [DBG] pgmap v46: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2021-09-23T22:00:18.475 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:18 smithi074 bash[12738]: audit 2021-09-23T22:00:17.357130+0000 mon.a (mon.0) 173 : audit [DBG] from='client.? v1:172.21.15.76:0/2164750106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch
2021-09-23T22:00:18.475 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:18 smithi074 bash[12738]: audit 2021-09-23T22:00:18.086933+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch
2021-09-23T22:00:20.946 INFO:journalctl@ceph.mon.b.smithi076.stdout:Sep 23 22:00:20 smithi076 bash[12703]: cluster 2021-09-23T22:00:19.018783+0000 mgr.y (mgr.14143) 62 : cluster [DBG] pgmap v47: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2021-09-23T22:00:20.975 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:20 smithi074 bash[12738]: cluster 2021-09-23T22:00:19.018783+0000 mgr.y (mgr.14143) 62 : cluster [DBG] pgmap v47: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2021-09-23T22:00:21.946 INFO:journalctl@ceph.mon.b.smithi076.stdout:Sep 23 22:00:21 smithi076 bash[12703]: audit 2021-09-23T22:00:20.966139+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch
2021-09-23T22:00:21.946 INFO:journalctl@ceph.mon.b.smithi076.stdout:Sep 23 22:00:21 smithi076 bash[12703]: audit 2021-09-23T22:00:20.967362+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
2021-09-23T22:00:21.947 INFO:journalctl@ceph.mon.b.smithi076.stdout:Sep 23 22:00:21 smithi076 bash[12703]: audit 2021-09-23T22:00:20.968692+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.c", "key": "container_image"}]: dispatch
2021-09-23T22:00:21.975 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:21 smithi074 bash[12738]: audit 2021-09-23T22:00:20.966139+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch
2021-09-23T22:00:21.975 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:21 smithi074 bash[12738]: audit 2021-09-23T22:00:20.967362+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
2021-09-23T22:00:21.975 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:21 smithi074 bash[12738]: audit 2021-09-23T22:00:20.968692+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.c", "key": "container_image"}]: dispatch
2021-09-23T22:00:22.946 INFO:journalctl@ceph.mon.b.smithi076.stdout:Sep 23 22:00:22 smithi076 bash[12703]: audit 2021-09-23T22:00:20.964572+0000 mgr.y (mgr.14143) 63 : audit [DBG] from='client.14176 v1:172.21.15.163:0/3875769332' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi163:[v1:172.21.15.163:6789]=c", "target": ["mon-mgr", ""]}]: dispatch
2021-09-23T22:00:22.946 INFO:journalctl@ceph.mon.b.smithi076.stdout:Sep 23 22:00:22 smithi076 bash[12703]: cephadm 2021-09-23T22:00:20.968235+0000 mgr.y (mgr.14143) 64 : cephadm [INF] Deploying daemon mon.c on smithi163
2021-09-23T22:00:22.946 INFO:journalctl@ceph.mon.b.smithi076.stdout:Sep 23 22:00:22 smithi076 bash[12703]: cluster 2021-09-23T22:00:21.019118+0000 mgr.y (mgr.14143) 65 : cluster [DBG] pgmap v48: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2021-09-23T22:00:22.975 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:22 smithi074 bash[12738]: audit 2021-09-23T22:00:20.964572+0000 mgr.y (mgr.14143) 63 : audit [DBG] from='client.14176 v1:172.21.15.163:0/3875769332' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi163:[v1:172.21.15.163:6789]=c", "target": ["mon-mgr", ""]}]: dispatch
2021-09-23T22:00:22.975 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:22 smithi074 bash[12738]: cephadm 2021-09-23T22:00:20.968235+0000 mgr.y (mgr.14143) 64 : cephadm [INF] Deploying daemon mon.c on smithi163
2021-09-23T22:00:22.975 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:22 smithi074 bash[12738]: cluster 2021-09-23T22:00:21.019118+0000 mgr.y (mgr.14143) 65 : cluster [DBG] pgmap v48: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2021-09-23T22:00:23.945 INFO:journalctl@ceph.mon.b.smithi076.stdout:Sep 23 22:00:23 smithi076 bash[12703]: audit 2021-09-23T22:00:21.785972+0000 mgr.y (mgr.14143) 66 : audit [DBG] from='client.14176 v1:172.21.15.163:0/3875769332' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi163:[v1:172.21.15.163:6789]=c", "target": ["mon-mgr", ""]}]: dispatch
2021-09-23T22:00:23.975 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:23 smithi074 bash[12738]: audit 2021-09-23T22:00:21.785972+0000 mgr.y (mgr.14143) 66 : audit [DBG] from='client.14176 v1:172.21.15.163:0/3875769332' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi163:[v1:172.21.15.163:6789]=c", "target": ["mon-mgr", ""]}]: dispatch
2021-09-23T22:00:24.945 INFO:journalctl@ceph.mon.b.smithi076.stdout:Sep 23 22:00:24 smithi076 bash[12703]: cluster 2021-09-23T22:00:23.019560+0000 mgr.y (mgr.14143) 67 : cluster [DBG] pgmap v49: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2021-09-23T22:00:24.951 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:24 smithi074 bash[12738]: cluster 2021-09-23T22:00:23.019560+0000 mgr.y (mgr.14143) 67 : cluster [DBG] pgmap v49: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2021-09-23T22:00:24.954 INFO:teuthology.orchestra.run.smithi163.stderr:Error EINVAL: name mon.c already in use

successfully added mon.b logs:

2021-09-23T22:00:03.289 INFO:tasks.cephadm:Adding mon.b on smithi076
2021-09-23T22:00:03.290 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c02a53453e418d89a20f49c1eaae7fb050b79d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31509b24-1cb9-11ec-8c25-001a4aab830c -- ceph orch daemon add mon 'smithi076:[v1:172.21.15.76:6789]=b'
2021-09-23T22:00:03.975 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:03 smithi074 bash[12738]: audit 2021-09-23T22:00:02.669416+0000 mon.a (mon.0) 143 : audit [INF] from='client.? v1:172.21.15.74:0/4292592070' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished
2021-09-23T22:00:03.976 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:03 smithi074 bash[12738]: cluster 2021-09-23T22:00:02.669533+0000 mon.a (mon.0) 144 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in
2021-09-23T22:00:03.976 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:03 smithi074 bash[12738]: audit 2021-09-23T22:00:02.674684+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch
2021-09-23T22:00:03.976 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:03 smithi074 bash[12738]: cluster 2021-09-23T22:00:03.015740+0000 mgr.y (mgr.14143) 51 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail
2021-09-23T22:00:04.975 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:04 smithi074 bash[12738]: audit 2021-09-23T22:00:03.680733+0000 mon.a (mon.0) 146 : audit [INF] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished
2021-09-23T22:00:04.976 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:04 smithi074 bash[12738]: cluster 2021-09-23T22:00:03.680912+0000 mon.a (mon.0) 147 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in
2021-09-23T22:00:04.976 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:04 smithi074 bash[12738]: audit 2021-09-23T22:00:03.682410+0000 mon.a (mon.0) 148 : audit [INF] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch
2021-09-23T22:00:05.976 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:05 smithi074 bash[12738]: audit 2021-09-23T22:00:04.681063+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished
2021-09-23T22:00:05.976 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:05 smithi074 bash[12738]: cluster 2021-09-23T22:00:04.681187+0000 mon.a (mon.0) 150 : cluster [DBG] osdmap e6: 0 total, 0 up, 0 in
2021-09-23T22:00:05.976 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:05 smithi074 bash[12738]: cluster 2021-09-23T22:00:05.016147+0000 mgr.y (mgr.14143) 52 : cluster [DBG] pgmap v40: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2021-09-23T22:00:06.976 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:06 smithi074 bash[12738]: audit 2021-09-23T22:00:06.186677+0000 mgr.y (mgr.14143) 53 : audit [DBG] from='client.14169 v1:172.21.15.76:0/2294400580' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi076:[v1:172.21.15.76:6789]=b", "target": ["mon-mgr", ""]}]: dispatch
2021-09-23T22:00:06.977 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:06 smithi074 bash[12738]: audit 2021-09-23T22:00:06.190625+0000 mon.a (mon.0) 151 : audit [INF] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch
2021-09-23T22:00:06.977 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:06 smithi074 bash[12738]: audit 2021-09-23T22:00:06.192341+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
2021-09-23T22:00:06.977 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:06 smithi074 bash[12738]: cephadm 2021-09-23T22:00:06.193564+0000 mgr.y (mgr.14143) 54 : cephadm [INF] Deploying daemon mon.b on smithi076
2021-09-23T22:00:06.977 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:06 smithi074 bash[12738]: audit 2021-09-23T22:00:06.194225+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.b", "key": "container_image"}]: dispatch
2021-09-23T22:00:07.975 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:07 smithi074 bash[12738]: cluster 2021-09-23T22:00:07.016468+0000 mgr.y (mgr.14143) 55 : cluster [DBG] pgmap v41: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2021-09-23T22:00:10.283 INFO:teuthology.orchestra.run.smithi076.stdout:Deployed mon.b on host 'smithi076'
2021-09-23T22:00:10.975 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:10 smithi074 bash[12738]: cluster 2021-09-23T22:00:09.016865+0000 mgr.y (mgr.14143) 56 : cluster [DBG] pgmap v42: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2021-09-23T22:00:10.976 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:10 smithi074 bash[12738]: audit 2021-09-23T22:00:10.282903+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y'
2021-09-23T22:00:10.976 INFO:journalctl@ceph.mon.a.smithi074.stdout:Sep 23 22:00:10 smithi074 bash[12738]: audit 2021-09-23T22:00:10.283456+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14143 v1:172.21.15.74:0/2250466230' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch
2021-09-23T22:00:11.437 DEBUG:teuthology.orchestra.run.smithi076:mon.b> sudo journalctl -f -n 0 -u ceph-31509b24-1cb9-11ec-8c25-001a4aab830c@mon.b.service

http://qa-proxy.ceph.com/teuthology/yuriw-2021-09-23_20:23:10-rados-wip-yuri-testing-2021-09-23-0811-octopus-distro-basic-smithi/6404819/teuthology.log