[dpivonka@localhost kcli_plans]$ ssh fedora@192.168.122.246 fedora@192.168.122.246's password: [fedora@node00 ~]$ [fedora@node00 ~]$ [fedora@node00 ~]$ [fedora@node00 ~]$ [fedora@node00 ~]$ rsync dpivonka@192.168.0.115:/home/dpivonka/Documents/ceph/src/cephadm/cephadm . The authenticity of host '192.168.0.115 (192.168.0.115)' can't be established. ECDSA key fingerprint is SHA256:J3NnR0OIbse1AehwtLkORKLoOSHS9NhDMyv5aRWgsLY. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.0.115' (ECDSA) to the list of known hosts. dpivonka@192.168.0.115's password: [fedora@node00 ~]$ [fedora@node00 ~]$ [fedora@node00 ~]$ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.122.246 node00 192.168.122.167 node01 192.168.122.30 node02 [fedora@node00 ~]$ [fedora@node00 ~]$ [fedora@node00 ~]$ ./cephadm --image quay.io/daniel_pivonka/ceph:latest bootstrap --mon-ip 192.168.122.246 --ssh-user fedora ERROR: cephadm should be run as root [fedora@node00 ~]$ [fedora@node00 ~]$ sudo ./cephadm --image quay.io/daniel_pivonka/ceph:latest bootstrap --mon-ip 192.168.122.246 --ssh-user fedora Creating directory /etc/ceph for ceph.conf Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... podman|docker (/usr/bin/podman) is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK Cluster fsid: bfe58fc0-9c71-11eb-893e-52540031b82d Verifying IP 192.168.122.246 port 3300 ... Verifying IP 192.168.122.246 port 6789 ... Mon IP 192.168.122.246 is in CIDR network 192.168.122.0/24 - internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Pulling container image quay.io/daniel_pivonka/ceph:latest... Ceph version: ceph version 17.0.0-2956-g9d7a42e7 (9d7a42e709558e51e7b83fc404c5ee4223c061ea) quincy (dev) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting mon public_network to 192.168.122.0/24 Wrote config to /etc/ceph/ceph.conf Wrote keyring to /etc/ceph/ceph.client.admin.keyring Creating mgr... Verifying port 9283 ... Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/15)... mgr not available, waiting (2/15)... mgr not available, waiting (3/15)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for mgr epoch 5... mgr epoch 5 is available Setting orchestrator backend to cephadm... Generating ssh key... Wrote public SSH key to /etc/ceph/ceph.pub Adding key to fedora@localhost authorized_keys... Adding host node00... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Enabling mgr prometheus module... Deploying prometheus service with default placement... Deploying grafana service with default placement... Deploying node-exporter service with default placement... Deploying alertmanager service with default placement... Enabling the dashboard module... Waiting for the mgr to restart... Waiting for mgr epoch 13... mgr epoch 13 is available Generating a dashboard self-signed certificate... Creating initial admin user... Fetching dashboard port number... Ceph Dashboard is now available at: URL: https://node00:8443/ User: admin Password: 79mmczlpgc You can access the Ceph CLI with: sudo ./cephadm shell --fsid bfe58fc0-9c71-11eb-893e-52540031b82d -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/docs/master/mgr/telemetry/ Bootstrap complete. [fedora@node00 ~]$ [fedora@node00 ~]$ [fedora@node00 ~]$ sudo ./cephadm shell Inferring fsid bfe58fc0-9c71-11eb-893e-52540031b82d Inferring config /var/lib/ceph/bfe58fc0-9c71-11eb-893e-52540031b82d/mon.node00/config Using recent ceph image quay.io/daniel_pivonka/ceph@sha256:74ad88107efd38f1d336bfb7b6fc48358ef566f72049664b027a573817ef08cb [ceph: root@node00 /]# [ceph: root@node00 /]# [ceph: root@node00 /]# [ceph: root@node00 /]# ceph orch host ls HOST ADDR LABELS STATUS node00 192.168.122.246 [ceph: root@node00 /]# [ceph: root@node00 /]# [ceph: root@node00 /]# exit [fedora@node00 ~]$ [fedora@node00 ~]$ [fedora@node00 ~]$ ssh-copy-id -f -i /etc/ceph/ceph.pub fedora@node01 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/ceph/ceph.pub" The authenticity of host 'node01 (192.168.122.167)' can't be established. ECDSA key fingerprint is SHA256:BVQaq5n6JczginbY4npcTWYBLHq28aUdj2cu4aSnPCg. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes fedora@node01's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'fedora@node01'" and check to make sure that only the key(s) you wanted were added. [fedora@node00 ~]$ [fedora@node00 ~]$ [fedora@node00 ~]$ [fedora@node00 ~]$ sudo ./cephadm shell Inferring fsid bfe58fc0-9c71-11eb-893e-52540031b82d Inferring config /var/lib/ceph/bfe58fc0-9c71-11eb-893e-52540031b82d/mon.node00/config Using recent ceph image quay.io/daniel_pivonka/ceph@sha256:74ad88107efd38f1d336bfb7b6fc48358ef566f72049664b027a573817ef08cb [ceph: root@node00 /]# [ceph: root@node00 /]# [ceph: root@node00 /]# ceph orch host add node01 Error EINVAL: Failed to connect to node01 (node01). Please make sure that the host is reachable and accepts connections using the cephadm SSH key To add the cephadm SSH key to the host: > ceph cephadm get-pub-key > ~/ceph.pub > ssh-copy-id -f -i ~/ceph.pub fedora@node01 To check that the host is reachable: > ceph cephadm get-ssh-config > ssh_config > ceph config-key get mgr/cephadm/ssh_identity_key > ~/cephadm_private_key > chmod 0600 ~/cephadm_private_key > ssh -F ssh_config -i ~/cephadm_private_key fedora@node01 [ceph: root@node00 /]# ceph cephadm get-ssh-config > ssh_config [ceph: root@node00 /]# ceph config-key get mgr/cephadm/ssh_identity_key > ~/cephadm_private_key [ceph: root@node00 /]# chmod 0600 ~/cephadm_private_key [ceph: root@node00 /]# ssh -F ssh_config -i ~/cephadm_private_key fedora@node01 Warning: Permanently added 'node01,192.168.122.167' (ECDSA) to the list of known hosts. [fedora@node01 ~]$ [fedora@node01 ~]$ [fedora@node01 ~]$ # WHY DID THIS WORK BUT THE HOST COULD NOT BE ADDED TO THE CLUSTER [fedora@node01 ~]$ [fedora@node01 ~]$ [fedora@node01 ~]$ logout Connection to node01 closed. [ceph: root@node00 /]# [ceph: root@node00 /]# [ceph: root@node00 /]# [ceph: root@node00 /]# [ceph: root@node00 /]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.122.246 node00 192.168.122.167 node01 192.168.122.30 node02 [ceph: root@node00 /]# [ceph: root@node00 /]# # /ETC/HOSTS IS IN THE SHELL CONTIANER [ceph: root@node00 /]# exit [fedora@node00 ~]$ [fedora@node00 ~]$ sudo podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1f32d184f3e1 docker.io/prom/alertmanager:v0.20.0 --web.listen-addr... About an hour ago Up About an hour ago ceph-bfe58fc0-9c71-11eb-893e-52540031b82d-alertmanager.node00 fead1b7a10ad docker.io/ceph/ceph-grafana:6.7.4 /bin/bash About an hour ago Up About an hour ago ceph-bfe58fc0-9c71-11eb-893e-52540031b82d-grafana.node00 e97603b3e87b docker.io/prom/prometheus:v2.18.1 --config.file=/et... About an hour ago Up About an hour ago ceph-bfe58fc0-9c71-11eb-893e-52540031b82d-prometheus.node00 cf4a39893f10 docker.io/prom/node-exporter:v0.18.1 --no-collector.ti... About an hour ago Up About an hour ago ceph-bfe58fc0-9c71-11eb-893e-52540031b82d-node-exporter.node00 c74d0cbafe7d quay.io/daniel_pivonka/ceph:latest -n client.crash.n... About an hour ago Up About an hour ago ceph-bfe58fc0-9c71-11eb-893e-52540031b82d-crash.node00 9124c0eb4296 quay.io/daniel_pivonka/ceph:latest -n mgr.node00.fkv... About an hour ago Up About an hour ago ceph-bfe58fc0-9c71-11eb-893e-52540031b82d-mgr.node00.fkveyp b01d1ba02f18 quay.io/daniel_pivonka/ceph:latest -n mon.node00 -f ... About an hour ago Up About an hour ago ceph-bfe58fc0-9c71-11eb-893e-52540031b82d-mon.node00 [fedora@node00 ~]$ [fedora@node00 ~]$ [fedora@node00 ~]$ sudo podman exec -it 9124c0eb4296 bash [root@node00 /]# [root@node00 /]# [root@node00 /]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 [root@node00 /]# [root@node00 /]# [root@node00 /]# # /ETC/HOSTS IS NOT IN THE MGR CONTIANER