Ceph » Orchestrator

2021-02-12 15:38:41,511 INFO Applying /home/ceph-admin/specs/ceph_spec.yaml to cluster
2021-02-12 15:38:41,511 INFO Adding ssh key to oc0-ceph-3
2021-02-12 15:38:41,511 DEBUG Running command: ssh-copy-id -f -i /home/ceph-admin/.ssh/id_rsa.pub ceph-admin@oc0-ceph-3
2021-02-12 15:38:41,519 DEBUG ssh-copy-id:stderr /bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ceph-admin/.ssh/id_rsa.pub" 
2021-02-12 15:38:41,531 DEBUG Acquiring lock 139806070650584 on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock
2021-02-12 15:38:41,532 DEBUG Lock 139806070650584 not acquired on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock, waiting 0.05 seconds ...
2021-02-12 15:38:41,582 DEBUG Acquiring lock 139806070650584 on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock
2021-02-12 15:38:41,582 DEBUG Lock 139806070650584 not acquired on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock, waiting 0.05 seconds ...
2021-02-12 15:38:41,632 DEBUG Acquiring lock 139806070650584 on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock
2021-02-12 15:38:41,633 DEBUG Lock 139806070650584 not acquired on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock, waiting 0.05 seconds ...
2021-02-12 15:38:41,683 DEBUG Acquiring lock 139806070650584 on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock
2021-02-12 15:38:41,683 DEBUG Lock 139806070650584 not acquired on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock, waiting 0.05 seconds ...
2021-02-12 15:38:41,733 DEBUG Acquiring lock 139806070650584 on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock
2021-02-12 15:38:41,733 DEBUG Lock 139806070650584 not acquired on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock, waiting 0.05 seconds ...
2021-02-12 15:38:41,783 DEBUG Acquiring lock 139806070650584 on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock
2021-02-12 15:38:41,783 DEBUG Lock 139806070650584 not acquired on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock, waiting 0.05 seconds ...
2021-02-12 15:38:41,834 DEBUG Acquiring lock 139806070650584 on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock
2021-02-12 15:38:41,834 DEBUG Lock 139806070650584 not acquired on /run/cephadm/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135.lock, waiting 0.05 seconds ...
... snip ... 

looks like we're doing too much synchronously here.

I reproduced this with 15.2.10. I'll follow up on IRC to offer the assignee live access to the reproducer.

[root@oc0-ceph-2 ~]# cephadm ls
[
    {
        "style": "cephadm:v1",
        "name": "mon.oc0-ceph-2",
        "fsid": "ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135",
        "systemd_unit": "ceph-ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135@mon.oc0-ceph-2",
        "enabled": true,
        "state": "running",
        "container_id": "fd1caf3c583e5767dff00b9319d053c150f92ed572ac1f7ea3b23b1448784f18",
        "container_image_name": "quay.ceph.io/ceph-ci/daemon:v5.0.9-stable-5.0-octopus-centos-8-x86_64",
        "container_image_id": "1f47579e3c610d981f4dbe27f372fa7cca94a16620152ddad4bd20285c8c95e7",
        "version": "15.2.10",
        "started": "2021-04-08T19:17:26.194598Z",
        "created": "2021-04-08T19:17:23.868279Z",
        "deployed": "2021-04-08T19:17:22.814175Z",
        "configured": "2021-04-08T19:17:23.868279Z" 
    },
    {
        "style": "cephadm:v1",
        "name": "mgr.oc0-ceph-2.mgogfr",
        "fsid": "ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135",
        "systemd_unit": "ceph-ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135@mgr.oc0-ceph-2.mgogfr",
        "enabled": true,
        "state": "running",
        "container_id": "6c597539ccfe12a7324c9eb4f1e0386f69865db392fefbec81dc1d99662d846e",
        "container_image_name": "quay.ceph.io/ceph-ci/daemon:v5.0.9-stable-5.0-octopus-centos-8-x86_64",
        "container_image_id": "1f47579e3c610d981f4dbe27f372fa7cca94a16620152ddad4bd20285c8c95e7",
        "version": "15.2.10",
        "started": "2021-04-08T19:17:27.372219Z",
        "created": "2021-04-08T19:17:27.536640Z",
        "deployed": "2021-04-08T19:17:26.869575Z",
        "configured": "2021-04-08T19:17:27.536640Z" 
    }
]
[root@oc0-ceph-2 ~]# rpm -q cephadm
cephadm-15.2.10-4.el8.noarch
[root@oc0-ceph-2 ~]# cat /etc/redhat-release 
CentOS Stream release 8
[root@oc0-ceph-2 ~]# uname -a
Linux oc0-ceph-2 4.18.0-294.el8.x86_64 #1 SMP Mon Mar 15 22:38:42 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@oc0-ceph-2 ~]# podman --version
podman version 3.0.2-dev
[root@oc0-ceph-2 ~]# 
<pre>

We think this is a duplicate of https://tracker.ceph.com/issues/50041 and fixed by https://github.com/ceph/ceph/pull/40477/files

I reproduced the problem by running this:

sudo /usr/sbin/cephadm bootstrap \
  --ssh-private-key /home/ceph-admin/.ssh/id_rsa \
  --ssh-public-key /home/ceph-admin/.ssh/id_rsa.pub \
  --ssh-user ceph-admin \
  --apply-spec /home/ceph-admin/specs/ceph_spec.yaml \
  ...

Next steps:
1. confirm I cannot reproduce 49277 with pacific
2. close 49277 as a duplicate of 50041

When using pacific with the fixing patch https://github.com/ceph/ceph/pull/40477 from Bug #50041, the deployment fails better (it doesn't hang waiting for me to accept the ssh key) but it still fails [1]. Similar underlying root cause though: ceph-admin's SSH keys were set up correctly but because of sudo, root was trying to SSH and root's keys were not distributed. Thus, it tried to distribute root's keys as root and failed.

[ceph-admin@oc0-ceph-2 ~]$ sudo ssh oc0-ceph-3
The authenticity of host 'oc0-ceph-3 (192.168.24.14)' can't be established.
ECDSA key fingerprint is SHA256:VDnoF5dEU7gmwmT9RV6eJg/I1Hby0GhRN7VFCs6fZ0Q.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'oc0-ceph-3,192.168.24.14' (ECDSA) to the list of known hosts.
root@oc0-ceph-3: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
[ceph-admin@oc0-ceph-2 ~]$ 

I undersatnd cephadm defaulting to root for the SSH user and root SSH keys, but maybe cephadm needs to look at these parameters:

  --ssh-private-key /home/ceph-admin/.ssh/id_rsa \
  --ssh-public-key /home/ceph-admin/.ssh/id_rsa.pub \
  --ssh-user ceph-admin \

and if they are set, then it should use the above for the SSH connection instead. I.e. let the above override the default?

[1]

[ceph-admin@oc0-ceph-2 ~]$ sudo /usr/sbin/cephadm --image quay.ceph.io/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64 bootstrap --skip-firewalld --ssh-private-key /home/ceph-admin/.ssh/id_rsa --ssh-public-key /home/ceph-admin/.ssh/id_rsa.pub --ssh-user ceph-admin --allow-fqdn-hostname --output-keyring /etc/ceph/ceph.client.admin.keyring --output-config /etc/ceph/ceph.conf --fsid ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135 --apply-spec /home/ceph-admin/specs/ceph_spec.yaml --config /home/ceph-admin/bootstrap_ceph.conf --skip-monitoring-stack --skip-dashboard --mon-ip 192.168.24.22                                            
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman|docker (/bin/podman) is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135
Verifying IP 192.168.24.22 port 3300 ...
Verifying IP 192.168.24.22 port 6789 ...
Mon IP 192.168.24.22 is in CIDR network 192.168.24.0/24
- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Pulling container image quay.ceph.io/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64...
Ceph version: ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting mon public_network to 192.168.24.0/24
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 9283 ...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Using provided ssh keys...
Adding host oc0-ceph-2...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Applying /home/ceph-admin/specs/ceph_spec.yaml to cluster
Adding ssh key to oc0-ceph-3
Adding ssh key to oc0-ceph-4
Non-zero exit code 22 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64 -e NODE_NAME=oc0-ceph-2 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135:/var/log/ceph:z -v /tmp/ceph-tmpbfayxvwp:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjm_n51dx:/etc/ceph/ceph.conf:z -v /home/ceph-admin/specs/ceph_spec.yaml:/tmp/spec.yml:z quay.ceph.io/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64 orch apply -i /tmp/spec.yml                                            
/usr/bin/ceph: stderr Error EINVAL: Failed to connect to oc0-ceph-3 (oc0-ceph-3).
/usr/bin/ceph: stderr Please make sure that the host is reachable and accepts connections using the cephadm SSH key
/usr/bin/ceph: stderr
/usr/bin/ceph: stderr To add the cephadm SSH key to the host:
/usr/bin/ceph: stderr > ceph cephadm get-pub-key > ~/ceph.pub
/usr/bin/ceph: stderr > ssh-copy-id -f -i ~/ceph.pub ceph-admin@oc0-ceph-3
/usr/bin/ceph: stderr
/usr/bin/ceph: stderr To check that the host is reachable:
/usr/bin/ceph: stderr > ceph cephadm get-ssh-config > ssh_config
/usr/bin/ceph: stderr > ceph config-key get mgr/cephadm/ssh_identity_key > ~/cephadm_private_key
/usr/bin/ceph: stderr > chmod 0600 ~/cephadm_private_key
/usr/bin/ceph: stderr > ssh -F ssh_config -i ~/cephadm_private_key ceph-admin@oc0-ceph-3
Traceback (most recent call last):
  File "/usr/sbin/cephadm", line 7924, in <module>
    main()
  File "/usr/sbin/cephadm", line 7912, in main
    r = ctx.func(ctx)
  File "/usr/sbin/cephadm", line 1717, in _default_image
    return func(ctx)
  File "/usr/sbin/cephadm", line 4037, in command_bootstrap
    out = cli(['orch', 'apply', '-i', '/tmp/spec.yml'], extra_mounts=mounts)
  File "/usr/sbin/cephadm", line 3931, in cli
    ).run(timeout=timeout)
  File "/usr/sbin/cephadm", line 3174, in run
    desc=self.entrypoint, timeout=timeout)
  File "/usr/sbin/cephadm", line 1411, in call_throws
    raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64 -e NODE_NAME=oc0-ceph-2 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/ca9bf37b-ed0f-4e5a-bb21-e5b5f9b75135:/var/log/ceph:z -v /tmp/ceph-tmpbfayxvwp:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjm_n51dx:/etc/ceph/ceph.conf:z -v /home/ceph-admin/specs/ceph_spec.yaml:/tmp/spec.yml:z quay.ceph.io/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64 orch apply -i /tmp/spec.yml                                         
[ceph-admin@oc0-ceph-2 ~]$ 

Because I'm having the same issue in Pacific let's use a new bug https://tracker.ceph.com/issues/50306