Ceph » Orchestrator

spider@docker02:~$ sudo ./cephadm version
Using recent ceph image ceph/ceph:v15
ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)
spider@docker02:~$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ceph/ceph           v15                 5553b0cb212c        3 weeks ago         943MB

I modified cephadm so I can poke around and see during this run that the following file is created:

root@docker02:/home/spider# ls -al /tmp/ceph-tmpj1ljfgr1
-rw------- 1 root root 0 Jan  7 18:12 /tmp/ceph-tmpj1ljfgr1
root@docker02:/home/spider# cat /tmp/ceph-tmpj1ljfgr1

When I try to run the the docker command manually I still receive the same error:

root@docker02:/home/spider# /snap/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/monmaptool -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=docker02 -v /tmp/ceph-tmpj1ljfgr1:/tmp/monmap:z docker.io/ceph/ceph:v15 --create --clobber --fsid 5bb7a9ec-5113-11eb-9137-7147df6a1932 --addv docker02 [v2:10.0.0.12:3300,v1:10.0.0.12:6789] /tmp/monmap
/usr/bin/monmaptool: monmap file /tmp/monmap
/usr/bin/monmaptool: set fsid to 5bb7a9ec-5113-11eb-9137-7147df6a1932
/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
bufferlist::write_file(/tmp/monmap): failed to open file: (21) Is a directory
monmaptool: error writing to '/tmp/monmap': (21) Is a directory

Lets run the container again and take a peek inside:

root@docker02:/home/spider# /snap/bin/docker run -it --rm --ipc=host --net=host  -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=docker02 -v /tmp/ceph-tmpj1ljfgr1:/tmp/monmap:z docker.io/ceph/ceph:v15 bash
[root@docker02 /]#
[root@docker02 /]# ls -al /tmp/monmap
total 8
drwxr-xr-x 2 root root 4096 Jan  7 18:15 .
drwxrwxrwt 1 root root 4096 Jan  7 18:16 ..
[root@docker02 /]#

Interesting. The file is being mounted as a directory.

For kicks I tried this on a different VM which is not using snap and it works:

root@docker01:~# touch /tmp/ceph-tmpj1ljfgr1
root@docker01:~# docker run -it --rm --ipc=host --net=host  -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=docker02 -v /tmp/ceph-tmpj1ljfgr1:/tmp/monmap:z docker.io/ceph/ceph:v15 bash
Unable to find image 'ceph/ceph:v15' locally
v15: Pulling from ceph/ceph
7a0437f04f83: Pull complete
ed95fc83a07f: Pull complete
Digest: sha256:37939a3739e4e037dcf1b1f5828058d721d8c6de958212609f9e7d920b9c62bf
Status: Downloaded newer image for ceph/ceph:v15
[root@docker01 /]#
[root@docker01 /]#
[root@docker01 /]# ls /tmp/monmap
/tmp/monmap
[root@docker01 /]# ls -al /tmp/monmap
-rw-r--r-- 1 root root 0 Jan  7 18:21 /tmp/monmap

Back to docker02 where I uninstalled the snap docker page and installed from Docker's repo:

spider@docker02:~$ touch /tmp/ceph-tmpj1ljfgr1
spider@docker02:~$ docker run -it --rm --ipc=host --net=host  -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=docker02 -v /tmp/ceph-tmpj1ljfgr1:/tmp/monmap docker.io/ceph/ceph:v15 bash
Unable to find image 'ceph/ceph:v15' locally
v15: Pulling from ceph/ceph
7a0437f04f83: Pull complete
ed95fc83a07f: Pull complete
Digest: sha256:37939a3739e4e037dcf1b1f5828058d721d8c6de958212609f9e7d920b9c62bf
Status: Downloaded newer image for ceph/ceph:v15
[root@docker02 /]# ls -al /tmp/monmap
-rw-rw-r-- 1 1000 1000 0 Jan  7 18:29 /tmp/monmap
[root@docker02 /]#

And the file is correctly mounted as a file and not a directory!

Tried cephadm bootstrap again after switching to normal docker package:

Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting mon public_network...
Creating mgr...
Verifying port 9283 ...
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Wrote config to /etc/ceph/ceph.conf
Waiting for mgr to start...
Waiting for mgr...

It looks good. I will do more research into why snap docker mounts this incorrectly. Perhaps we can work around this within cephadm.

Some additional color: I would normally install docker from docker's repos but on this VM I chose to use the Ubuntu installer option for docker which used snap.