Ceph

I reinstalled the operating system of my host. I then wanted to use `ceph cephadm osd activate` for bringing the osds in the host back into my cluster.

The `ceph cephadm osd activate` only brought back the crash and the node-exporter docker containers.

- Mailing List Thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ALGZCKU7B4NUUBCE5NDDBANTONFWWDDR/

Following hints on the mailing list, I called `cephadm -v -v -v ceph-volume lvm activate --all --no-systemd`. This did not create any folders in `/var/lib/ceph/FSID` so `systemctl start ceph-d221bc3c-8ff4-11ec-b4ba-b02628267680@osd.12` fails with `/bin/bash: /var/lib/ceph/d221bc3c-8ff4-11ec-b4ba-b02628267680/osd.12/unit.run: No such file or directory`.

Output of cephadm ceph-volume lvm activate --all --nosystemd is below.

# cephadm -v -v -v ceph-volume lvm activate --all --no-systemd 
--------------------------------------------------------------------------------
cephadm ['-v', '-v', '-v', 'ceph-volume', 'lvm', 'activate', '--all', '--no-systemd']
Using default config: /etc/ceph/ceph.conf
/bin/docker: 920d72f71171,79MiB / 251.5GiB
/bin/docker: 72570e59f28e,77.43MiB / 251.5GiB
/bin/docker: b1a0de17c02d,854.8MiB / 251.5GiB
/bin/docker: cce740906d36,1.158GiB / 251.5GiB
/bin/docker: 4fba9fb86122,209.3MiB / 251.5GiB
/bin/docker: 9b06ce4f696b,96.83MiB / 251.5GiB
/bin/docker: 35269d267dfc,91.14MiB / 251.5GiB
/bin/docker: 54a2a92d68fa,278.1MiB / 251.5GiB
/bin/docker: cee7af4d922d,871.7MiB / 251.5GiB
/bin/docker: f821b8d5e473,627.6MiB / 251.5GiB
/bin/docker: 805061ed9471,473.3MiB / 251.5GiB
/bin/docker: 52e339e96d47,451.7MiB / 251.5GiB
/bin/docker: 542bda96ee76,409.9MiB / 251.5GiB
/bin/docker: 1d76652acb43,192MiB / 251.5GiB
/bin/docker: 0934566689b5,120.9MiB / 251.5GiB
/bin/docker: 17c036fbb15d,285MiB / 251.5GiB
/bin/docker: 63d0925cfc81,1.647GiB / 251.5GiB
/bin/docker: 06e96a3e5cbd,572.6MiB / 251.5GiB
/bin/docker: 2d2fc47ec89a,19.6MiB / 251.5GiB
/bin/docker: 26adae09c237,233.6MiB / 251.5GiB
/bin/docker: 86cf7b732b13,90.75MiB / 251.5GiB
/bin/docker: 92d366bd19aa,2.008MiB / 251.5GiB
/bin/docker: 01e9dc38faab,208.1MiB / 251.5GiB
/bin/docker: da22e3f7811c,655MiB / 251.5GiB
/bin/docker: d88137996826,783.1MiB / 251.5GiB
/bin/docker: a9f1e48f4a04,585.5MiB / 251.5GiB
/bin/docker: 49742307a5b4,478.1MiB / 251.5GiB
/bin/docker: 2bb7d623ed93,205.7MiB / 251.5GiB
/bin/docker: 0b323f254217,243.5MiB / 251.5GiB
/bin/docker: 8835104b0515,123.5MiB / 251.5GiB
/bin/docker: 67472592f1b4,658.9MiB / 251.5GiB
/bin/docker: 5aef3941d33b,474.4MiB / 251.5GiB
/bin/docker: 0d571f24c0e9,100.1MiB / 251.5GiB
/bin/docker: 7e706a832e4a,2.11GiB / 251.5GiB
/bin/docker: 23a669b4fd7c,1.164GiB / 251.5GiB
/bin/docker: 74581172b6fb,1.902MiB / 251.5GiB
/bin/docker: 37e980f9f83d,2.273MiB / 251.5GiB
/bin/docker: 858a9b82375d,179.5MiB / 251.5GiB
/bin/docker: dc97abdfc393,16.78MiB / 251.5GiB
/bin/docker: d561d2aa6053,5.488MiB / 251.5GiB
/bin/docker: 804b450b057f,655.5MiB / 251.5GiB
/bin/docker: 011180b66a7a,2.621MiB / 251.5GiB
/bin/docker: 21b984cdc711,56.94MiB / 251.5GiB
/bin/docker: c197971ee2ba,1.754MiB / 251.5GiB
/bin/docker: a10112ce1fa7,1.887MiB / 251.5GiB
/bin/docker: 546536f82ada,160.5MiB / 251.5GiB
/bin/docker: 5822da42e73b,38.15MiB / 251.5GiB
/bin/docker: 38e044b7d42f,6.645MiB / 251.5GiB
Inferring fsid d221bc3c-8ff4-11ec-b4ba-b02628267680
/bin/docker: quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb
/bin/docker: quay.io/ceph/ceph@
Using recent ceph image quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb
stat: 167 167
Acquiring lock 140181136017056 on /run/cephadm/d221bc3c-8ff4-11ec-b4ba-b02628267680.lock
Lock 140181136017056 acquired on /run/cephadm/d221bc3c-8ff4-11ec-b4ba-b02628267680.lock
sestatus: SELinux status:                 disabled
sestatus: SELinux status:                 disabled
/bin/docker: --> Activating OSD ID 12 FSID e2ebb627-28aa-45a3-9261-d7c27bc08448
/bin/docker: Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-12
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12
/bin/docker: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448 --path /var/lib/ceph/osd/ceph-12 --no-mon-config
/bin/docker: Running command: /usr/bin/ln -snf /dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448 /var/lib/ceph/osd/ceph-12/block
/bin/docker: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-12/block
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12
/bin/docker: --> ceph-volume lvm activate successful for osd ID: 12
/bin/docker: --> Activating OSD ID 25 FSID 3f3d61f8-6964-4922-98cb-6620aff5cb6f
/bin/docker: Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-25
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25
/bin/docker: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-6f69c24e-2930-48ff-a18f-278470e558e1/osd-block-3f3d61f8-6964-4922-98cb-6620aff5cb6f --path /var/lib/ceph/osd/ceph-25 --no-mon-config
/bin/docker: Running command: /usr/bin/ln -snf /dev/ceph-6f69c24e-2930-48ff-a18f-278470e558e1/osd-block-3f3d61f8-6964-4922-98cb-6620aff5cb6f /var/lib/ceph/osd/ceph-25/block
/bin/docker: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-25/block
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25
/bin/docker: --> ceph-volume lvm activate successful for osd ID: 25
/bin/docker: --> Activating OSD ID 0 FSID 8d0b1bad-069a-4acf-b13b-982fab58f285
/bin/docker: Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
/bin/docker: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-8d9bd6eb-ce97-4940-8fed-f57f7bed3f5a/osd-block-8d0b1bad-069a-4acf-b13b-982fab58f285 --path /var/lib/ceph/osd/ceph-0 --no-mon-config
/bin/docker: Running command: /usr/bin/ln -snf /dev/ceph-8d9bd6eb-ce97-4940-8fed-f57f7bed3f5a/osd-block-8d0b1bad-069a-4acf-b13b-982fab58f285 /var/lib/ceph/osd/ceph-0/block
/bin/docker: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
/bin/docker: --> ceph-volume lvm activate successful for osd ID: 0
/bin/docker: --> Activating OSD ID 8 FSID ff82d0d0-6d55-49c2-85cc-bb8a0a74ae89
/bin/docker: Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-8
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8
/bin/docker: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-a8c3f39b-ec8d-4dd5-a85a-13a5773b99fa/osd-block-ff82d0d0-6d55-49c2-85cc-bb8a0a74ae89 --path /var/lib/ceph/osd/ceph-8 --no-mon-config
/bin/docker: Running command: /usr/bin/ln -snf /dev/ceph-a8c3f39b-ec8d-4dd5-a85a-13a5773b99fa/osd-block-ff82d0d0-6d55-49c2-85cc-bb8a0a74ae89 /var/lib/ceph/osd/ceph-8/block
/bin/docker: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-8/block
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8
/bin/docker: --> ceph-volume lvm activate successful for osd ID: 8
/bin/docker: --> Activating OSD ID 4 FSID 313160a8-594d-4384-9640-68c4d8c1b6da
/bin/docker: Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-4
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4
/bin/docker: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-bf94d5a5-eb04-42e3-867a-dee2886daa62/osd-block-313160a8-594d-4384-9640-68c4d8c1b6da --path /var/lib/ceph/osd/ceph-4 --no-mon-config
/bin/docker: Running command: /usr/bin/ln -snf /dev/ceph-bf94d5a5-eb04-42e3-867a-dee2886daa62/osd-block-313160a8-594d-4384-9640-68c4d8c1b6da /var/lib/ceph/osd/ceph-4/block
/bin/docker: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4
/bin/docker: --> ceph-volume lvm activate successful for osd ID: 4
/bin/docker: --> Activating OSD ID 20 FSID f7e67343-4fde-4e45-bc70-f44c92a178bd
/bin/docker: Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-20
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-20
/bin/docker: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-ee2517dc-ecb6-4c14-ab22-59d9c54f0952/osd-block-f7e67343-4fde-4e45-bc70-f44c92a178bd --path /var/lib/ceph/osd/ceph-20 --no-mon-config
/bin/docker: Running command: /usr/bin/ln -snf /dev/ceph-ee2517dc-ecb6-4c14-ab22-59d9c54f0952/osd-block-f7e67343-4fde-4e45-bc70-f44c92a178bd /var/lib/ceph/osd/ceph-20/block
/bin/docker: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-20/block
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5
/bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-20
/bin/docker: --> ceph-volume lvm activate successful for osd ID: 20