Ceph » Orchestrator

Just some more info to help understand the scenario, I'm going to mess with osd.7 which is device vdd on node-01

[ceph: root@ceph-node-00 /]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
1 9.00000 root default
-3 3.00000 host ceph-node-00
-9 0 chassis ceph-node-00-fast
0 hdd 1.00000 osd.0 up 1.00000 1.00000
3 hdd 1.00000 osd.3 up 1.00000 1.00000
6 hdd 1.00000 osd.6 up 1.00000 1.00000
-7 3.00000 host ceph-node-01
-11 0 chassis ceph-node-01-fast
1 hdd 1.00000 osd.1 up 1.00000 1.00000
4 hdd 1.00000 osd.4 up 1.00000 1.00000
7 ssd 1.00000 osd.7 up 1.00000 1.00000
-5 3.00000 host ceph-node-02
-12 1.00000 chassis ceph-node-02-fast
8 ssd 1.00000 osd.8 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
[ceph: root@ceph-node-00 /]# ceph orch device zap ceph-node-01.mylab.com /dev/vdd --force
Error EINVAL: Zap failed: /bin/podman: -> Zapping: /dev/vdd
/bin/podman: --> Zapping lvm member /dev/vdd. lv_path is /dev/ceph-515bbdf9-6a83-4e2c-b4db-53f333382df4/osd-block-9321a37a-bb2f-4033-a3b2-0a259b92ff50
/bin/podman: Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-515bbdf9-6a83-4e2c-b4db-53f333382df4/osd-block-9321a37a-bb2f-4033-a3b2-0a259b92ff50 bs=1M count=10 conv=fsync
/bin/podman: stderr: 10+0 records in
/bin/podman: 10+0 records out
/bin/podman: 10485760 bytes (10 MB, 10 MiB) copied, 0.0993142 s, 106 MB/s
/bin/podman: --> Only 1 LV left in VG, will proceed to destroy volume group ceph-515bbdf9-6a83-4e2c-b4db-53f333382df4
/bin/podman: Running command: /usr/sbin/vgremove v -f ceph-515bbdf9-6a83-4e2c-b4db-53f333382df4
/bin/podman: stderr: Logical volume ceph-515bbdf9-6a83-4e2c-b4db-53f333382df4/osd-block-9321a37a-bb2f-4033-a3b2-0a259b92ff50 in use.
/bin/podman: -> Unable to remove vg ceph-515bbdf9-6a83-4e2c-b4db-53f333382df4
/bin/podman: --> RuntimeError: command returned non-zero exit status: 5
Traceback (most recent call last):
File "/var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/cephadm.708de480567bdfe2a5c94c6dff5aed01f37d0ef82a37fd29a96a6c9669beb2e3", line 8186, in <module>
main()
File "/var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/cephadm.708de480567bdfe2a5c94c6dff5aed01f37d0ef82a37fd29a96a6c9669beb2e3", line 8174, in main
r = ctx.func(ctx)
File "/var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/cephadm.708de480567bdfe2a5c94c6dff5aed01f37d0ef82a37fd29a96a6c9669beb2e3", line 1654, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/cephadm.708de480567bdfe2a5c94c6dff5aed01f37d0ef82a37fd29a96a6c9669beb2e3", line 1738, in _infer_image
return func(ctx)
File "/var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/cephadm.708de480567bdfe2a5c94c6dff5aed01f37d0ef82a37fd29a96a6c9669beb2e3", line 4576, in command_ceph_volume
out, err, code = call_throws(ctx, c.run_cmd(), verbosity=verbosity)
File "/var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/cephadm.708de480567bdfe2a5c94c6dff5aed01f37d0ef82a37fd29a96a6c9669beb2e3", line 1464, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:0560ece981b19c7cb36223fec3cb61b114f5f9ebcf0652e69ba1821774afcaea -e NODE_NAME=ceph-node-01.mylab.com -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5:/var/run/ceph:z -v /var/log/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5:/var/log/ceph:z -v /var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/selinux:/sys/fs/selinux:ro quay.ceph.io/ceph-ci/ceph@sha256:0560ece981b19c7cb36223fec3cb61b114f5f9ebcf0652e69ba1821774afcaea lvm zap --destroy /dev/vdd

then when the osd is restarted (at some future time)
[root@ceph-node-01 ~]# systemctl restart ceph-61b9fb8c-ad26-11eb-9cd6-5254005a31f5@osd.7.service
Job for ceph-61b9fb8c-ad26-11eb-9cd6-5254005a31f5@osd.7.service failed because the control process exited with error code.
See "systemctl status ceph-61b9fb8c-ad26-11eb-9cd6-5254005a31f5@osd.7.service" and "journalctl -xe" for details.

To me this looks like the fact that dd works, then the vgremove fails is the problem...but the root cause is the orch CLI should never have allowed the command to reach ceph-volume, imo.

The net result is we lose one or more OSDs(shared SSD/NVME for example), from a potential fat finger moment.

[ceph: root@ceph-node-00 /]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 9.00000 root default
-3 3.00000 host ceph-node-00
-9 0 chassis ceph-node-00-fast
0 hdd 1.00000 osd.0 up 1.00000 1.00000
3 hdd 1.00000 osd.3 up 1.00000 1.00000
6 hdd 1.00000 osd.6 up 1.00000 1.00000
-7 3.00000 host ceph-node-01
-11 0 chassis ceph-node-01-fast
1 hdd 1.00000 osd.1 up 1.00000 1.00000
4 hdd 1.00000 osd.4 up 1.00000 1.00000
7 ssd 1.00000 osd.7 down 0 1.00000
-5 3.00000 host ceph-node-02
-12 1.00000 chassis ceph-node-02-fast
8 ssd 1.00000 osd.8 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000

If you agree that this is a problem, I can provide a fix. However, if this is working as intended - please feel free to close the report.