Project

General

Profile

Bug #51028

device zap doesn't perform any checks

Added by Paul Cuzner 19 days ago. Updated 18 days ago.

Status:
New
Priority:
Normal
Assignee:
Category:
orchestrator
Target version:
% Done:

0%

Source:
Tags:
ux
Backport:
pacific
Regression:
No
Severity:
2 - major
Reviewed:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

An orch device zap <host> <dev> doesn't perform any validation checking and can expose the admin to a situation where the wrong device is zapped removing an active OSD from the cluster.

I would suggest that to remediate;
- orchestrator should validate the hostname exists and it's status is empty (i.e. host is available)
- cephadm should check the device is not part of an active osd

History

#1 Updated by Sebastian Wagner 18 days ago

  • Tags set to ux

#2 Updated by Paul Cuzner 18 days ago

Just some more info to help understand the scenario, I'm going to mess with osd.7 which is device vdd on node-01

[ceph: root@ceph-node-00 /]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 9.00000 root default
-3 3.00000 host ceph-node-00
-9 0 chassis ceph-node-00-fast
0 hdd 1.00000 osd.0 up 1.00000 1.00000
3 hdd 1.00000 osd.3 up 1.00000 1.00000
6 hdd 1.00000 osd.6 up 1.00000 1.00000
-7 3.00000 host ceph-node-01
-11 0 chassis ceph-node-01-fast
1 hdd 1.00000 osd.1 up 1.00000 1.00000
4 hdd 1.00000 osd.4 up 1.00000 1.00000
7 ssd 1.00000 osd.7 up 1.00000 1.00000
-5 3.00000 host ceph-node-02
-12 1.00000 chassis ceph-node-02-fast
8 ssd 1.00000 osd.8 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
[ceph: root@ceph-node-00 /]# ceph orch device zap ceph-node-01.mylab.com /dev/vdd --force
Error EINVAL: Zap failed: /bin/podman: --> Zapping: /dev/vdd
/bin/podman: --> Zapping lvm member /dev/vdd. lv_path is /dev/ceph-515bbdf9-6a83-4e2c-b4db-53f333382df4/osd-block-9321a37a-bb2f-4033-a3b2-0a259b92ff50
/bin/podman: Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-515bbdf9-6a83-4e2c-b4db-53f333382df4/osd-block-9321a37a-bb2f-4033-a3b2-0a259b92ff50 bs=1M count=10 conv=fsync
/bin/podman: stderr: 10+0 records in
/bin/podman: 10+0 records out
/bin/podman: 10485760 bytes (10 MB, 10 MiB) copied, 0.0993142 s, 106 MB/s
/bin/podman: --> Only 1 LV left in VG, will proceed to destroy volume group ceph-515bbdf9-6a83-4e2c-b4db-53f333382df4
/bin/podman: Running command: /usr/sbin/vgremove -v -f ceph-515bbdf9-6a83-4e2c-b4db-53f333382df4
/bin/podman: stderr: Logical volume ceph-515bbdf9-6a83-4e2c-b4db-53f333382df4/osd-block-9321a37a-bb2f-4033-a3b2-0a259b92ff50 in use.
/bin/podman: --> Unable to remove vg ceph-515bbdf9-6a83-4e2c-b4db-53f333382df4
/bin/podman: --> RuntimeError: command returned non-zero exit status: 5
Traceback (most recent call last):
File "/var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/cephadm.708de480567bdfe2a5c94c6dff5aed01f37d0ef82a37fd29a96a6c9669beb2e3", line 8186, in <module>
main()
File "/var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/cephadm.708de480567bdfe2a5c94c6dff5aed01f37d0ef82a37fd29a96a6c9669beb2e3", line 8174, in main
r = ctx.func(ctx)
File "/var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/cephadm.708de480567bdfe2a5c94c6dff5aed01f37d0ef82a37fd29a96a6c9669beb2e3", line 1654, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/cephadm.708de480567bdfe2a5c94c6dff5aed01f37d0ef82a37fd29a96a6c9669beb2e3", line 1738, in _infer_image
return func(ctx)
File "/var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/cephadm.708de480567bdfe2a5c94c6dff5aed01f37d0ef82a37fd29a96a6c9669beb2e3", line 4576, in command_ceph_volume
out, err, code = call_throws(ctx, c.run_cmd(), verbosity=verbosity)
File "/var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/cephadm.708de480567bdfe2a5c94c6dff5aed01f37d0ef82a37fd29a96a6c9669beb2e3", line 1464, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:0560ece981b19c7cb36223fec3cb61b114f5f9ebcf0652e69ba1821774afcaea -e NODE_NAME=ceph-node-01.mylab.com -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5:/var/run/ceph:z -v /var/log/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5:/var/log/ceph:z -v /var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61b9fb8c-ad26-11eb-9cd6-5254005a31f5/selinux:/sys/fs/selinux:ro quay.ceph.io/ceph-ci/ceph@sha256:0560ece981b19c7cb36223fec3cb61b114f5f9ebcf0652e69ba1821774afcaea lvm zap --destroy /dev/vdd

then when the osd is restarted (at some future time)
[root@ceph-node-01 ~]# systemctl restart
Job for failed because the control process exited with error code.
See "systemctl status " and "journalctl -xe" for details.

To me this looks like the fact that dd works, then the vgremove fails is the problem...but the root cause is the orch CLI should never have allowed the command to reach ceph-volume, imo.

The net result is we lose one or more OSDs(shared SSD/NVME for example), from a potential fat finger moment.

[ceph: root@ceph-node-00 /]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 9.00000 root default
-3 3.00000 host ceph-node-00
-9 0 chassis ceph-node-00-fast
0 hdd 1.00000 osd.0 up 1.00000 1.00000
3 hdd 1.00000 osd.3 up 1.00000 1.00000
6 hdd 1.00000 osd.6 up 1.00000 1.00000
-7 3.00000 host ceph-node-01
-11 0 chassis ceph-node-01-fast
1 hdd 1.00000 osd.1 up 1.00000 1.00000
4 hdd 1.00000 osd.4 up 1.00000 1.00000
7 ssd 1.00000 osd.7 down 0 1.00000
-5 3.00000 host ceph-node-02
-12 1.00000 chassis ceph-node-02-fast
8 ssd 1.00000 osd.8 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000

If you agree that this is a problem, I can provide a fix. However, if this is working as intended - please feel free to close the report.

Also available in: Atom PDF