Bug #54328
openProblems using OSDSpec with DB and WAL on same NVME device
0%
Description
When provisioning a new cluster using Pacific 16.2.7, I find that in spite of using OSDSpec with cephadm for the addition of OSDs, a WAL device is not included and the WAL is instead saved on the data disks.
The data disks are HDDs, so obviously this is not desired.
Here is the output of a dry-run:
----------------------------+----------------------------------+----------+--------------+-----+
|SERVICE |NAME |HOST |DATA |DB |WAL |----------------------------+----------------------------------+----------+--------------+-----+
|osd |default_drive_group |cepho-p000 |/dev/sda |/dev/nvme1n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdb |/dev/nvme1n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdc |/dev/nvme1n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdd |/dev/nvme1n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sde |/dev/nvme1n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdf |/dev/nvme1n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdg |/dev/nvme1n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdh |/dev/nvme1n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdi |/dev/nvme1n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdj |/dev/nvme0n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdk |/dev/nvme0n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdl |/dev/nvme0n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdm |/dev/nvme0n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdn |/dev/nvme0n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdo |/dev/nvme0n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdp |/dev/nvme0n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdq |/dev/nvme0n1 |- |
|osd |default_drive_group |cepho-p000 |/dev/sdr |/dev/nvme0n1 |- |----------------------------+----------------------------------+----------+--------------+-----+
Here is the OSDSpec file I used:
service_type: osd
service_id: default_drive_group
placement:
host_pattern: 'cepho-*'
spec:
data_devices:
rotational: 1
db_devices:
rotational: 0
Based on what it says at https://docs.ceph.com/en/latest/cephadm/services/osd/#the-simple-case I assumed that DB and WAL placement would be on the same non-rotational device.
I tried altering the file to include wal_devices too, but in this case it did not specify a DB device in the preview.
Hence, it appears to be offering either DB or WAL but not both.
Updated by Adam Huffman about 2 years ago
This afternoon I tried a new cluster with these settings from the Dashboard:
service_type: osd
service_id: dashboard-admin-1645460246886
service_name: osd.dashboard-admin-1645460246886
placement:
host_pattern: '*'
spec:
data_devices:
rotational: true
db_devices:
model: SSDPE2KE032T8L
filter_logic: AND
objectstore: bluestore
wal_devices:
model: SSDPE2KE032T8L
following some suggestions on the mailing list that specifying the model instead of the rotational status worked better.
The result was similar - WAL on NVMe, DB on one of the data disks:
bluefs 1 bluefs_dedicated_db 0 bluefs_dedicated_wal 1 bluefs_single_shared_device 0 bluefs_wal_access_mode blk bluefs_wal_block_size 4096 bluefs_wal_dev_node /dev/dm-3 bluefs_wal_devices nvme1n1 bluefs_wal_driver KernelDevice bluefs_wal_partition_path /dev/dm-3 bluefs_wal_rotational 0 bluefs_wal_size 177809129472 bluefs_wal_support_discard 1 bluefs_wal_type ssd bluestore_bdev_access_mode blk bluestore_bdev_block_size 4096 bluestore_bdev_dev_node /dev/dm-2 bluestore_bdev_devices sda bluestore_bdev_driver KernelDevice bluestore_bdev_partition_path /dev/dm-2
Also, here's a friendlier rendering of the earlier dry-run output:
+---------+---------------------+----------------------------------+----------+--------------+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+---------------------+----------------------------------+----------+--------------+-----+ |osd |default_drive_group |cepho-p000 |/dev/sda |/dev/nvme1n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdb |/dev/nvme1n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdc |/dev/nvme1n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdd |/dev/nvme1n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sde |/dev/nvme1n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdf |/dev/nvme1n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdg |/dev/nvme1n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdh |/dev/nvme1n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdi |/dev/nvme1n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdj |/dev/nvme0n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdk |/dev/nvme0n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdl |/dev/nvme0n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdm |/dev/nvme0n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdn |/dev/nvme0n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdo |/dev/nvme0n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdp |/dev/nvme0n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdq |/dev/nvme0n1 |- | |osd |default_drive_group |cepho-p000 |/dev/sdr |/dev/nvme0n1 |- | +---------+---------------------+----------------------------------+----------+--------------+-----+