Ceph » bluestore

Hi, there

I'm trying to create spdk backend osd, but it cant' success. My operations like this:

1. build ceph 18.2.1 with -DWITH_SPDK=ON option
2. config spdk env by running

# sudo spdk/scripts/setup.sh
0000:00:0e.0 (80ee 4e56): nvme -> uio_pci_generic

3. my ceph.conf which like this:

[global]
plugin dir = lib
erasure code dir = lib

fsid = dca4b307-dfbf-4bd5-9358-028c6ad23b85
mon_initial_members = localhost
mon_host = 127.0.0.1

auth_cluster_required = none
auth service_required = none
auth_client_required = none
cephx_sign_messages = false
auth_supported = none

debug_bluestore = 20/20

[osd]
bluestore = true
#bluestore_block_db_path = "" 
#bluestore_block_db_size = 0
#bluestore_block_wal_path = "" 
#bluestore_block_wal_size = 0
#bluestore_block_size = 0
#bluestore_allocator=bitmap
#bluefs_allocator=bitmap
#osd_memory_target = 10737418240
#bluestore_cache_size = 7516192768

[osd.0]
host = local
osd data = /var/lib/ceph/osd/ceph-0
bluestore_block_path = spdk:trtype:PCIe traddr:0000:00:0e.0

4. create osd by running the command

$ sudo ./bin/ceph-osd -i 0 --mkfs --osd-uuid dca4b307-dfbf-4bd5-9358-028c6ad23b85 -c ceph.conf --no-mon-config

Starting SPDK v20.07 git sha1 1a527e501 / DPDK 20.05.0 initialization...
[ DPDK EAL parameters: nvme-device-manager --no-shconf -c 0x1 -m 512 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1474 ]
EAL: VFIO support initialized
EAL: No legacy callbacks, legacy socket not created
EAL:   Invalid NUMA socket, default to 0
2024-03-05T10:21:57.829+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:57.845+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:57.849+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:57.857+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:57.861+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:57.873+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:57.885+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:57.885+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:57.885+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.013+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.013+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.013+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.013+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.013+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.013+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.185+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.189+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.189+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.189+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.269+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.269+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.269+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.269+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.269+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.269+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.417+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.421+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.433+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.437+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.445+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]
2024-03-05T10:21:58.457+0800 7f04f8d75b40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label at offset 0: End of buffer [buffer:2]

After execute the command, /var/lib/ceph/osd/ceph-0 looks like this

# ll /var/lib/ceph/osd/ceph-0/
total 72
drwxr-xr-x 2 root root 4096 Mar  5 10:21  ./
drwxr-xr-x 3 root root 4096 Mar  1 10:16  ../
-rw------- 1 root root    8 Mar  5 10:21  bfm_blocks
-rw------- 1 root root    4 Mar  5 10:21  bfm_blocks_per_key
-rw------- 1 root root    5 Mar  5 10:21  bfm_bytes_per_block
-rw------- 1 root root   12 Mar  5 10:21  bfm_size
lrwxrwxrwx 1 root root   36 Mar  5 10:21  block -> 'spdk:trtype:PCIe traddr:0000:00:0e.0'
-rw------- 1 root root    2 Mar  5 10:21  bluefs
-rw------- 1 root root   37 Mar  5 10:21  ceph_fsid
-rw------- 1 root root   92 Mar  5 10:21  ceph_version_when_created
-rw------- 1 root root   28 Mar  5 10:21  created_at
-rw-r--r-- 1 root root   37 Mar  5 10:21  fsid
-rw------- 1 root root    8 Mar  5 10:21  kv_backend
-rw------- 1 root root   21 Mar  5 10:21  magic
-rw------- 1 root root    4 Mar  5 10:21  mkfs_done
-rw------- 1 root root    6 Mar  5 10:21  ready
-rw-r--r-- 1 root root   31 Mar  5 10:21 'spdk:trtype:PCIe traddr:0000:00:0e.0'
-rw------- 1 root root   10 Mar  5 10:21  type
-rw------- 1 root root    2 Mar  5 10:21  whoami

After debug code, I found the issue should be _write_bdev_label and _read_bdev_label don't execute on the right device. As there is no detail spdk configuration material, did I miss anything?

Thanks

AFAIK SPDK backend for BlueStore isn't properly maintained and I've never seen it in production so I doubt it's functioning.

Thanks @Igor Gajowiak Fedotov your reply.

I figured it out. The errors could be ignored as spdk backend device doesn't support bdev label. When we call _read_bdev_label, the errors are printed. You can find the code in NVMEDevice.h

bool supported_bdev_label() override { return false; } 

The one who want to try to use spdk backend osd could follow the steps above. At last, I started the osd process by nohup in my environment as ceph-osd.service can't get started.