Ceph

I am testing bluestore with a PMEM device. I use ceph-deploy create to create the bluestore OSD, but as a result of ceph-deploy's device partitioning, the block device size is not an even multiple of 4096:

pmem17 259:5 0 31138512896 0 disk
├─pmem17p1 259:7 0 104857600 0 part /var/lib/ceph/tmp/mnt.LS8m_Q
└─pmem17p2 259:8 0 31032589824 0 part

Code in routine create() in BitmapFreelistManager.cc assumes that all bluestore devices will be a 4096-byte multiple, and so it asserts like this:

2016-07-06 22:22:23.823557 7f9f9626c800 -1 ** Caught signal (Aborted) *
in thread 7f9f9626c800 thread_name:ceph-osd

ceph version 11.0.0-196-g85bb43e (85bb43e111692989d2296a389ce45377d2297d6f)
 1: (()+0x980a0a) [0x7f9f95bd3a0a]
 2: (()+0xf100) [0x7f9f93b9c100]
 3: (gsignal()+0x37) [0x7f9f91ceb5f7]
 4: (abort()+0x148) [0x7f9f91cecce8]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x267) [0x7f9f95ceafe7]
 6: (BitmapFreelistManager::_xor(unsigned long, unsigned long, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x1065) [0x7f9f959ecf75]
 7: (BitmapFreelistManager::create(unsigned long, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x110) [0x7f9f959ef0c0]
 8: (BlueStore::_open_fm(bool)+0x2ba) [0x7f9f958e741a]
 9: (BlueStore::mkfs()+0x8a2) [0x7f9f959091e2]
 10: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x274) [0x7f9f955f2a84]
 11: (main()+0x10b5) [0x7f9f95577185]
 12: (__libc_start_main()+0xf5) [0x7f9f91cd7b15]
 13: (()+0x37e3e9) [0x7f9f955d13e9]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

The fix is straightforward. create() already checks to see if the block device is an even multiple of (bytes_per_block * blocks_per_key), and it updates the bitmap to make any excess bytes at the top of the device appear to be in use. The code just needs to be adjusted to include any partial block in the byte range that it marks as being in use. This patch works for me:

diff --git a/src/os/bluestore/BitmapFreelistManager.cc b/src/os/bluestore/Bitmap
index f652a40..f3055bf 100644
--- a/src/os/bluestore/BitmapFreelistManager.cc
+++ b/src/os/bluestore/BitmapFreelistManager.cc
@ -63,13 +63,14 @ int BitmapFreelistManager::create(uint64_t new_size, KeyValu
_init_misc();

blocks = size / bytes_per_block;
-  if (blocks / blocks_per_key * blocks_per_key != blocks) {
+  if (size / (blocks * blocks_per_key) * (blocks * blocks_per_key) != size) {
     blocks = (blocks / blocks_per_key + 1) * blocks_per_key;
     dout(10) << func << " rounding blocks up from 0x" << std::hex << size
             << " to 0x" << (blocks * bytes_per_block)
             << " (0x" << blocks << " blocks)" << std::dec << dendl;
     // set past-eof blocks as allocated
-    _xor(size, blocks * bytes_per_block - size, txn);
+    uint64_t min_sz = size / bytes_per_block * bytes_per_block;
+    _xor(min_sz, blocks * bytes_per_block - min_sz, txn);
   }
   dout(10) << func
           << " size 0x" << std::hex << size