Bug #22435
fail to create bluestore osd with ceph-volume command on ubuntu 14.04 with ceph 12.2.2
0%
Description
Tried to create a bluestore osd manually with ceph-volume tool on a Ubuntu 14.04 system, but with no luck. The Ceph version is Luminous 12.2.2.
Several Linux commands' parameter or output format used by ceph-volume are not compatible with Ubuntu 14.04:
1. lvscreate command doesn't support "--yes" parameter
2. blkid command doesn't show PARTUUID
After modified ceph-volume code to bypass these problem
1. remove "--yes" parameter
2. trucated "sgdisk" output as PARTUUID
The modified version ceph-volume still failed. When ceph-volume tried to run this command, ceph raised a core dump:
sudo ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 96 --monmap /var/lib/ceph/osd/ceph-96/activate.monmap --key ******************************** --bluestore-block-wal-path /dev/nvme0n1p1 --bluestore-block-db-path /dev/nvme0n1p1 --osd-data /var/lib/ceph/osd/ceph-96/ --osd-uuid 0e30f7f8-3fef-4a91-b748-443b55fbb1d6 --setuser ceph --setgroup ceph
The command that trigger the problem is as following, this command was ran on OSD host:
$ sudo ceph-volume lvm prepare --bluestore --data /dev/sdy --block.wal /dev/nvme0n1p1 --block.db /dev/nvme0n1p1
ceph-volume.log output is uploaded in attachment. The following is the start part of error output:
[2017-12-13 20:49:50,735][ceph_volume.process][INFO ] Running command: sudo ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 96 --monmap /var/lib/ceph/osd/ceph-96/activate.monmap --key ******************************** --bluestore-block-wal-path /dev/nvme0n1p1 --bluestore-block-db-path /dev/nvme0n1p1 --osd-data /var/lib/ceph/osd/ceph-96/ --osd-uuid 0e30f7f8-3fef-4a91-b748-443b55fbb1d6 --setuser ceph --setgroup ceph
[2017-12-13 20:49:50,763][ceph_volume.process][INFO ] stderr 2017-12-13 20:49:50.763027 7f2789e87d00 -1 bluestore(/var/lib/ceph/osd/ceph-96//block) read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding
[2017-12-13 20:49:50,763][ceph_volume.process][INFO ] stderr 2017-12-13 20:49:50.763233 7f2789e87d00 -1 bluestore(/var/lib/ceph/osd/ceph-96//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding
[2017-12-13 20:49:50,763][ceph_volume.process][INFO ] stderr 2017-12-13 20:49:50.763393 7f2789e87d00 -1 bluestore(/var/lib/ceph/osd/ceph-96//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding
2017-12-13 20:49:50.763452 7f2789e87d00 -1 bluestore(/var/lib/ceph/osd/ceph-96/) _read_fsid unparsable uuid
[2017-12-13 20:49:51,784][ceph_volume.process][INFO ] stderr /build/ceph-12.2.2/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f2789e87d00 time 2017-12-13 20:49:51.784626
/build/ceph-12.2.2/src/os/bluestore/StupidAllocator.cc: 300: FAILED assert(rm.empty())
[2017-12-13 20:49:51,787][ceph_volume.process][INFO ] stderr ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
1: (ceph::_ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x55cb5ba7d82e]
2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x291a) [0x55cb5ba2e66a]
3: (BlueFS::mount()+0x275) [0x55cb5ba0f9c5]
4: (BlueStore::_open_db(bool)+0x1680) [0x55cb5b92d2a0]
5: (BlueStore::_fsck(bool, bool)+0x3ca) [0x55cb5b95ebba]
6: (BlueStore::mkfs()+0xf20) [0x55cb5b936480]
7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x2a5) [0x55cb5b5299a5]
8: (main()+0x114e) [0x55cb5b441b1e]
9: (__libc_start_main()+0xf5) [0x7f27873b1f45]
10: (()+0x4b36d6) [0x55cb5b4e16d6]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2017-12-13 20:49:51.787609 7f2789e87d00 -1 /build/ceph-12.2.2/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f2789e87d00 time 2017-12-13 20:49:51.784626
History
#1 Updated by Alfredo Deza over 6 years ago
- Status changed from New to Rejected
Unfortunately, there is no way to make this work with ceph-volume. We are testing with ceph-ansible which also doesn't support Trusty.
On top of the problems you found, ceph-volume relies exclusively on systemd. And there is just no systemd capabilities in Trusty, so even if blkid/lsblk were able to work here, the OSD would not come up.
The alternatives here are either to keep using ceph-disk, or to upgrade to Xenial or CentOS7
We can't really fix this correctly for Trusty.
#2 Updated by Bruce Jiang over 6 years ago
Alfredo,
Thanks for you reply. I understand that, but, from documents from http://www.ceph.com, it says Luminous release is tested under Ubuntu 14.04 comprehensively.
http://docs.ceph.com/docs/master/start/os-recommendations/
Ubuntu 14.04 Trusty Tahr linux-3.13.0 B, I, C
If ceph-volume can't work on Ubuntu 14.04, this may be a little misleading.
BR,
Bruce J.