Project

General

Profile

Support #54365

Setup 64kB block size of Bluestore RAW HDD

Added by Pavel Zajíc about 2 years ago. Updated about 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Tags:
Reviewed:
Affected Versions:
Pull request ID:

Description

Hello,
we are using Ceph like object storage with our proprietary client using librados. We store directly to Ceph pools live video streams MP4 files of 4MB size each. My setup is a 9x Dell 730 server with 21x 3.6TB NL-SAS HDD for data and 3x 960GB SSD for RockDB and WAL, deployed by cephadm version 16.2.7.
I would like to set up (change) block_size of the Bluestore to 64kB to increase the HDD performance. I had decommissioned all OSD from one server and then I have changed the parameter bdev_block_size from 4096 to 65536. When I executed "ceph orch apply osd -i ./osd_spec.yml" the cephadm was not able to create BlueFS on LVM partitions and ended with the following error. The idea was to roll over all cluster OSD and recreate them with the new block size parameter.
Are you able to give me a hint on how to properly change the block size of Bluestore (HDD and SSD)?

---
root@ceph-storage-prod-brq-003:~# ceph health detail
HEALTH_WARN Failed to apply 1 service(s): osd.osd_spec_hdd
[WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.osd_spec_hdd
osd.osd_spec_hdd: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:bb6a71f7f481985f6d3b358e3b9ef64c6755b3db5aa53198e0aac38be5c8ae54 -e NODE_NAME=ceph-storage-prod-brq-004 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd_spec_hdd -v /var/run/ceph/35d6151c-b485-11eb-9d51-593463c27b26:/var/run/ceph:z -v /var/log/ceph/35d6151c-b485-11eb-9d51-593463c27b26:/var/log/ceph:z -v /var/lib/ceph/35d6151c-b485-11eb-9d51-593463c27b26/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /tmp/ceph-tmp916kv2_e:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpo1vkripr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:bb6a71f7f481985f6d3b358e3b9ef64c6755b3db5aa53198e0aac38be5c8ae54 lvm batch --no-auto /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu --db-devices /dev/sdv /dev/sdw /dev/sdx --yes --no-systemd
/usr/bin/docker: stderr --> passed data devices: 21 physical, 0 LVM
/usr/bin/docker: stderr --> relative data size: 1.0
/usr/bin/docker: stderr --> passed block_db devices: 3 physical, 0 LVM
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8c469ae5-08fb-462f-925e-21654aa34310
/usr/bin/docker: stderr Running command: /usr/sbin/vgcreate --force --yes ceph-d1bc058f-52fd-4976-9cdc-c3adc381ce19 /dev/sdk
/usr/bin/docker: stderr stdout: Physical volume "/dev/sdk" successfully created.
/usr/bin/docker: stderr stdout: Volume group "ceph-d1bc058f-52fd-4976-9cdc-c3adc381ce19" successfully created
/usr/bin/docker: stderr Running command: /usr/sbin/lvcreate --yes -l 953861 -n osd-block-8c469ae5-08fb-462f-925e-21654aa34310 ceph-d1bc058f-52fd-4976-9cdc-c3adc381ce19
/usr/bin/docker: stderr stdout: Wiping atari signature on /dev/ceph-d1bc058f-52fd-4976-9cdc-c3adc381ce19/osd-block-8c469ae5-08fb-462f-925e-21654aa34310.
/usr/bin/docker: stderr stdout: Logical volume "osd-block-8c469ae5-08fb-462f-925e-21654aa34310" created.
/usr/bin/docker: stderr Running command: /usr/sbin/lvcreate --yes -l 65407 -n osd-db-4752d160-fd86-4a04-b77d-fcd7de6c2d39 ceph-97cecde1-18f3-4707-bf08-01240e1eb606
/usr/bin/docker: stderr stdout: Logical volume "osd-db-4752d160-fd86-4a04-b77d-fcd7de6c2d39" created.
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
/usr/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-5
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-d1bc058f-52fd-4976-9cdc-c3adc381ce19/osd-block-8c469ae5-08fb-462f-925e-21654aa34310
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--d1bc058f--52fd--4976--9cdc--c3adc381ce19-osd--block--8c469ae5--08fb--462f--925e--21654aa34310
/usr/bin/docker: stderr Running command: /usr/bin/ln -s /dev/ceph-d1bc058f-52fd-4976-9cdc-c3adc381ce19/osd-block-8c469ae5-08fb-462f-925e-21654aa34310 /var/lib/ceph/osd/ceph-5/block
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-5/activate.monmap
/usr/bin/docker: stderr stderr: got monmap epoch 16
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-5/keyring --create-keyring --name osd.5 --add-key AQDRxhRi/lkXOxAAqa7kRFjDxgsyiXbNi46OGw==
/usr/bin/docker: stderr stdout: creating /var/lib/ceph/osd/ceph-5/keyring
/usr/bin/docker: stderr added entity osd.5 auth(key=AQDRxhRi/lkXOxAAqa7kRFjDxgsyiXbNi46OGw==)
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5/keyring
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5/
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-97cecde1-18f3-4707-bf08-01240e1eb606/osd-db-4752d160-fd86-4a04-b77d-fcd7de6c2d39
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--97cecde1--18f3--4707--bf08--01240e1eb606-osd--db--4752d160--fd86--4a04--b77d--fcd7de6c2d39
/usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 5 --monmap /var/lib/ceph/osd/ceph-5/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-97cecde1-18f3-4707-bf08-01240e1eb606/osd-db-4752d160-fd86-4a04-b77d-fcd7de6c2d39 --osdspec-affinity osd_spec_hdd --osd-data /var/lib/ceph/osd/ceph-5/ --osd-uuid 8c469ae5-08fb-462f-925e-21654aa34310 --setuser ceph --setgroup ceph
/usr/bin/docker: stderr stderr: 2022-02-22T11:19:48.768+0000 7f1e058bf080 -1 bluestore(/var/lib/ceph/osd/ceph-5/) read_fsid unparsable uuid
/usr/bin/docker: stderr stderr: 2022-02-22T11:19:48.772+0000 7f1e058bf080 -1 bdev is_valid_io 1000~1000 block_size 10000 size 3fdfc00000
/usr/bin/docker: stderr stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/blk/kernel/KernelDevice.cc: In function 'virtual int KernelDevice::write(uint64_t, ceph::bufferlist&, bool, int)' thread 7f1e058bf080 time 2022-02-22T11:19:48.775929+0000
/usr/bin/docker: stderr stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/blk/kernel/KernelDevice.cc: 903: FAILED ceph_assert(is_valid_io(off, len))
/usr/bin/docker: stderr stderr: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
/usr/bin/docker: stderr stderr: 1: (ceph::
_ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x564e4bb0eb7e]
/usr/bin/docker: stderr stderr: 2: /usr/bin/ceph-osd(+0x56ad98) [0x564e4bb0ed98]
/usr/bin/docker: stderr stderr: 3: (KernelDevice::write(unsigned long, ceph::buffer::v15_2_0::list&, bool, int)+0x697) [0x564e4c662f87]
/usr/bin/docker: stderr stderr: 4: (BlueFS::_write_super(int)+0xdc) [0x564e4c20299c]
/usr/bin/docker: stderr stderr: 5: (BlueFS::mkfs(uuid_d, bluefs_layout_t const&)+0x56d) [0x564e4c20fd1d]
/usr/bin/docker: stderr stderr: 6: (BlueStore::_open_bluefs(bool, bool)+0x255) [0x564e4c102625]
/usr/bin/docker: stderr stderr: 7: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x6d9) [0x564e4c1035b9]
/usr/bin/docker: stderr stderr: 8: (BlueStore::_open_db(bool, bool, bool)+0x155) [0x564e4c1048b5]
/usr/bin/docker: stderr stderr: 9: (BlueStore::mkfs()+0x116e) [0x564e4c1746ae]
/usr/bin/docker: stderr stderr: 10: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1af) [0x564e4bbef05f]
/usr/bin/docker: stderr stderr: 11: main()
/usr/bin/docker: stderr stderr: 12: _libc_start_main()
/usr/bin/docker: stderr stderr: 13: _start()
/usr/bin/docker: stderr stderr: * Caught signal (Aborted) *
/usr/bin/docker: stderr stderr: in thread 7f1e058bf080 thread_name:ceph-osd
/usr/bin/docker: stderr stderr: 2022-02-22T11:19:48.780+0000 7f1e058bf080 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/blk/kernel/KernelDevice.cc: In function 'virtual int KernelDevice::write(uint64_t, ceph::bufferlist&, bool, int)' thread 7f1e058bf080 time 2022-02-22T11:19:48.775929+0000
/usr/bin/docker: stderr stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/blk/kernel/KernelDevice.cc: 903: FAILED ceph_assert(is_valid_io(off, len))
/usr/bin/docker: stderr stderr: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
/usr/bin/docker: stderr stderr: 1: (ceph::
_ceph_assert_fail(char const, char const*, int, char const*)+0x158) [0x564e4bb0eb7e]
/usr/bin/docker: stderr stderr: 2: /usr/bin/ceph-osd(+0x56ad98) [0x564e4bb0ed98]
/usr/bin/docker: stderr stderr: 3: (KernelDevice::write(unsigned long, ceph::buffer::v15_2_0::list&, bool, int)+0x697) [0x564e4c662f87]
/usr/bin/docker: stderr stderr: 4: (BlueFS::_write_super(int)+0xdc) [0x564e4c20299c]
/usr/bin/docker: stderr stderr: 5: (BlueFS::mkfs(uuid_d, bluefs_layout_t const&)+0x56d) [0x564e4c20fd1d]
/usr/bin/docker: stderr stderr: 6: (BlueStore::_open_bluefs(bool, bool)+0x255) [0x564e4c102625]
/usr/bin/docker: stderr stderr: 7: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x6d9) [0x564e4c1035b9]
/usr/bin/docker: stderr stderr: 8: (BlueStore::_open_db(bool, bool, bool)+0x155) [0x564e4c1048b5]
/usr/bin/docker: stderr stderr: 9: (BlueStore::mkfs()+0x116e) [0x564e4c1746ae]
/usr/bin/docker: stderr stderr: 10: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1af) [0x564e4bbef05f]
/usr/bin/docker: stderr stderr: 11: main()
/usr/bin/docker: stderr stderr: 12: _libc_start_main()
/usr/bin/docker: stderr stderr: 13: _start()
/usr/bin/docker: stderr stderr: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
/usr/bin/docker: stderr stderr: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f1e03862c20]
/usr/bin/docker: stderr stderr: 2: gsignal()
/usr/bin/docker: stderr stderr: 3: abort()
/usr/bin/docker: stderr stderr: 4: (ceph::
_ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x564e4bb0ebcf]
/usr/bin/docker: stderr stderr: 5: /usr/bin/ceph-osd(+0x56ad98) [0x564e4bb0ed98]
/usr/bin/docker: stderr stderr: 6: (KernelDevice::write(unsigned long, ceph::buffer::v15_2_0::list&, bool, int)+0x697) [0x564e4c662f87]
/usr/bin/docker: stderr stderr: 7: (BlueFS::_write_super(int)+0xdc) [0x564e4c20299c]
/usr/bin/docker: stderr stderr: 8: (BlueFS::mkfs(uuid_d, bluefs_layout_t const&)+0x56d) [0x564e4c20fd1d]
/usr/bin/docker: stderr stderr: 9: (BlueStore::_open_bluefs(bool, bool)+0x255) [0x564e4c102625]
/usr/bin/docker: stderr stderr: 10: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x6d9) [0x564e4c1035b9]
/usr/bin/docker: stderr stderr: 11: (BlueStore::_open_db(bool, bool, bool)+0x155) [0x564e4c1048b5]
/usr/bin/docker: stderr stderr: 12: (BlueStore::mkfs()+0x116e) [0x564e4c1746ae]
/usr/bin/docker: stderr stderr: 13: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1af) [0x564e4bbef05f]
/usr/bin/docker: stderr stderr: 14: main()
/usr/bin/docker: stderr stderr: 15: _libc_start_main()
/usr/bin/docker: stderr stderr: 16: _start()
/usr/bin/docker: stderr stderr: 2022-02-22T11:19:48.784+0000 7f1e058bf080 -1
Caught signal (Aborted)
/usr/bin/docker: stderr stderr: in thread 7f1e058bf080 thread_name:ceph-osd
/usr/bin/docker: stderr stderr: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
/usr/bin/docker: stderr stderr: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f1e03862c20]
/usr/bin/docker: stderr stderr: 2: gsignal()
/usr/bin/docker: stderr stderr: 3: abort()
/usr/bin/docker: stderr stderr: 4: (ceph::
_ceph_assert_fail(char const, char const*, int, char const*)+0x1a9) [0x564e4bb0ebcf]
/usr/bin/docker: stderr stderr: 5: /usr/bin/ceph-osd(+0x56ad98) [0x564e4bb0ed98]
/usr/bin/docker: stderr stderr: 6: (KernelDevice::write(unsigned long, ceph::buffer::v15_2_0::list&, bool, int)+0x697) [0x564e4c662f87]
/usr/bin/docker: stderr stderr: 7: (BlueFS::_write_super(int)+0xdc) [0x564e4c20299c]
/usr/bin/docker: stderr stderr: 8: (BlueFS::mkfs(uuid_d, bluefs_layout_t const&)+0x56d) [0x564e4c20fd1d]
/usr/bin/docker: stderr stderr: 9: (BlueStore::_open_bluefs(bool, bool)+0x255) [0x564e4c102625]
/usr/bin/docker: stderr stderr: 10: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x6d9) [0x564e4c1035b9]
/usr/bin/docker: stderr stderr: 11: (BlueStore::_open_db(bool, bool, bool)+0x155) [0x564e4c1048b5]
/usr/bin/docker: stderr stderr: 12: (BlueStore::mkfs()+0x116e) [0x564e4c1746ae]
/usr/bin/docker: stderr stderr: 13: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1af) [0x564e4bbef05f]
/usr/bin/docker: stderr stderr: 14: main()
/usr/bin/docker: stderr stderr: 15: _libc_start_main()
/usr/bin/docker: stderr stderr: 16: _start()
/usr/bin/docker: stderr stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
/usr/bin/docker: stderr stderr: -39> 2022-02-22T11:19:48.768+0000 7f1e058bf080 -1 bluestore(/var/lib/ceph/osd/ceph-5/) _read_fsid unparsable uuid
/usr/bin/docker: stderr stderr: -2> 2022-02-22T11:19:48.772+0000 7f1e058bf080 -1 bdev is_valid_io 1000~1000 block_size 10000 size 3fdfc00000
/usr/bin/docker: stderr stderr: -1> 2022-02-22T11:19:48.780+0000 7f1e058bf080 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/blk/kernel/KernelDevice.cc: In function 'virtual int KernelDevice::write(uint64_t, ceph::bufferlist&, bool, int)' thread 7f1e058bf080 time 2022-02-22T11:19:48.775929+0000
/usr/bin/docker: stderr stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/blk/kernel/KernelDevice.cc: 903: FAILED ceph_assert(is_valid_io(off, len))
/usr/bin/docker: stderr stderr: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
/usr/bin/docker: stderr stderr: 1: (ceph::
_ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x564e4bb0eb7e]
/usr/bin/docker: stderr stderr: 2: /usr/bin/ceph-osd(+0x56ad98) [0x564e4bb0ed98]
/usr/bin/docker: stderr stderr: 3: (KernelDevice::write(unsigned long, ceph::buffer::v15_2_0::list&, bool, int)+0x697) [0x564e4c662f87]
/usr/bin/docker: stderr stderr: 4: (BlueFS::_write_super(int)+0xdc) [0x564e4c20299c]
/usr/bin/docker: stderr stderr: 5: (BlueFS::mkfs(uuid_d, bluefs_layout_t const&)+0x56d) [0x564e4c20fd1d]
/usr/bin/docker: stderr stderr: 6: (BlueStore::_open_bluefs(bool, bool)+0x255) [0x564e4c102625]
/usr/bin/docker: stderr stderr: 7: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x6d9) [0x564e4c1035b9]
/usr/bin/docker: stderr stderr: 8: (BlueStore::_open_db(bool, bool, bool)+0x155) [0x564e4c1048b5]
/usr/bin/docker: stderr stderr: 9: (BlueStore::mkfs()+0x116e) [0x564e4c1746ae]
/usr/bin/docker: stderr stderr: 10: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1af) [0x564e4bbef05f]
/usr/bin/docker: stderr stderr: 11: main()
/usr/bin/docker: stderr stderr: 12: _libc_start_main()
/usr/bin/docker: stderr stderr: 13: _start()
/usr/bin/docker: stderr stderr: 0> 2022-02-22T11:19:48.784+0000 7f1e058bf080 -1
Caught signal (Aborted)
/usr/bin/docker: stderr stderr: in thread 7f1e058bf080 thread_name:ceph-osd
/usr/bin/docker: stderr stderr: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
/usr/bin/docker: stderr stderr: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f1e03862c20]
/usr/bin/docker: stderr stderr: 2: gsignal()
/usr/bin/docker: stderr stderr: 3: abort()
/usr/bin/docker: stderr stderr: 4: (ceph::
_ceph_assert_fail(char const, char const*, int, char const*)+0x1a9) [0x564e4bb0ebcf]
/usr/bin/docker: stderr stderr: 5: /usr/bin/ceph-osd(+0x56ad98) [0x564e4bb0ed98]
/usr/bin/docker: stderr stderr: 6: (KernelDevice::write(unsigned long, ceph::buffer::v15_2_0::list&, bool, int)+0x697) [0x564e4c662f87]
/usr/bin/docker: stderr stderr: 7: (BlueFS::_write_super(int)+0xdc) [0x564e4c20299c]
/usr/bin/docker: stderr stderr: 8: (BlueFS::mkfs(uuid_d, bluefs_layout_t const&)+0x56d) [0x564e4c20fd1d]
/usr/bin/docker: stderr stderr: 9: (BlueStore::_open_bluefs(bool, bool)+0x255) [0x564e4c102625]
/usr/bin/docker: stderr stderr: 10: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x6d9) [0x564e4c1035b9]
/usr/bin/docker: stderr stderr: 11: (BlueStore::_open_db(bool, bool, bool)+0x155) [0x564e4c1048b5]
/usr/bin/docker: stderr stderr: 12: (BlueStore::mkfs()+0x116e) [0x564e4c1746ae]
/usr/bin/docker: stderr stderr: 13: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1af) [0x564e4bbef05f]
/usr/bin/docker: stderr stderr: 14: main()
/usr/bin/docker: stderr stderr: 15: _libc_start_main()
/usr/bin/docker: stderr stderr: 16: _start()
/usr/bin/docker: stderr stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
/usr/bin/docker: stderr stderr: -39> 2022-02-22T11:19:48.768+0000 7f1e058bf080 -1 bluestore(/var/lib/ceph/osd/ceph-5/) _read_fsid unparsable uuid
/usr/bin/docker: stderr stderr: -2> 2022-02-22T11:19:48.772+0000 7f1e058bf080 -1 bdev is_valid_io 1000~1000 block_size 10000 size 3fdfc00000
/usr/bin/docker: stderr stderr: -1> 2022-02-22T11:19:48.780+0000 7f1e058bf080 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/blk/kernel/KernelDevice.cc: In function 'virtual int KernelDevice::write(uint64_t, ceph::bufferlist&, bool, int)' thread 7f1e058bf080 time 2022-02-22T11:19:48.775929+0000
/usr/bin/docker: stderr stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/blk/kernel/KernelDevice.cc: 903: FAILED ceph_assert(is_valid_io(off, len))
/usr/bin/docker: stderr stderr: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
/usr/bin/docker: stderr stderr: 1: (ceph::
_ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x564e4bb0eb7e]
/usr/bin/docker: stderr stderr: 2: /usr/bin/ceph-osd(+0x56ad98) [0x564e4bb0ed98]
/usr/bin/docker: stderr stderr: 3: (KernelDevice::write(unsigned long, ceph::buffer::v15_2_0::list&, bool, int)+0x697) [0x564e4c662f87]
/usr/bin/docker: stderr stderr: 4: (BlueFS::_write_super(int)+0xdc) [0x564e4c20299c]
/usr/bin/docker: stderr stderr: 5: (BlueFS::mkfs(uuid_d, bluefs_layout_t const&)+0x56d) [0x564e4c20fd1d]
/usr/bin/docker: stderr stderr: 6: (BlueStore::_open_bluefs(bool, bool)+0x255) [0x564e4c102625]
/usr/bin/docker: stderr stderr: 7: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x6d9) [0x564e4c1035b9]
/usr/bin/docker: stderr stderr: 8: (BlueStore::_open_db(bool, bool, bool)+0x155) [0x564e4c1048b5]
/usr/bin/docker: stderr stderr: 9: (BlueStore::mkfs()+0x116e) [0x564e4c1746ae]
/usr/bin/docker: stderr stderr: 10: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1af) [0x564e4bbef05f]
/usr/bin/docker: stderr stderr: 11: main()
/usr/bin/docker: stderr stderr: 12: _libc_start_main()
/usr/bin/docker: stderr stderr: 13: _start()
/usr/bin/docker: stderr stderr: 0> 2022-02-22T11:19:48.784+0000 7f1e058bf080 -1
Caught signal (Aborted) *
/usr/bin/docker: stderr stderr: in thread 7f1e058bf080 thread_name:ceph-osd
/usr/bin/docker: stderr stderr: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
/usr/bin/docker: stderr stderr: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f1e03862c20]
/usr/bin/docker: stderr stderr: 2: gsignal()
/usr/bin/docker: stderr stderr: 3: abort()
/usr/bin/docker: stderr stderr: 4: (ceph::
_ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x564e4bb0ebcf]
/usr/bin/docker: stderr stderr: 5: /usr/bin/ceph-osd(+0x56ad98) [0x564e4bb0ed98]
/usr/bin/docker: stderr stderr: 6: (KernelDevice::write(unsigned long, ceph::buffer::v15_2_0::list&, bool, int)+0x697) [0x564e4c662f87]
/usr/bin/docker: stderr stderr: 7: (BlueFS::_write_super(int)+0xdc) [0x564e4c20299c]
/usr/bin/docker: stderr stderr: 8: (BlueFS::mkfs(uuid_d, bluefs_layout_t const&)+0x56d) [0x564e4c20fd1d]
/usr/bin/docker: stderr stderr: 9: (BlueStore::_open_bluefs(bool, bool)+0x255) [0x564e4c102625]
/usr/bin/docker: stderr stderr: 10: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x6d9) [0x564e4c1035b9]
/usr/bin/docker: stderr stderr: 11: (BlueStore::_open_db(bool, bool, bool)+0x155) [0x564e4c1048b5]
/usr/bin/docker: stderr stderr: 12: (BlueStore::mkfs()+0x116e) [0x564e4c1746ae]
/usr/bin/docker: stderr stderr: 13: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1af) [0x564e4bbef05f]
/usr/bin/docker: stderr stderr: 14: main()
/usr/bin/docker: stderr stderr: 15: __libc_start_main()
/usr/bin/docker: stderr stderr: 16: _start()
/usr/bin/docker: stderr stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
/usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.5 --yes-i-really-mean-it
/usr/bin/docker: stderr stderr: purged osd.5
/usr/bin/docker: stderr --> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 5 --monmap /var/lib/ceph/osd/ceph-5/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-97cecde1-18f3-4707-bf08-01240e1eb606/osd-db-4752d160-fd86-4a04-b77d-fcd7de6c2d39 --osdspec-affinity osd_spec_hdd --osd-data /var/lib/ceph/osd/ceph-5/ --osd-uuid 8c469ae5-08fb-462f-925e-21654aa34310 --setuser ceph --setgroup ceph
Traceback (most recent call last):
File "/var/lib/ceph/35d6151c-b485-11eb-9d51-593463c27b26/cephadm.55e70975756e8c180366666f9fa21d3301c67edc3a5000698fd6e7ccb6fcafee", line 8571, in <module>
main()
File "/var/lib/ceph/35d6151c-b485-11eb-9d51-593463c27b26/cephadm.55e70975756e8c180366666f9fa21d3301c67edc3a5000698fd6e7ccb6fcafee", line 8559, in main
r = ctx.func(ctx)
File "/var/lib/ceph/35d6151c-b485-11eb-9d51-593463c27b26/cephadm.55e70975756e8c180366666f9fa21d3301c67edc3a5000698fd6e7ccb6fcafee", line 1737, in _infer_config
return func(ctx)
File "/var/lib/ceph/35d6151c-b485-11eb-9d51-593463c27b26/cephadm.55e70975756e8c180366666f9fa21d3301c67edc3a5000698fd6e7ccb6fcafee", line 1678, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/35d6151c-b485-11eb-9d51-593463c27b26/cephadm.55e70975756e8c180366666f9fa21d3301c67edc3a5000698fd6e7ccb6fcafee", line 1765, in _infer_image
return func(ctx)
File "/var/lib/ceph/35d6151c-b485-11eb-9d51-593463c27b26/cephadm.55e70975756e8c180366666f9fa21d3301c67edc3a5000698fd6e7ccb6fcafee", line 1665, in _validate_fsid
return func(ctx)
File "/var/lib/ceph/35d6151c-b485-11eb-9d51-593463c27b26/cephadm.55e70975756e8c180366666f9fa21d3301c67edc3a5000698fd6e7ccb6fcafee", line 4822, in command_ceph_volume
out, err, code = call_throws(ctx, c.run_cmd())
File "/var/lib/ceph/35d6151c-b485-11eb-9d51-593463c27b26/cephadm.55e70975756e8c180366666f9fa21d3301c67edc3a5000698fd6e7ccb6fcafee", line 1467, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:bb6a71f7f481985f6d3b358e3b9ef64c6755b3db5aa53198e0aac38be5c8ae54 -e NODE_NAME=ceph-storage-prod-brq-004 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd_spec_hdd -v /var/run/ceph/35d6151c-b485-11eb-9d51-593463c27b26:/var/run/ceph:z -v /var/log/ceph/35d6151c-b485-11eb-9d51-593463c27b26:/var/log/ceph:z -v /var/lib/ceph/35d6151c-b485-11eb-9d51-593463c27b26/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /tmp/ceph-tmp916kv2_e:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpo1vkripr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:bb6a71f7f481985f6d3b358e3b9ef64c6755b3db5aa53198e0aac38be5c8ae54 lvm batch --no-auto /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu --db-devices /dev/sdv /dev/sdw /dev/sdx --yes --no-systemd

History

#1 Updated by Igor Fedotov about 2 years ago

Please do not alter bdev_block_size parameter. You should use bluestore_min_alloc_size_hdd parameter instead.

Also available in: Atom PDF