Actions
Bug #19039
closedceph-osd --mkkey --mkfs segfaults with bluestore
Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
When deploying ceph (master) with bluestore results in a segmentation fault with attempting to create OSD key.
Steps:- ceph build with LTTNG and EVENTTRACE enabled. (Errors out even without EVENTTRACE)
- ceph-deploy to start monitor.
- Both ceph-deploy & manual OSD activation fails.
ceph.conf
[global]
fsid = cf887c34-175d-4591-bc38-042066a37e3e
mon_initial_members = otccldstore04
mon_host = 172.25.110.43
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd pool default size = 1
enable experimental unrecoverable data corrupting features = bluestore
osd objectstore = bluestore
Logs:
sudo ceph-osd -f -i 0 --mkfs --mkkey --osd-uuid 47ef2e44-c926-47f7-b28f-4aad808a0562
2017-02-22 05:35:57.605480 7f5af0375a80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore
2017-02-22 05:35:57.605899 7f5af0375a80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore
2017-02-22 05:35:57.606012 7f5af0375a80 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
- Caught signal (Segmentation fault)
in thread 7f5af0375a80 thread_name:ceph-osd
ceph version 12.0.0-618-g7f66672 (7f66672b675abbc0262769d32a38112c781fefac)
1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x55a0be22b287]
2: (()+0x250dbff) [0x55a0be482bff]
3: (()+0x11390) [0x7f5aeea69390]
4: (std::enable_if<boost::is_same<bluestore_pextent_t const, bluestore_pextent_t>::value||boost::is_same<bluestore_pextent_t const, bluestore_pextent_t const>::value, void>::type denc_friend<bluestore_pextent_t const, unsigned long>(bluestore_pextent_t const&, unsigned long&)+0x14) [0x55a0be34a047]
5: (bluestore_pextent_t::bound_encode(unsigned long&) const+0x23) [0x55a0be33fcd9]
6: (denc_traits<bluestore_pextent_t, void>::bound_encode(bluestore_pextent_t const&, unsigned long&, unsigned long)+0x27) [0x55a0be33fd4f]
7: (std::enable_if<denc_traits<bluestore_pextent_t, void>::supported&&(!denc_traits<bluestore_pextent_t, void>::featured), void>::type denc<bluestore_pextent_t, denc_traits<bluestore_pextent_t, void> >(bluestore_pextent_t const&, unsigned long&, unsigned long)+0x2c) [0x55a0be34a11e]
8: (denc_traits<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, void>::bound_encode(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> > const&, unsigned long&)+0x4f) [0x55a0be33fdf5]
9: (std::enable_if<denc_traits<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, void>::supported&&(!denc_traits<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, void>::featured), void>::type denc<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, denc_traits<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, void> >(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> > const&, unsigned long&, unsigned long)+0x27) [0x55a0be34a880]
10: (bluestore_blob_t::bound_encode(unsigned long&, unsigned long) const+0x59) [0x55a0be340917]
11: (denc_traits<bluestore_blob_t, void>::bound_encode(bluestore_blob_t const&, unsigned long&, unsigned long)+0x2b) [0x55a0be342035]
12: (std::enable_if<denc_traits<bluestore_blob_t, void>::supported&&denc_traits<bluestore_blob_t, void>::featured, void>::type denc<bluestore_blob_t, denc_traits<bluestore_blob_t, void> >(bluestore_blob_t const&, unsigned long&, unsigned long)+0x2b) [0x55a0be34c7ec]
13: (BlueStore::Blob::bound_encode(unsigned long&, unsigned long, unsigned long, bool) const+0x39) [0x55a0be3447d1]
14: (BlueStore::ExtentMap::encode_some(unsigned int, unsigned int, ceph::buffer::list&, unsigned int*)+0x514) [0x55a0be2db6ea]
15: (BlueStore::ExtentMap::update(std::shared_ptr<KeyValueDB::TransactionImpl>, bool)+0x231) [0x55a0be2d73c3]
16: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x293) [0x55a0be313f95]
17: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x7c7) [0x55a0be31b413]
18: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x106) [0x55a0bdcb2b8e]
19: (ObjectStore::apply_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*)+0x14e) [0x55a0be1e44c6]
20: (ObjectStore::apply_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*)+0x6a) [0x55a0bdcb2934]
21: (OSD::mkfs(CephContext*, ObjectStore*, std::_cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0xc2a) [0x55a0bdc4cf66]
22: (main()+0x121d) [0x55a0bdc2a278]
23: (__libc_start_main()+0xf0) [0x7f5aed9f0830]
24: (_start()+0x29) [0x55a0bdc28779]
2017-02-22 05:36:09.227104 7f5af0375a80 -1 Caught signal (Segmentation fault) *
in thread 7f5af0375a80 thread_name:ceph-osd
ceph version 12.0.0-618-g7f66672 (7f66672b675abbc0262769d32a38112c781fefac)
1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x55a0be22b287]
2: (()+0x250dbff) [0x55a0be482bff]
3: (()+0x11390) [0x7f5aeea69390]
4: (std::enable_if<boost::is_same<bluestore_pextent_t const, bluestore_pextent_t>::value||boost::is_same<bluestore_pextent_t const, bluestore_pextent_t const>::value, void>::type denc_friend<bluestore_pextent_t const, unsigned long>(bluestore_pextent_t const&, unsigned long&)+0x14) [0x55a0be34a047]
5: (bluestore_pextent_t::bound_encode(unsigned long&) const+0x23) [0x55a0be33fcd9]
6: (denc_traits<bluestore_pextent_t, void>::bound_encode(bluestore_pextent_t const&, unsigned long&, unsigned long)+0x27) [0x55a0be33fd4f]
7: (std::enable_if<denc_traits<bluestore_pextent_t, void>::supported&&(!denc_traits<bluestore_pextent_t, void>::featured), void>::type denc<bluestore_pextent_t, denc_traits<bluestore_pextent_t, void> >(bluestore_pextent_t const&, unsigned long&, unsigned long)+0x2c) [0x55a0be34a11e]
8: (denc_traits<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, void>::bound_encode(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> > const&, unsigned long&)+0x4f) [0x55a0be33fdf5]
9: (std::enable_if<denc_traits<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, void>::supported&&(!denc_traits<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, void>::featured), void>::type denc<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, denc_traits<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, void> >(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> > const&, unsigned long&, unsigned long)+0x27) [0x55a0be34a880]
10: (bluestore_blob_t::bound_encode(unsigned long&, unsigned long) const+0x59) [0x55a0be340917]
11: (denc_traits<bluestore_blob_t, void>::bound_encode(bluestore_blob_t const&, unsigned long&, unsigned long)+0x2b) [0x55a0be342035]
12: (std::enable_if<denc_traits<bluestore_blob_t, void>::supported&&denc_traits<bluestore_blob_t, void>::featured, void>::type denc<bluestore_blob_t, denc_traits<bluestore_blob_t, void> >(bluestore_blob_t const&, unsigned long&, unsigned long)+0x2b) [0x55a0be34c7ec]
13: (BlueStore::Blob::bound_encode(unsigned long&, unsigned long, unsigned long, bool) const+0x39) [0x55a0be3447d1]
14: (BlueStore::ExtentMap::encode_some(unsigned int, unsigned int, ceph::buffer::list&, unsigned int*)+0x514) [0x55a0be2db6ea]
15: (BlueStore::ExtentMap::update(std::shared_ptr<KeyValueDB::TransactionImpl>, bool)+0x231) [0x55a0be2d73c3]
16: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x293) [0x55a0be313f95]
17: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x7c7) [0x55a0be31b413]
18: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x106) [0x55a0bdcb2b8e]
19: (ObjectStore::apply_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*)+0x14e) [0x55a0be1e44c6]
20: (ObjectStore::apply_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*)+0x6a) [0x55a0bdcb2934]
21: (OSD::mkfs(CephContext*, ObjectStore*, std::_cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0xc2a) [0x55a0bdc4cf66]
22: (main()+0x121d) [0x55a0bdc2a278]
23: (__libc_start_main()+0xf0) [0x7f5aed9f0830]
24: (_start()+0x29) [0x55a0bdc28779]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.-399> 2017-02-22 05:35:57.605480 7f5af0375a80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore
-398> 2017-02-22 05:35:57.605899 7f5af0375a80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore
-395> 2017-02-22 05:35:57.606012 7f5af0375a80 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.-394> 2017-02-22 05:35:57.609182 7f5af0375a80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore
0> 2017-02-22 05:36:09.227104 7f5af0375a80 -1 ** Caught signal (Segmentation fault) *
in thread 7f5af0375a80 thread_name:ceph-osdceph version 12.0.0-618-g7f66672 (7f66672b675abbc0262769d32a38112c781fefac)
1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x55a0be22b287]
2: (()+0x250dbff) [0x55a0be482bff]
3: (()+0x11390) [0x7f5aeea69390]
4: (std::enable_if<boost::is_same<bluestore_pextent_t const, bluestore_pextent_t>::value||boost::is_same<bluestore_pextent_t const, bluestore_pextent_t const>::value, void>::type denc_friend<bluestore_pextent_t const, unsigned long>(bluestore_pextent_t const&, unsigned long&)+0x14) [0x55a0be34a047]
5: (bluestore_pextent_t::bound_encode(unsigned long&) const+0x23) [0x55a0be33fcd9]
6: (denc_traits<bluestore_pextent_t, void>::bound_encode(bluestore_pextent_t const&, unsigned long&, unsigned long)+0x27) [0x55a0be33fd4f]
7: (std::enable_if<denc_traits<bluestore_pextent_t, void>::supported&&(!denc_traits<bluestore_pextent_t, void>::featured), void>::type denc<bluestore_pextent_t, denc_traits<bluestore_pextent_t, void> >(bluestore_pextent_t const&, unsigned long&, unsigned long)+0x2c) [0x55a0be34a11e]
8: (denc_traits<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, void>::bound_encode(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> > const&, unsigned long&)+0x4f) [0x55a0be33fdf5]
9: (std::enable_if<denc_traits<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, void>::supported&&(!denc_traits<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, void>::featured), void>::type denc<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, denc_traits<std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> >, void> >(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)2, bluestore_pextent_t> > const&, unsigned long&, unsigned long)+0x27) [0x55a0be34a880]
10: (bluestore_blob_t::bound_encode(unsigned long&, unsigned long) const+0x59) [0x55a0be340917]
11: (denc_traits<bluestore_blob_t, void>::bound_encode(bluestore_blob_t const&, unsigned long&, unsigned long)+0x2b) [0x55a0be342035]
12: (std::enable_if<denc_traits<bluestore_blob_t, void>::supported&&denc_traits<bluestore_blob_t, void>::featured, void>::type denc<bluestore_blob_t, denc_traits<bluestore_blob_t, void> >(bluestore_blob_t const&, unsigned long&, unsigned long)+0x2b) [0x55a0be34c7ec]
13: (BlueStore::Blob::bound_encode(unsigned long&, unsigned long, unsigned long, bool) const+0x39) [0x55a0be3447d1]
14: (BlueStore::ExtentMap::encode_some(unsigned int, unsigned int, ceph::buffer::list&, unsigned int*)+0x514) [0x55a0be2db6ea]
15: (BlueStore::ExtentMap::update(std::shared_ptr<KeyValueDB::TransactionImpl>, bool)+0x231) [0x55a0be2d73c3]
16: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x293) [0x55a0be313f95]
17: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x7c7) [0x55a0be31b413]
18: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x106) [0x55a0bdcb2b8e]
19: (ObjectStore::apply_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*)+0x14e) [0x55a0be1e44c6]
20: (ObjectStore::apply_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*)+0x6a) [0x55a0bdcb2934]
21: (OSD::mkfs(CephContext*, ObjectStore*, std::_cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0xc2a) [0x55a0bdc4cf66]
22: (main()+0x121d) [0x55a0bdc2a278]
23: (__libc_start_main()+0xf0) [0x7f5aed9f0830]
24: (_start()+0x29) [0x55a0bdc28779]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.Segmentation fault (core dumped)
Updated by Ganesh Mahalingam about 7 years ago
This issue only happens when I set WITH_LTTNG, WITH_EVENTTRACE, HAVE_BABELTRACE flags and build with DEB_BUILD_OPTIONS='nostrip noopt debug'. This leads to an issue where 'ceph-disk prepare --bluestore ......' to not complete and the OSD drive is not properly mounted or contain the right files (especially file ready). This causes the subsequent 'ceph-deploy osd activate...' to try and re-initialize the drives leading to the crash.
Updated by Ganesh Mahalingam about 7 years ago
The culprit seems to be passing
DEB_BUILD_OPTIONS='nostrip noopt debug'
when building the package. While there is still a crash, i suspect someone running/building deb packages in this fashion is highly unlikely. Please feel free to reduce priority/close the issue if you think this usecase in invalid. Apologies if this has caused some noise.
Actions