Project

General

Profile

Actions
Copy link

Bug #58441

open

ceph-bluestore-tool fsck crash with "FAILED ceph_assert(v.length() == p->shard_info->bytes)"

Added by Changyuan Yu over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

After OSD crashed with "FAILED ceph_assert(v.length() == p->shard_info->bytes)" (crash report here https://gist.github.com/yuchangyuan/2016f259175940f64e2eed528d633794 ) , I try to run ceph-bluestore-tool fsck on crashed OSD, but also crashed with same error. I think ceph-bluestore-tool should be able to handle this situation, but not.

Below is the output of ceph-bluestore-tool fsck


/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)' thread 7f5be5a6f540 time 2023-01-13T11:52:08.334952+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/os/bluestore/BlueStore.cc: 3228: FAILED ceph_assert(v.length() == p->shard_info->bytes)
 ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f5bdae76de8]
 2: /usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7f5bdae77002]
 3: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x7cf) [0x5614500a826f]
 4: (BlueStore::fsck_check_objects_shallow(BlueStore::FSCKDepth, long, boost::intrusive_ptr<BlueStore::Collection>, ghobject_t const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list const&, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mempool::pool_allocator<(mempool::pool_index_t)11, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::map<boost::intrusive_ptr<BlueStore::Blob>, unsigned short, std::less<boost::intrusive_ptr<BlueStore::Blob> >, std::allocator<std::pair<boost::intrusive_ptr<BlueStore::Blob> const, unsigned short> > >*, BlueStore::FSCK_ObjectCtx const&)+0x234) [0x5614501115b4]
 5: (BlueStore::_fsck_check_objects(BlueStore::FSCKDepth, BlueStore::FSCK_ObjectCtx&)+0x15ba) [0x56145011551a]
 6: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x1e7e) [0x561450119a1e]
 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x21c) [0x561450126dbc]
 8: main()
 9: __libc_start_main()
 10: _start()
2023-01-13T11:52:08.343+0000 7f5be5a6f540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)' thread 7f5be5a6f540 time 2023-01-13T11:52:08.334952+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/os/bluestore/BlueStore.cc: 3228: FAILED ceph_assert(v.length() == p->shard_info->bytes)

 ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f5bdae76de8]
 2: /usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7f5bdae77002]
 3: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x7cf) [0x5614500a826f]
 4: (BlueStore::fsck_check_objects_shallow(BlueStore::FSCKDepth, long, boost::intrusive_ptr<BlueStore::Collection>, ghobject_t const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list const&, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mempool::pool_allocator<(mempool::pool_index_t)11, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::map<boost::intrusive_ptr<BlueStore::Blob>, unsigned short, std::less<boost::intrusive_ptr<BlueStore::Blob> >, std::allocator<std::pair<boost::intrusive_ptr<BlueStore::Blob> const, unsigned short> > >*, BlueStore::FSCK_ObjectCtx const&)+0x234) [0x5614501115b4]
 5: (BlueStore::_fsck_check_objects(BlueStore::FSCKDepth, BlueStore::FSCK_ObjectCtx&)+0x15ba) [0x56145011551a]
 6: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x1e7e) [0x561450119a1e]
 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x21c) [0x561450126dbc]
 8: main()
 9: __libc_start_main()
 10: _start()

*** Caught signal (Aborted) **
 in thread 7f5be5a6f540 thread_name:ceph-bluestore-
 ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f5bd9e12cf0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f5bdae76e39]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7f5bdae77002]
 6: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x7cf) [0x5614500a826f]
 7: (BlueStore::fsck_check_objects_shallow(BlueStore::FSCKDepth, long, boost::intrusive_ptr<BlueStore::Collection>, ghobject_t const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list const&, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mempool::pool_allocator<(mempool::pool_index_t)11, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::map<boost::intrusive_ptr<BlueStore::Blob>, unsigned short, std::less<boost::intrusive_ptr<BlueStore::Blob> >, std::allocator<std::pair<boost::intrusive_ptr<BlueStore::Blob> const, unsigned short> > >*, BlueStore::FSCK_ObjectCtx const&)+0x234) [0x5614501115b4]
 8: (BlueStore::_fsck_check_objects(BlueStore::FSCKDepth, BlueStore::FSCK_ObjectCtx&)+0x15ba) [0x56145011551a]
 9: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x1e7e) [0x561450119a1e]
 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x21c) [0x561450126dbc]
 11: main()
 12: __libc_start_main()
 13: _start()
2023-01-13T11:52:08.346+0000 7f5be5a6f540 -1 *** Caught signal (Aborted) **
 in thread 7f5be5a6f540 thread_name:ceph-bluestore-

 ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f5bd9e12cf0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f5bdae76e39]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7f5bdae77002]
 6: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x7cf) [0x5614500a826f]
 7: (BlueStore::fsck_check_objects_shallow(BlueStore::FSCKDepth, long, boost::intrusive_ptr<BlueStore::Collection>, ghobject_t const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list const&, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mempool::pool_allocator<(mempool::pool_index_t)11, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::map<boost::intrusive_ptr<BlueStore::Blob>, unsigned short, std::less<boost::intrusive_ptr<BlueStore::Blob> >, std::allocator<std::pair<boost::intrusive_ptr<BlueStore::Blob> const, unsigned short> > >*, BlueStore::FSCK_ObjectCtx const&)+0x234) [0x5614501115b4]
 8: (BlueStore::_fsck_check_objects(BlueStore::FSCKDepth, BlueStore::FSCK_ObjectCtx&)+0x15ba) [0x56145011551a]
 9: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x1e7e) [0x561450119a1e]
 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x21c) [0x561450126dbc]
 11: main()
 12: __libc_start_main()
 13: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

 -1894> 2023-01-13T11:52:08.343+0000 7f5be5a6f540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)' thread 7f5be5a6f540 time 2023-01-13T11:52:08.334952+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/os/bluestore/BlueStore.cc: 3228: FAILED ceph_assert(v.length() == p->shard_info->bytes)

 ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f5bdae76de8]
 2: /usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7f5bdae77002]
 3: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x7cf) [0x5614500a826f]
 4: (BlueStore::fsck_check_objects_shallow(BlueStore::FSCKDepth, long, boost::intrusive_ptr<BlueStore::Collection>, ghobject_t const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list const&, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mempool::pool_allocator<(mempool::pool_index_t)11, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::map<boost::intrusive_ptr<BlueStore::Blob>, unsigned short, std::less<boost::intrusive_ptr<BlueStore::Blob> >, std::allocator<std::pair<boost::intrusive_ptr<BlueStore::Blob> const, unsigned short> > >*, BlueStore::FSCK_ObjectCtx const&)+0x234) [0x5614501115b4]
 5: (BlueStore::_fsck_check_objects(BlueStore::FSCKDepth, BlueStore::FSCK_ObjectCtx&)+0x15ba) [0x56145011551a]
 6: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x1e7e) [0x561450119a1e]
 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x21c) [0x561450126dbc]
 8: main()
 9: __libc_start_main()
 10: _start()

 -1893> 2023-01-13T11:52:08.346+0000 7f5be5a6f540 -1 *** Caught signal (Aborted) **
 in thread 7f5be5a6f540 thread_name:ceph-bluestore-

 ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f5bd9e12cf0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f5bdae76e39]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7f5bdae77002]
 6: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x7cf) [0x5614500a826f]
 7: (BlueStore::fsck_check_objects_shallow(BlueStore::FSCKDepth, long, boost::intrusive_ptr<BlueStore::Collection>, ghobject_t const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list const&, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mempool::pool_allocator<(mempool::pool_index_t)11, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::map<boost::intrusive_ptr<BlueStore::Blob>, unsigned short, std::less<boost::intrusive_ptr<BlueStore::Blob> >, std::allocator<std::pair<boost::intrusive_ptr<BlueStore::Blob> const, unsigned short> > >*, BlueStore::FSCK_ObjectCtx const&)+0x234) [0x5614501115b4]
 8: (BlueStore::_fsck_check_objects(BlueStore::FSCKDepth, BlueStore::FSCK_ObjectCtx&)+0x15ba) [0x56145011551a]
 9: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x1e7e) [0x561450119a1e]
 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x21c) [0x561450126dbc]
 11: main()
 12: __libc_start_main()
 13: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

    -1> 2023-01-13T11:52:08.343+0000 7f5be5a6f540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)' thread 7f5be5a6f540 time 2023-01-13T11:52:08.334952+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/os/bluestore/BlueStore.cc: 3228: FAILED ceph_assert(v.length() == p->shard_info->bytes)

 ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f5bdae76de8]
 2: /usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7f5bdae77002]
 3: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x7cf) [0x5614500a826f]
 4: (BlueStore::fsck_check_objects_shallow(BlueStore::FSCKDepth, long, boost::intrusive_ptr<BlueStore::Collection>, ghobject_t const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list const&, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mempool::pool_allocator<(mempool::pool_index_t)11, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::map<boost::intrusive_ptr<BlueStore::Blob>, unsigned short, std::less<boost::intrusive_ptr<BlueStore::Blob> >, std::allocator<std::pair<boost::intrusive_ptr<BlueStore::Blob> const, unsigned short> > >*, BlueStore::FSCK_ObjectCtx const&)+0x234) [0x5614501115b4]
 5: (BlueStore::_fsck_check_objects(BlueStore::FSCKDepth, BlueStore::FSCK_ObjectCtx&)+0x15ba) [0x56145011551a]
 6: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x1e7e) [0x561450119a1e]
 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x21c) [0x561450126dbc]
 8: main()
 9: __libc_start_main()
 10: _start()

     0> 2023-01-13T11:52:08.346+0000 7f5be5a6f540 -1 *** Caught signal (Aborted) **
 in thread 7f5be5a6f540 thread_name:ceph-bluestore-

 ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f5bd9e12cf0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f5bdae76e39]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7f5bdae77002]
 6: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x7cf) [0x5614500a826f]
 7: (BlueStore::fsck_check_objects_shallow(BlueStore::FSCKDepth, long, boost::intrusive_ptr<BlueStore::Collection>, ghobject_t const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list const&, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mempool::pool_allocator<(mempool::pool_index_t)11, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::map<boost::intrusive_ptr<BlueStore::Blob>, unsigned short, std::less<boost::intrusive_ptr<BlueStore::Blob> >, std::allocator<std::pair<boost::intrusive_ptr<BlueStore::Blob> const, unsigned short> > >*, BlueStore::FSCK_ObjectCtx const&)+0x234) [0x5614501115b4]
 8: (BlueStore::_fsck_check_objects(BlueStore::FSCKDepth, BlueStore::FSCK_ObjectCtx&)+0x15ba) [0x56145011551a]
 9: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x1e7e) [0x561450119a1e]
 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x21c) [0x561450126dbc]
 11: main()
 12: __libc_start_main()
 13: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Aborted (core dumped)

No data to display

Actions
Copy link

Also available in: Atom PDF