Project

General

Profile

Bug #42223 » osd4-kvstore.txt

Tobias Fischer, 10/28/2019 03:26 PM

 
2019-10-28 16:23:27.947 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _mount path /var/lib/ceph/osd/ceph-4/
2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block type kernel
2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057e700 /var/lib/ceph/osd/ceph-4//block) open path /var/lib/ceph/osd/ceph-4//block
2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057e700 /var/lib/ceph/osd/ceph-4//block) open size 299964039168 (0x45d7400000, 279 GiB) block_size 4096 (4 KiB) rotational discard not supported
2019-10-28 16:23:27.947 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2
2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block.db type kernel
2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057ee00 /var/lib/ceph/osd/ceph-4//block.db) open path /var/lib/ceph/osd/ceph-4//block.db
2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057ee00 /var/lib/ceph/osd/ceph-4//block.db) open size 28989980672 (0x6bff00000, 27 GiB) block_size 4096 (4 KiB) non-rotational discard not supported
2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-4//block.db size 27 GiB
2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block type kernel
2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f180 /var/lib/ceph/osd/ceph-4//block) open path /var/lib/ceph/osd/ceph-4//block
2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f180 /var/lib/ceph/osd/ceph-4//block) open size 299964039168 (0x45d7400000, 279 GiB) block_size 4096 (4 KiB) rotational discard not supported
2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-4//block size 279 GiB
2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block.wal type kernel
2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f500 /var/lib/ceph/osd/ceph-4//block.wal) open path /var/lib/ceph/osd/ceph-4//block.wal
2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f500 /var/lib/ceph/osd/ceph-4//block.wal) open size 1073741824 (0x40000000, 1 GiB) block_size 4096 (4 KiB) non-rotational discard not supported
2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-4//block.wal size 1 GiB
2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs mount
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compaction_readahead_size = 2097152
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compression = kNoCompression
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_background_compactions = 2
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_write_buffer_number = 4
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option min_write_buffer_number_to_merge = 1
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option recycle_log_file_num = 4
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option writable_file_max_buffer_size = 0
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option write_buffer_size = 268435456
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compaction_readahead_size = 2097152
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compression = kNoCompression
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_background_compactions = 2
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_write_buffer_number = 4
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option min_write_buffer_number_to_merge = 1
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option recycle_log_file_num = 4
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option writable_file_max_buffer_size = 0
2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option write_buffer_size = 268435456
2019-10-28 16:23:27.963 7fb8ba257180 1 rocksdb: do_open column families: [default]
2019-10-28 16:23:29.255 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2
2019-10-28 16:23:29.283 7fb8ba257180 1 freelist init
2019-10-28 16:23:29.291 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_alloc opening allocation metadata
2019-10-28 16:23:29.295 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_alloc loaded 0 B in 0 extents
/data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb8ba257180 time 2019-10-28 16:23:29.297245
/data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: 750: FAILED ceph_assert(available >= allocated)
ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb8bb4e460e]
2: (()+0x2807e6) [0x7fb8bb4e47e6]
3: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163]
4: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95]
5: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2]
6: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a]
7: (StoreTool::load_bluestore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+0x42) [0x55e80c962f22]
8: (StoreTool::StoreTool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool)+0x226) [0x55e80c965f56]
9: (main()+0x28e) [0x55e80c93de4e]
10: (__libc_start_main()+0xeb) [0x7fb8ba76409b]
11: (_start()+0x2a) [0x55e80c962daa]
2019-10-28 16:23:29.295 7fb8ba257180 -1 /data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb8ba257180 time 2019-10-28 16:23:29.297245
/data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: 750: FAILED ceph_assert(available >= allocated)

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb8bb4e460e]
2: (()+0x2807e6) [0x7fb8bb4e47e6]
3: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163]
4: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95]
5: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2]
6: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a]
7: (StoreTool::load_bluestore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+0x42) [0x55e80c962f22]
8: (StoreTool::StoreTool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool)+0x226) [0x55e80c965f56]
9: (main()+0x28e) [0x55e80c93de4e]
10: (__libc_start_main()+0xeb) [0x7fb8ba76409b]
11: (_start()+0x2a) [0x55e80c962daa]

*** Caught signal (Aborted) **
in thread 7fb8ba257180 thread_name:ceph-kvstore-to
ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)
1: (()+0x12730) [0x7fb8bacbd730]
2: (gsignal()+0x10b) [0x7fb8ba7777bb]
3: (abort()+0x121) [0x7fb8ba762535]
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x7fb8bb4e465f]
5: (()+0x2807e6) [0x7fb8bb4e47e6]
6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163]
7: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95]
8: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2]
9: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a]
10: (StoreTool::load_bluestore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+0x42) [0x55e80c962f22]
11: (StoreTool::StoreTool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool)+0x226) [0x55e80c965f56]
12: (main()+0x28e) [0x55e80c93de4e]
13: (__libc_start_main()+0xeb) [0x7fb8ba76409b]
14: (_start()+0x2a) [0x55e80c962daa]
2019-10-28 16:23:29.295 7fb8ba257180 -1 *** Caught signal (Aborted) **
in thread 7fb8ba257180 thread_name:ceph-kvstore-to

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)
1: (()+0x12730) [0x7fb8bacbd730]
2: (gsignal()+0x10b) [0x7fb8ba7777bb]
3: (abort()+0x121) [0x7fb8ba762535]
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x7fb8bb4e465f]
5: (()+0x2807e6) [0x7fb8bb4e47e6]
6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163]
7: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95]
8: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2]
9: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a]
10: (StoreTool::load_bluestore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+0x42) [0x55e80c962f22]
11: (StoreTool::StoreTool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool)+0x226) [0x55e80c965f56]
12: (main()+0x28e) [0x55e80c93de4e]
13: (__libc_start_main()+0xeb) [0x7fb8ba76409b]
14: (_start()+0x2a) [0x55e80c962daa]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-68> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command assert hook 0x55e80f8ca090
-67> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command abort hook 0x55e80f8ca090
-66> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perfcounters_dump hook 0x55e80f8ca090
-65> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command 1 hook 0x55e80f8ca090
-64> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf dump hook 0x55e80f8ca090
-63> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perfcounters_schema hook 0x55e80f8ca090
-62> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf histogram dump hook 0x55e80f8ca090
-61> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command 2 hook 0x55e80f8ca090
-60> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf schema hook 0x55e80f8ca090
-59> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf histogram schema hook 0x55e80f8ca090
-58> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf reset hook 0x55e80f8ca090
-57> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config show hook 0x55e80f8ca090
-56> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config help hook 0x55e80f8ca090
-55> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config set hook 0x55e80f8ca090
-54> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config unset hook 0x55e80f8ca090
-53> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config get hook 0x55e80f8ca090
-52> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config diff hook 0x55e80f8ca090
-51> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config diff get hook 0x55e80f8ca090
-50> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command log flush hook 0x55e80f8ca090
-49> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command log dump hook 0x55e80f8ca090
-48> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command log reopen hook 0x55e80f8ca090
-47> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command dump_mempools hook 0x55e80f9aa068
-46> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _mount path /var/lib/ceph/osd/ceph-4/
-45> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block type kernel
-44> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057e700 /var/lib/ceph/osd/ceph-4//block) open path /var/lib/ceph/osd/ceph-4//block
-43> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057e700 /var/lib/ceph/osd/ceph-4//block) open size 299964039168 (0x45d7400000, 279 GiB) block_size 4096 (4 KiB) rotational discard not supported
-42> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2
-41> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block.db type kernel
-40> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057ee00 /var/lib/ceph/osd/ceph-4//block.db) open path /var/lib/ceph/osd/ceph-4//block.db
-39> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057ee00 /var/lib/ceph/osd/ceph-4//block.db) open size 28989980672 (0x6bff00000, 27 GiB) block_size 4096 (4 KiB) non-rotational discard not supported
-38> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-4//block.db size 27 GiB
-37> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block type kernel
-36> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f180 /var/lib/ceph/osd/ceph-4//block) open path /var/lib/ceph/osd/ceph-4//block
-35> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f180 /var/lib/ceph/osd/ceph-4//block) open size 299964039168 (0x45d7400000, 279 GiB) block_size 4096 (4 KiB) rotational discard not supported
-34> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-4//block size 279 GiB
-33> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block.wal type kernel
-32> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f500 /var/lib/ceph/osd/ceph-4//block.wal) open path /var/lib/ceph/osd/ceph-4//block.wal
-31> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f500 /var/lib/ceph/osd/ceph-4//block.wal) open size 1073741824 (0x40000000, 1 GiB) block_size 4096 (4 KiB) non-rotational discard not supported
-30> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-4//block.wal size 1 GiB
-29> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs mount
-28> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compaction_readahead_size = 2097152
-27> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compression = kNoCompression
-26> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_background_compactions = 2
-25> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_write_buffer_number = 4
-24> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option min_write_buffer_number_to_merge = 1
-23> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option recycle_log_file_num = 4
-22> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option writable_file_max_buffer_size = 0
-21> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option write_buffer_size = 268435456
-20> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compaction_readahead_size = 2097152
-19> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compression = kNoCompression
-18> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_background_compactions = 2
-17> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_write_buffer_number = 4
-16> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option min_write_buffer_number_to_merge = 1
-15> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option recycle_log_file_num = 4
-14> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option writable_file_max_buffer_size = 0
-13> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option write_buffer_size = 268435456
-12> 2019-10-28 16:23:27.963 7fb8ba257180 1 rocksdb: do_open column families: [default]
-11> 2019-10-28 16:23:29.255 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2
-10> 2019-10-28 16:23:29.283 7fb8ba257180 1 get compressor snappy = 0
-9> 2019-10-28 16:23:29.283 7fb8ba257180 1 load compressor snappy
-8> 2019-10-28 16:23:29.283 7fb8ba257180 1 add compressor snappy 0x55e80f98adb0
-7> 2019-10-28 16:23:29.283 7fb8ba257180 1 get compressor snappy = 0x55e80f98adb0
-6> 2019-10-28 16:23:29.283 7fb8ba257180 1 load: compressor snappy loaded and registered
-5> 2019-10-28 16:23:29.283 7fb8ba257180 1 get compressor snappy = 0x55e80f98adb0
-4> 2019-10-28 16:23:29.283 7fb8ba257180 1 freelist init
-3> 2019-10-28 16:23:29.291 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_alloc opening allocation metadata
-2> 2019-10-28 16:23:29.295 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_alloc loaded 0 B in 0 extents
-1> 2019-10-28 16:23:29.295 7fb8ba257180 -1 /data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb8ba257180 time 2019-10-28 16:23:29.297245
/data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: 750: FAILED ceph_assert(available >= allocated)

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb8bb4e460e]
2: (()+0x2807e6) [0x7fb8bb4e47e6]
3: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163]
4: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95]
5: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2]
6: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a]
7: (StoreTool::load_bluestore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+0x42) [0x55e80c962f22]
8: (StoreTool::StoreTool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool)+0x226) [0x55e80c965f56]
9: (main()+0x28e) [0x55e80c93de4e]
10: (__libc_start_main()+0xeb) [0x7fb8ba76409b]
11: (_start()+0x2a) [0x55e80c962daa]

0> 2019-10-28 16:23:29.295 7fb8ba257180 -1 *** Caught signal (Aborted) **
in thread 7fb8ba257180 thread_name:ceph-kvstore-to

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)
1: (()+0x12730) [0x7fb8bacbd730]
2: (gsignal()+0x10b) [0x7fb8ba7777bb]
3: (abort()+0x121) [0x7fb8ba762535]
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x7fb8bb4e465f]
5: (()+0x2807e6) [0x7fb8bb4e47e6]
6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163]
7: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95]
8: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2]
9: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a]
10: (StoreTool::load_bluestore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+0x42) [0x55e80c962f22]
11: (StoreTool::StoreTool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool)+0x226) [0x55e80c965f56]
12: (main()+0x28e) [0x55e80c93de4e]
13: (__libc_start_main()+0xeb) [0x7fb8ba76409b]
14: (_start()+0x2a) [0x55e80c962daa]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
2/ 2 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 500
max_new 1000
log_file
--- end dump of recent events ---
--- begin dump of recent events ---
-68> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command assert hook 0x55e80f8ca090
-67> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command abort hook 0x55e80f8ca090
-66> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perfcounters_dump hook 0x55e80f8ca090
-65> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command 1 hook 0x55e80f8ca090
-64> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf dump hook 0x55e80f8ca090
-63> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perfcounters_schema hook 0x55e80f8ca090
-62> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf histogram dump hook 0x55e80f8ca090
-61> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command 2 hook 0x55e80f8ca090
-60> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf schema hook 0x55e80f8ca090
-59> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf histogram schema hook 0x55e80f8ca090
-58> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf reset hook 0x55e80f8ca090
-57> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config show hook 0x55e80f8ca090
-56> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config help hook 0x55e80f8ca090
-55> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config set hook 0x55e80f8ca090
-54> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config unset hook 0x55e80f8ca090
-53> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config get hook 0x55e80f8ca090
-52> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config diff hook 0x55e80f8ca090
-51> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config diff get hook 0x55e80f8ca090
-50> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command log flush hook 0x55e80f8ca090
-49> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command log dump hook 0x55e80f8ca090
-48> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command log reopen hook 0x55e80f8ca090
-47> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command dump_mempools hook 0x55e80f9aa068
-46> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _mount path /var/lib/ceph/osd/ceph-4/
-45> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block type kernel
-44> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057e700 /var/lib/ceph/osd/ceph-4//block) open path /var/lib/ceph/osd/ceph-4//block
-43> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057e700 /var/lib/ceph/osd/ceph-4//block) open size 299964039168 (0x45d7400000, 279 GiB) block_size 4096 (4 KiB) rotational discard not supported
-42> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2
-41> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block.db type kernel
-40> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057ee00 /var/lib/ceph/osd/ceph-4//block.db) open path /var/lib/ceph/osd/ceph-4//block.db
-39> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057ee00 /var/lib/ceph/osd/ceph-4//block.db) open size 28989980672 (0x6bff00000, 27 GiB) block_size 4096 (4 KiB) non-rotational discard not supported
-38> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-4//block.db size 27 GiB
-37> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block type kernel
-36> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f180 /var/lib/ceph/osd/ceph-4//block) open path /var/lib/ceph/osd/ceph-4//block
-35> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f180 /var/lib/ceph/osd/ceph-4//block) open size 299964039168 (0x45d7400000, 279 GiB) block_size 4096 (4 KiB) rotational discard not supported
-34> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-4//block size 279 GiB
-33> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block.wal type kernel
-32> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f500 /var/lib/ceph/osd/ceph-4//block.wal) open path /var/lib/ceph/osd/ceph-4//block.wal
-31> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f500 /var/lib/ceph/osd/ceph-4//block.wal) open size 1073741824 (0x40000000, 1 GiB) block_size 4096 (4 KiB) non-rotational discard not supported
-30> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-4//block.wal size 1 GiB
-29> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs mount
-28> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compaction_readahead_size = 2097152
-27> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compression = kNoCompression
-26> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_background_compactions = 2
-25> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_write_buffer_number = 4
-24> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option min_write_buffer_number_to_merge = 1
-23> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option recycle_log_file_num = 4
-22> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option writable_file_max_buffer_size = 0
-21> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option write_buffer_size = 268435456
-20> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compaction_readahead_size = 2097152
-19> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compression = kNoCompression
-18> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_background_compactions = 2
-17> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_write_buffer_number = 4
-16> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option min_write_buffer_number_to_merge = 1
-15> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option recycle_log_file_num = 4
-14> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option writable_file_max_buffer_size = 0
-13> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option write_buffer_size = 268435456
-12> 2019-10-28 16:23:27.963 7fb8ba257180 1 rocksdb: do_open column families: [default]
-11> 2019-10-28 16:23:29.255 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2
-10> 2019-10-28 16:23:29.283 7fb8ba257180 1 get compressor snappy = 0
-9> 2019-10-28 16:23:29.283 7fb8ba257180 1 load compressor snappy
-8> 2019-10-28 16:23:29.283 7fb8ba257180 1 add compressor snappy 0x55e80f98adb0
-7> 2019-10-28 16:23:29.283 7fb8ba257180 1 get compressor snappy = 0x55e80f98adb0
-6> 2019-10-28 16:23:29.283 7fb8ba257180 1 load: compressor snappy loaded and registered
-5> 2019-10-28 16:23:29.283 7fb8ba257180 1 get compressor snappy = 0x55e80f98adb0
-4> 2019-10-28 16:23:29.283 7fb8ba257180 1 freelist init
-3> 2019-10-28 16:23:29.291 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_alloc opening allocation metadata
-2> 2019-10-28 16:23:29.295 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_alloc loaded 0 B in 0 extents
-1> 2019-10-28 16:23:29.295 7fb8ba257180 -1 /data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb8ba257180 time 2019-10-28 16:23:29.297245
/data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: 750: FAILED ceph_assert(available >= allocated)

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb8bb4e460e]
2: (()+0x2807e6) [0x7fb8bb4e47e6]
3: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163]
4: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95]
5: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2]
6: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a]
7: (StoreTool::load_bluestore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+0x42) [0x55e80c962f22]
8: (StoreTool::StoreTool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool)+0x226) [0x55e80c965f56]
9: (main()+0x28e) [0x55e80c93de4e]
10: (__libc_start_main()+0xeb) [0x7fb8ba76409b]
11: (_start()+0x2a) [0x55e80c962daa]

0> 2019-10-28 16:23:29.295 7fb8ba257180 -1 *** Caught signal (Aborted) **
in thread 7fb8ba257180 thread_name:ceph-kvstore-to

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)
1: (()+0x12730) [0x7fb8bacbd730]
2: (gsignal()+0x10b) [0x7fb8ba7777bb]
3: (abort()+0x121) [0x7fb8ba762535]
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x7fb8bb4e465f]
5: (()+0x2807e6) [0x7fb8bb4e47e6]
6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163]
7: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95]
8: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2]
9: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a]
10: (StoreTool::load_bluestore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+0x42) [0x55e80c962f22]
11: (StoreTool::StoreTool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool)+0x226) [0x55e80c965f56]
12: (main()+0x28e) [0x55e80c93de4e]
13: (__libc_start_main()+0xeb) [0x7fb8ba76409b]
14: (_start()+0x2a) [0x55e80c962daa]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
2/ 2 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 500
max_new 1000
log_file /var/lib/ceph/crash/2019-10-28_15:23:29.299670Z_9b7c305a-8772-43d0-8a4b-d42b5f121362/log
--- end dump of recent events ---
(14-14/31)