2019-10-28 16:23:27.947 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _mount path /var/lib/ceph/osd/ceph-4/ 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block type kernel 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057e700 /var/lib/ceph/osd/ceph-4//block) open path /var/lib/ceph/osd/ceph-4//block 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057e700 /var/lib/ceph/osd/ceph-4//block) open size 299964039168 (0x45d7400000, 279 GiB) block_size 4096 (4 KiB) rotational discard not supported 2019-10-28 16:23:27.947 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block.db type kernel 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057ee00 /var/lib/ceph/osd/ceph-4//block.db) open path /var/lib/ceph/osd/ceph-4//block.db 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057ee00 /var/lib/ceph/osd/ceph-4//block.db) open size 28989980672 (0x6bff00000, 27 GiB) block_size 4096 (4 KiB) non-rotational discard not supported 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-4//block.db size 27 GiB 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block type kernel 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f180 /var/lib/ceph/osd/ceph-4//block) open path /var/lib/ceph/osd/ceph-4//block 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f180 /var/lib/ceph/osd/ceph-4//block) open size 299964039168 (0x45d7400000, 279 GiB) block_size 4096 (4 KiB) rotational discard not supported 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-4//block size 279 GiB 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block.wal type kernel 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f500 /var/lib/ceph/osd/ceph-4//block.wal) open path /var/lib/ceph/osd/ceph-4//block.wal 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f500 /var/lib/ceph/osd/ceph-4//block.wal) open size 1073741824 (0x40000000, 1 GiB) block_size 4096 (4 KiB) non-rotational discard not supported 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-4//block.wal size 1 GiB 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs mount 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compaction_readahead_size = 2097152 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compression = kNoCompression 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_background_compactions = 2 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_write_buffer_number = 4 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option min_write_buffer_number_to_merge = 1 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option recycle_log_file_num = 4 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option writable_file_max_buffer_size = 0 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option write_buffer_size = 268435456 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compaction_readahead_size = 2097152 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compression = kNoCompression 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_background_compactions = 2 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_write_buffer_number = 4 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option min_write_buffer_number_to_merge = 1 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option recycle_log_file_num = 4 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option writable_file_max_buffer_size = 0 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option write_buffer_size = 268435456 2019-10-28 16:23:27.963 7fb8ba257180 1 rocksdb: do_open column families: [default] 2019-10-28 16:23:29.255 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2 2019-10-28 16:23:29.283 7fb8ba257180 1 freelist init 2019-10-28 16:23:29.291 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_alloc opening allocation metadata 2019-10-28 16:23:29.295 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_alloc loaded 0 B in 0 extents /data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb8ba257180 time 2019-10-28 16:23:29.297245 /data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: 750: FAILED ceph_assert(available >= allocated) ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb8bb4e460e] 2: (()+0x2807e6) [0x7fb8bb4e47e6] 3: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163] 4: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95] 5: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2] 6: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a] 7: (StoreTool::load_bluestore(std::__cxx11::basic_string, std::allocator > const&, bool)+0x42) [0x55e80c962f22] 8: (StoreTool::StoreTool(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool)+0x226) [0x55e80c965f56] 9: (main()+0x28e) [0x55e80c93de4e] 10: (__libc_start_main()+0xeb) [0x7fb8ba76409b] 11: (_start()+0x2a) [0x55e80c962daa] 2019-10-28 16:23:29.295 7fb8ba257180 -1 /data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb8ba257180 time 2019-10-28 16:23:29.297245 /data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: 750: FAILED ceph_assert(available >= allocated) ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb8bb4e460e] 2: (()+0x2807e6) [0x7fb8bb4e47e6] 3: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163] 4: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95] 5: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2] 6: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a] 7: (StoreTool::load_bluestore(std::__cxx11::basic_string, std::allocator > const&, bool)+0x42) [0x55e80c962f22] 8: (StoreTool::StoreTool(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool)+0x226) [0x55e80c965f56] 9: (main()+0x28e) [0x55e80c93de4e] 10: (__libc_start_main()+0xeb) [0x7fb8ba76409b] 11: (_start()+0x2a) [0x55e80c962daa] *** Caught signal (Aborted) ** in thread 7fb8ba257180 thread_name:ceph-kvstore-to ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) 1: (()+0x12730) [0x7fb8bacbd730] 2: (gsignal()+0x10b) [0x7fb8ba7777bb] 3: (abort()+0x121) [0x7fb8ba762535] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x7fb8bb4e465f] 5: (()+0x2807e6) [0x7fb8bb4e47e6] 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163] 7: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95] 8: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2] 9: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a] 10: (StoreTool::load_bluestore(std::__cxx11::basic_string, std::allocator > const&, bool)+0x42) [0x55e80c962f22] 11: (StoreTool::StoreTool(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool)+0x226) [0x55e80c965f56] 12: (main()+0x28e) [0x55e80c93de4e] 13: (__libc_start_main()+0xeb) [0x7fb8ba76409b] 14: (_start()+0x2a) [0x55e80c962daa] 2019-10-28 16:23:29.295 7fb8ba257180 -1 *** Caught signal (Aborted) ** in thread 7fb8ba257180 thread_name:ceph-kvstore-to ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) 1: (()+0x12730) [0x7fb8bacbd730] 2: (gsignal()+0x10b) [0x7fb8ba7777bb] 3: (abort()+0x121) [0x7fb8ba762535] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x7fb8bb4e465f] 5: (()+0x2807e6) [0x7fb8bb4e47e6] 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163] 7: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95] 8: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2] 9: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a] 10: (StoreTool::load_bluestore(std::__cxx11::basic_string, std::allocator > const&, bool)+0x42) [0x55e80c962f22] 11: (StoreTool::StoreTool(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool)+0x226) [0x55e80c965f56] 12: (main()+0x28e) [0x55e80c93de4e] 13: (__libc_start_main()+0xeb) [0x7fb8ba76409b] 14: (_start()+0x2a) [0x55e80c962daa] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -68> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command assert hook 0x55e80f8ca090 -67> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command abort hook 0x55e80f8ca090 -66> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perfcounters_dump hook 0x55e80f8ca090 -65> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command 1 hook 0x55e80f8ca090 -64> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf dump hook 0x55e80f8ca090 -63> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perfcounters_schema hook 0x55e80f8ca090 -62> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf histogram dump hook 0x55e80f8ca090 -61> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command 2 hook 0x55e80f8ca090 -60> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf schema hook 0x55e80f8ca090 -59> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf histogram schema hook 0x55e80f8ca090 -58> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf reset hook 0x55e80f8ca090 -57> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config show hook 0x55e80f8ca090 -56> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config help hook 0x55e80f8ca090 -55> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config set hook 0x55e80f8ca090 -54> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config unset hook 0x55e80f8ca090 -53> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config get hook 0x55e80f8ca090 -52> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config diff hook 0x55e80f8ca090 -51> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config diff get hook 0x55e80f8ca090 -50> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command log flush hook 0x55e80f8ca090 -49> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command log dump hook 0x55e80f8ca090 -48> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command log reopen hook 0x55e80f8ca090 -47> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command dump_mempools hook 0x55e80f9aa068 -46> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _mount path /var/lib/ceph/osd/ceph-4/ -45> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block type kernel -44> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057e700 /var/lib/ceph/osd/ceph-4//block) open path /var/lib/ceph/osd/ceph-4//block -43> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057e700 /var/lib/ceph/osd/ceph-4//block) open size 299964039168 (0x45d7400000, 279 GiB) block_size 4096 (4 KiB) rotational discard not supported -42> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2 -41> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block.db type kernel -40> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057ee00 /var/lib/ceph/osd/ceph-4//block.db) open path /var/lib/ceph/osd/ceph-4//block.db -39> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057ee00 /var/lib/ceph/osd/ceph-4//block.db) open size 28989980672 (0x6bff00000, 27 GiB) block_size 4096 (4 KiB) non-rotational discard not supported -38> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-4//block.db size 27 GiB -37> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block type kernel -36> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f180 /var/lib/ceph/osd/ceph-4//block) open path /var/lib/ceph/osd/ceph-4//block -35> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f180 /var/lib/ceph/osd/ceph-4//block) open size 299964039168 (0x45d7400000, 279 GiB) block_size 4096 (4 KiB) rotational discard not supported -34> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-4//block size 279 GiB -33> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block.wal type kernel -32> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f500 /var/lib/ceph/osd/ceph-4//block.wal) open path /var/lib/ceph/osd/ceph-4//block.wal -31> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f500 /var/lib/ceph/osd/ceph-4//block.wal) open size 1073741824 (0x40000000, 1 GiB) block_size 4096 (4 KiB) non-rotational discard not supported -30> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-4//block.wal size 1 GiB -29> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs mount -28> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compaction_readahead_size = 2097152 -27> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compression = kNoCompression -26> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_background_compactions = 2 -25> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_write_buffer_number = 4 -24> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option min_write_buffer_number_to_merge = 1 -23> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option recycle_log_file_num = 4 -22> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option writable_file_max_buffer_size = 0 -21> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option write_buffer_size = 268435456 -20> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compaction_readahead_size = 2097152 -19> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compression = kNoCompression -18> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_background_compactions = 2 -17> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_write_buffer_number = 4 -16> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option min_write_buffer_number_to_merge = 1 -15> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option recycle_log_file_num = 4 -14> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option writable_file_max_buffer_size = 0 -13> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option write_buffer_size = 268435456 -12> 2019-10-28 16:23:27.963 7fb8ba257180 1 rocksdb: do_open column families: [default] -11> 2019-10-28 16:23:29.255 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2 -10> 2019-10-28 16:23:29.283 7fb8ba257180 1 get compressor snappy = 0 -9> 2019-10-28 16:23:29.283 7fb8ba257180 1 load compressor snappy -8> 2019-10-28 16:23:29.283 7fb8ba257180 1 add compressor snappy 0x55e80f98adb0 -7> 2019-10-28 16:23:29.283 7fb8ba257180 1 get compressor snappy = 0x55e80f98adb0 -6> 2019-10-28 16:23:29.283 7fb8ba257180 1 load: compressor snappy loaded and registered -5> 2019-10-28 16:23:29.283 7fb8ba257180 1 get compressor snappy = 0x55e80f98adb0 -4> 2019-10-28 16:23:29.283 7fb8ba257180 1 freelist init -3> 2019-10-28 16:23:29.291 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_alloc opening allocation metadata -2> 2019-10-28 16:23:29.295 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_alloc loaded 0 B in 0 extents -1> 2019-10-28 16:23:29.295 7fb8ba257180 -1 /data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb8ba257180 time 2019-10-28 16:23:29.297245 /data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: 750: FAILED ceph_assert(available >= allocated) ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb8bb4e460e] 2: (()+0x2807e6) [0x7fb8bb4e47e6] 3: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163] 4: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95] 5: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2] 6: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a] 7: (StoreTool::load_bluestore(std::__cxx11::basic_string, std::allocator > const&, bool)+0x42) [0x55e80c962f22] 8: (StoreTool::StoreTool(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool)+0x226) [0x55e80c965f56] 9: (main()+0x28e) [0x55e80c93de4e] 10: (__libc_start_main()+0xeb) [0x7fb8ba76409b] 11: (_start()+0x2a) [0x55e80c962daa] 0> 2019-10-28 16:23:29.295 7fb8ba257180 -1 *** Caught signal (Aborted) ** in thread 7fb8ba257180 thread_name:ceph-kvstore-to ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) 1: (()+0x12730) [0x7fb8bacbd730] 2: (gsignal()+0x10b) [0x7fb8ba7777bb] 3: (abort()+0x121) [0x7fb8ba762535] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x7fb8bb4e465f] 5: (()+0x2807e6) [0x7fb8bb4e47e6] 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163] 7: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95] 8: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2] 9: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a] 10: (StoreTool::load_bluestore(std::__cxx11::basic_string, std::allocator > const&, bool)+0x42) [0x55e80c962f22] 11: (StoreTool::StoreTool(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool)+0x226) [0x55e80c965f56] 12: (main()+0x28e) [0x55e80c93de4e] 13: (__libc_start_main()+0xeb) [0x7fb8ba76409b] 14: (_start()+0x2a) [0x55e80c962daa] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 2/ 2 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 500 max_new 1000 log_file --- end dump of recent events --- --- begin dump of recent events --- -68> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command assert hook 0x55e80f8ca090 -67> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command abort hook 0x55e80f8ca090 -66> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perfcounters_dump hook 0x55e80f8ca090 -65> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command 1 hook 0x55e80f8ca090 -64> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf dump hook 0x55e80f8ca090 -63> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perfcounters_schema hook 0x55e80f8ca090 -62> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf histogram dump hook 0x55e80f8ca090 -61> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command 2 hook 0x55e80f8ca090 -60> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf schema hook 0x55e80f8ca090 -59> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf histogram schema hook 0x55e80f8ca090 -58> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command perf reset hook 0x55e80f8ca090 -57> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config show hook 0x55e80f8ca090 -56> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config help hook 0x55e80f8ca090 -55> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config set hook 0x55e80f8ca090 -54> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config unset hook 0x55e80f8ca090 -53> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config get hook 0x55e80f8ca090 -52> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config diff hook 0x55e80f8ca090 -51> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command config diff get hook 0x55e80f8ca090 -50> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command log flush hook 0x55e80f8ca090 -49> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command log dump hook 0x55e80f8ca090 -48> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command log reopen hook 0x55e80f8ca090 -47> 2019-10-28 16:23:27.939 7fb8ba257180 5 asok(0x55e80fab4000) register_command dump_mempools hook 0x55e80f9aa068 -46> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _mount path /var/lib/ceph/osd/ceph-4/ -45> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block type kernel -44> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057e700 /var/lib/ceph/osd/ceph-4//block) open path /var/lib/ceph/osd/ceph-4//block -43> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057e700 /var/lib/ceph/osd/ceph-4//block) open size 299964039168 (0x45d7400000, 279 GiB) block_size 4096 (4 KiB) rotational discard not supported -42> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2 -41> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block.db type kernel -40> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057ee00 /var/lib/ceph/osd/ceph-4//block.db) open path /var/lib/ceph/osd/ceph-4//block.db -39> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057ee00 /var/lib/ceph/osd/ceph-4//block.db) open size 28989980672 (0x6bff00000, 27 GiB) block_size 4096 (4 KiB) non-rotational discard not supported -38> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-4//block.db size 27 GiB -37> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block type kernel -36> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f180 /var/lib/ceph/osd/ceph-4//block) open path /var/lib/ceph/osd/ceph-4//block -35> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f180 /var/lib/ceph/osd/ceph-4//block) open size 299964039168 (0x45d7400000, 279 GiB) block_size 4096 (4 KiB) rotational discard not supported -34> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-4//block size 279 GiB -33> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev create path /var/lib/ceph/osd/ceph-4//block.wal type kernel -32> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f500 /var/lib/ceph/osd/ceph-4//block.wal) open path /var/lib/ceph/osd/ceph-4//block.wal -31> 2019-10-28 16:23:27.947 7fb8ba257180 1 bdev(0x55e81057f500 /var/lib/ceph/osd/ceph-4//block.wal) open size 1073741824 (0x40000000, 1 GiB) block_size 4096 (4 KiB) non-rotational discard not supported -30> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-4//block.wal size 1 GiB -29> 2019-10-28 16:23:27.947 7fb8ba257180 1 bluefs mount -28> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compaction_readahead_size = 2097152 -27> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compression = kNoCompression -26> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_background_compactions = 2 -25> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_write_buffer_number = 4 -24> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option min_write_buffer_number_to_merge = 1 -23> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option recycle_log_file_num = 4 -22> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option writable_file_max_buffer_size = 0 -21> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option write_buffer_size = 268435456 -20> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compaction_readahead_size = 2097152 -19> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option compression = kNoCompression -18> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_background_compactions = 2 -17> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option max_write_buffer_number = 4 -16> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option min_write_buffer_number_to_merge = 1 -15> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option recycle_log_file_num = 4 -14> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option writable_file_max_buffer_size = 0 -13> 2019-10-28 16:23:27.959 7fb8ba257180 0 set rocksdb option write_buffer_size = 268435456 -12> 2019-10-28 16:23:27.963 7fb8ba257180 1 rocksdb: do_open column families: [default] -11> 2019-10-28 16:23:29.255 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2 -10> 2019-10-28 16:23:29.283 7fb8ba257180 1 get compressor snappy = 0 -9> 2019-10-28 16:23:29.283 7fb8ba257180 1 load compressor snappy -8> 2019-10-28 16:23:29.283 7fb8ba257180 1 add compressor snappy 0x55e80f98adb0 -7> 2019-10-28 16:23:29.283 7fb8ba257180 1 get compressor snappy = 0x55e80f98adb0 -6> 2019-10-28 16:23:29.283 7fb8ba257180 1 load: compressor snappy loaded and registered -5> 2019-10-28 16:23:29.283 7fb8ba257180 1 get compressor snappy = 0x55e80f98adb0 -4> 2019-10-28 16:23:29.283 7fb8ba257180 1 freelist init -3> 2019-10-28 16:23:29.291 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_alloc opening allocation metadata -2> 2019-10-28 16:23:29.295 7fb8ba257180 1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_alloc loaded 0 B in 0 extents -1> 2019-10-28 16:23:29.295 7fb8ba257180 -1 /data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb8ba257180 time 2019-10-28 16:23:29.297245 /data/Debian/WORKDIR/ceph-14.2.4/src/os/bluestore/fastbmap_allocator_impl.h: 750: FAILED ceph_assert(available >= allocated) ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb8bb4e460e] 2: (()+0x2807e6) [0x7fb8bb4e47e6] 3: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163] 4: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95] 5: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2] 6: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a] 7: (StoreTool::load_bluestore(std::__cxx11::basic_string, std::allocator > const&, bool)+0x42) [0x55e80c962f22] 8: (StoreTool::StoreTool(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool)+0x226) [0x55e80c965f56] 9: (main()+0x28e) [0x55e80c93de4e] 10: (__libc_start_main()+0xeb) [0x7fb8ba76409b] 11: (_start()+0x2a) [0x55e80c962daa] 0> 2019-10-28 16:23:29.295 7fb8ba257180 -1 *** Caught signal (Aborted) ** in thread 7fb8ba257180 thread_name:ceph-kvstore-to ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) 1: (()+0x12730) [0x7fb8bacbd730] 2: (gsignal()+0x10b) [0x7fb8ba7777bb] 3: (abort()+0x121) [0x7fb8ba762535] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x7fb8bb4e465f] 5: (()+0x2807e6) [0x7fb8bb4e47e6] 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x733) [0x55e80cb6c163] 7: (BlueStore::_open_alloc()+0x1f5) [0x55e80c992f95] 8: (BlueStore::_open_db_and_around(bool)+0xa2) [0x55e80c9ab3f2] 9: (BlueStore::_mount(bool, bool)+0x58a) [0x55e80c9f162a] 10: (StoreTool::load_bluestore(std::__cxx11::basic_string, std::allocator > const&, bool)+0x42) [0x55e80c962f22] 11: (StoreTool::StoreTool(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool)+0x226) [0x55e80c965f56] 12: (main()+0x28e) [0x55e80c93de4e] 13: (__libc_start_main()+0xeb) [0x7fb8ba76409b] 14: (_start()+0x2a) [0x55e80c962daa] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 2/ 2 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 500 max_new 1000 log_file /var/lib/ceph/crash/2019-10-28_15:23:29.299670Z_9b7c305a-8772-43d0-8a4b-d42b5f121362/log --- end dump of recent events ---