Feb 09 10:25:37 ceph1 ceph-osd[360132]: 2019-02-09 10:25:37.191 7f6820628240 -1 Public network was set, but cluster network was not set
Feb 09 10:25:37 ceph1 ceph-osd[360132]: 2019-02-09 10:25:37.191 7f6820628240 -1     Using public network also for cluster network
Feb 09 10:25:37 ceph1 ceph-osd[360132]: starting osd.3 at - osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
Feb 09 10:26:28 ceph1 ceph-osd[360132]: /build/ceph-13.2.4/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)' thread 7f6808b33700 time 2019-02-09 10:26:28.224558
Feb 09 10:26:28 ceph1 ceph-osd[360132]: /build/ceph-13.2.4/src/os/bluestore/BlueFS.cc: 1048: FAILED assert(r == 0)
Feb 09 10:26:28 ceph1 ceph-osd[360132]: 2019-02-09 10:26:28.224 7f6808b33700 -1 bdev(0x557a2b01ca80 /var/lib/ceph/osd/ceph-3/block) read_random 0x29f4c849000~1b7000 error: (5) Input/output error
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable)
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f6817a483e2]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  2: (()+0x26d5a7) [0x7f6817a485a7]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  3: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x668) [0x557a29665f78]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  4: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x557a2968bee0]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  5: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x1a08) [0x557a2987ecd8]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  6: (rocksdb::BlockFetcher::ReadBlockContents()+0x3be) [0x557a29839a5e]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  7: (()+0xa98efd) [0x557a29827efd]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  8: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::GetContext*, rocksdb::Status)+0x4c7) [0x557a2982ae07]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  9: (rocksdb::BlockBasedTableIterator::InitDataBlock()+0xbd) [0x557a2982b62d]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  10: (rocksdb::BlockBasedTableIterator::FindKeyForward()+0x82) [0x557a2982b862]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  11: (()+0xa2b967) [0x557a297ba967]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  12: (rocksdb::MergingIterator::Next()+0x92) [0x557a2984e7d2]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  13: (rocksdb::CompactionIterator::Next()+0x1a2) [0x557a298c0642]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  14: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x569) [0x557a298c8899]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  15: (rocksdb::CompactionJob::Run()+0x314) [0x557a298ca264]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  16: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*)+0xc54) [0x557a2971aef4]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  17: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x397) [0x557a2971d807]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  18: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x97) [0x557a2971df87]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  19: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x266) [0x557a2990ab96]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  20: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x47) [0x557a2990ad17]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  21: (()+0xbd57f) [0x7f6815a4d57f]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  22: (()+0x76db) [0x7f68161416db]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  23: (clone()+0x3f) [0x7f681510a88f]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Feb 09 10:26:28 ceph1 ceph-osd[360132]: 2019-02-09 10:26:28.224 7f6808b33700 -1 /build/ceph-13.2.4/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)' thread 7f6808b33700 time 2019-02-09 10:26:28.224558
Feb 09 10:26:28 ceph1 ceph-osd[360132]: /build/ceph-13.2.4/src/os/bluestore/BlueFS.cc: 1048: FAILED assert(r == 0)
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable)
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f6817a483e2]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  2: (()+0x26d5a7) [0x7f6817a485a7]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  3: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x668) [0x557a29665f78]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  4: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x557a2968bee0]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  5: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x1a08) [0x557a2987ecd8]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  6: (rocksdb::BlockFetcher::ReadBlockContents()+0x3be) [0x557a29839a5e]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  7: (()+0xa98efd) [0x557a29827efd]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  8: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::GetContext*, rocksdb::Status)+0x4c7) [0x557a2982ae07]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  9: (rocksdb::BlockBasedTableIterator::InitDataBlock()+0xbd) [0x557a2982b62d]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  10: (rocksdb::BlockBasedTableIterator::FindKeyForward()+0x82) [0x557a2982b862]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  11: (()+0xa2b967) [0x557a297ba967]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  12: (rocksdb::MergingIterator::Next()+0x92) [0x557a2984e7d2]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  13: (rocksdb::CompactionIterator::Next()+0x1a2) [0x557a298c0642]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  14: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x569) [0x557a298c8899]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  15: (rocksdb::CompactionJob::Run()+0x314) [0x557a298ca264]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  16: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*)+0xc54) [0x557a2971aef4]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  17: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x397) [0x557a2971d807]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  18: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x97) [0x557a2971df87]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  19: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x266) [0x557a2990ab96]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  20: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x47) [0x557a2990ad17]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  21: (()+0xbd57f) [0x7f6815a4d57f]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  22: (()+0x76db) [0x7f68161416db]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  23: (clone()+0x3f) [0x7f681510a88f]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Feb 09 10:26:28 ceph1 ceph-osd[360132]:   -269> 2019-02-09 10:25:37.191 7f6820628240 -1 Public network was set, but cluster network was not set
Feb 09 10:26:28 ceph1 ceph-osd[360132]:   -268> 2019-02-09 10:25:37.191 7f6820628240 -1     Using public network also for cluster network
Feb 09 10:26:28 ceph1 ceph-osd[360132]:     -1> 2019-02-09 10:26:28.224 7f6808b33700 -1 bdev(0x557a2b01ca80 /var/lib/ceph/osd/ceph-3/block) read_random 0x29f4c849000~1b7000 error: (5) Input/output error
Feb 09 10:26:28 ceph1 ceph-osd[360132]:      0> 2019-02-09 10:26:28.224 7f6808b33700 -1 /build/ceph-13.2.4/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)' thread 7f6808b33700 time 2019-02-09 10:26:28.224558
Feb 09 10:26:28 ceph1 ceph-osd[360132]: /build/ceph-13.2.4/src/os/bluestore/BlueFS.cc: 1048: FAILED assert(r == 0)
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable)
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f6817a483e2]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  2: (()+0x26d5a7) [0x7f6817a485a7]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  3: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x668) [0x557a29665f78]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  4: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x557a2968bee0]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  5: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x1a08) [0x557a2987ecd8]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  6: (rocksdb::BlockFetcher::ReadBlockContents()+0x3be) [0x557a29839a5e]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  7: (()+0xa98efd) [0x557a29827efd]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  8: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::GetContext*, rocksdb::Status)+0x4c7) [0x557a2982ae07]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  9: (rocksdb::BlockBasedTableIterator::InitDataBlock()+0xbd) [0x557a2982b62d]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  10: (rocksdb::BlockBasedTableIterator::FindKeyForward()+0x82) [0x557a2982b862]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  11: (()+0xa2b967) [0x557a297ba967]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  12: (rocksdb::MergingIterator::Next()+0x92) [0x557a2984e7d2]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  13: (rocksdb::CompactionIterator::Next()+0x1a2) [0x557a298c0642]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  14: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x569) [0x557a298c8899]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  15: (rocksdb::CompactionJob::Run()+0x314) [0x557a298ca264]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  16: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*)+0xc54) [0x557a2971aef4]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  17: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x397) [0x557a2971d807]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  18: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x97) [0x557a2971df87]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  19: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x266) [0x557a2990ab96]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  20: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x47) [0x557a2990ad17]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  21: (()+0xbd57f) [0x7f6815a4d57f]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  22: (()+0x76db) [0x7f68161416db]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  23: (clone()+0x3f) [0x7f681510a88f]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Feb 09 10:26:28 ceph1 ceph-osd[360132]: *** Caught signal (Aborted) **
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  in thread 7f6808b33700 thread_name:rocksdb:bg0
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable)
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  1: (()+0x92aa40) [0x557a296b9a40]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  2: (()+0x12890) [0x7f681614c890]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  3: (gsignal()+0xc7) [0x7f6815027e97]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  4: (abort()+0x141) [0x7f6815029801]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f6817a48530]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  6: (()+0x26d5a7) [0x7f6817a485a7]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  7: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x668) [0x557a29665f78]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  8: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x557a2968bee0]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  9: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x1a08) [0x557a2987ecd8]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  10: (rocksdb::BlockFetcher::ReadBlockContents()+0x3be) [0x557a29839a5e]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  11: (()+0xa98efd) [0x557a29827efd]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  12: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::GetContext*, rocksdb::Status)+0x4c7) [0x557a2982ae07]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  13: (rocksdb::BlockBasedTableIterator::InitDataBlock()+0xbd) [0x557a2982b62d]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  14: (rocksdb::BlockBasedTableIterator::FindKeyForward()+0x82) [0x557a2982b862]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  15: (()+0xa2b967) [0x557a297ba967]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  16: (rocksdb::MergingIterator::Next()+0x92) [0x557a2984e7d2]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  17: (rocksdb::CompactionIterator::Next()+0x1a2) [0x557a298c0642]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  18: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x569) [0x557a298c8899]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  19: (rocksdb::CompactionJob::Run()+0x314) [0x557a298ca264]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  20: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*)+0xc54) [0x557a2971aef4]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  21: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x397) [0x557a2971d807]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  22: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x97) [0x557a2971df87]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  23: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x266) [0x557a2990ab96]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  24: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x47) [0x557a2990ad17]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  25: (()+0xbd57f) [0x7f6815a4d57f]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  26: (()+0x76db) [0x7f68161416db]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  27: (clone()+0x3f) [0x7f681510a88f]
Feb 09 10:26:28 ceph1 ceph-osd[360132]: 2019-02-09 10:26:28.232 7f6808b33700 -1 *** Caught signal (Aborted) **
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  in thread 7f6808b33700 thread_name:rocksdb:bg0
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable)
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  1: (()+0x92aa40) [0x557a296b9a40]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  2: (()+0x12890) [0x7f681614c890]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  3: (gsignal()+0xc7) [0x7f6815027e97]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  4: (abort()+0x141) [0x7f6815029801]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f6817a48530]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  6: (()+0x26d5a7) [0x7f6817a485a7]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  7: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x668) [0x557a29665f78]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  8: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x557a2968bee0]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  9: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x1a08) [0x557a2987ecd8]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  10: (rocksdb::BlockFetcher::ReadBlockContents()+0x3be) [0x557a29839a5e]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  11: (()+0xa98efd) [0x557a29827efd]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  12: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::GetContext*, rocksdb::Status)+0x4c7) [0x557a2982ae07]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  13: (rocksdb::BlockBasedTableIterator::InitDataBlock()+0xbd) [0x557a2982b62d]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  14: (rocksdb::BlockBasedTableIterator::FindKeyForward()+0x82) [0x557a2982b862]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  15: (()+0xa2b967) [0x557a297ba967]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  16: (rocksdb::MergingIterator::Next()+0x92) [0x557a2984e7d2]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  17: (rocksdb::CompactionIterator::Next()+0x1a2) [0x557a298c0642]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  18: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x569) [0x557a298c8899]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  19: (rocksdb::CompactionJob::Run()+0x314) [0x557a298ca264]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  20: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*)+0xc54) [0x557a2971aef4]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  21: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x397) [0x557a2971d807]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  22: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x97) [0x557a2971df87]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  23: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x266) [0x557a2990ab96]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  24: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x47) [0x557a2990ad17]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  25: (()+0xbd57f) [0x7f6815a4d57f]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  26: (()+0x76db) [0x7f68161416db]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  27: (clone()+0x3f) [0x7f681510a88f]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Feb 09 10:26:28 ceph1 ceph-osd[360132]:      0> 2019-02-09 10:26:28.232 7f6808b33700 -1 *** Caught signal (Aborted) **
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  in thread 7f6808b33700 thread_name:rocksdb:bg0
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable)
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  1: (()+0x92aa40) [0x557a296b9a40]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  2: (()+0x12890) [0x7f681614c890]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  3: (gsignal()+0xc7) [0x7f6815027e97]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  4: (abort()+0x141) [0x7f6815029801]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f6817a48530]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  6: (()+0x26d5a7) [0x7f6817a485a7]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  7: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x668) [0x557a29665f78]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  8: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x557a2968bee0]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  9: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x1a08) [0x557a2987ecd8]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  10: (rocksdb::BlockFetcher::ReadBlockContents()+0x3be) [0x557a29839a5e]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  11: (()+0xa98efd) [0x557a29827efd]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  12: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::GetContext*, rocksdb::Status)+0x4c7) [0x557a2982ae07]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  13: (rocksdb::BlockBasedTableIterator::InitDataBlock()+0xbd) [0x557a2982b62d]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  14: (rocksdb::BlockBasedTableIterator::FindKeyForward()+0x82) [0x557a2982b862]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  15: (()+0xa2b967) [0x557a297ba967]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  16: (rocksdb::MergingIterator::Next()+0x92) [0x557a2984e7d2]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  17: (rocksdb::CompactionIterator::Next()+0x1a2) [0x557a298c0642]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  18: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x569) [0x557a298c8899]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  19: (rocksdb::CompactionJob::Run()+0x314) [0x557a298ca264]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  20: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*)+0xc54) [0x557a2971aef4]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  21: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x397) [0x557a2971d807]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  22: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x97) [0x557a2971df87]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  23: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x266) [0x557a2990ab96]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  24: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x47) [0x557a2990ad17]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  25: (()+0xbd57f) [0x7f6815a4d57f]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  26: (()+0x76db) [0x7f68161416db]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  27: (clone()+0x3f) [0x7f681510a88f]
Feb 09 10:26:28 ceph1 ceph-osd[360132]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Feb 09 10:26:28 ceph1 systemd[1]: ceph-osd@3.service: Main process exited, code=killed, status=6/ABRT
Feb 09 10:26:28 ceph1 systemd[1]: ceph-osd@3.service: Failed with result 'signal'.