Support #22917
Updated by Kefu Chai about 6 years ago
mon keeps on crashing ( 0> 2018-02-05 00:22:49.915541 7f6d0a781700 -1 *** Caught signal (Aborted) ** in thread 7f6d0a781700 thread_name:ms_dispatch ) i thought it was resolved on 12.2.2 ( BUG #22041 ) but it doesn't . i'm able to restart it but it keeps crashing again after a while : <pre> -2> 2018-02-05 00:22:49.860326 7f6d0a781700 1 -- 10.63.4.55:6789/0 <== mon.0 10.63.4.53:6789/0 1279690678 ==== paxos(commit lc 11606204 fc 0 pn 12000 opn 0) v4 ==== 13258+0+0 (2462847767 0 0) 0x27b9b28300 con 0x27b298f800 -1> 2018-02-05 00:22:49.861068 7f6d06779700 5 -- 10.63.4.55:6789/0 >> 10.63.4.53:6789/0 conn(0x27b298f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25778395 cs=1 l=0). rx mon.0 seq 1279690679 0x27baf52400 paxos(lease lc 11606204 fc 11605488 pn 0 opn 0) v4 0> 2018-02-05 00:22:49.915541 7f6d0a781700 -1 *** Caught signal (Aborted) ** in thread 7f6d0a781700 thread_name:ms_dispatch ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) 1: (()+0x95c864) [0x27a878b864] 2: (()+0x11390) [0x7f6d124d4390] 3: (gsignal()+0x38) [0x7f6d10c22428] 4: (abort()+0x16a) [0x7f6d10c2402a] 5: (__gnu_cxx::__verbose_terminate_handler()+0x16d) [0x7f6d1156584d] 6: (()+0x8d6b6) [0x7f6d115636b6] 7: (()+0x8d701) [0x7f6d11563701] 8: (()+0x8d919) [0x7f6d11563919] 9: (()+0x1228f) [0x7f6d11a8428f] 10: (operator new[](unsigned long)+0x4e7) [0x7f6d11aa8417] 11: (rocksdb::Arena::AllocateNewBlock(unsigned long)+0x70) [0x27a88a4a10] 12: (rocksdb::Arena::AllocateFallback(unsigned long, bool)+0x45) [0x27a88a4b95] 13: (rocksdb::Arena::AllocateAligned(unsigned long, unsigned long, rocksdb::Logger*)+0x100) [0x27a88a4d20] 14: (rocksdb::ConcurrentArena::AllocateAligned(unsigned long, unsigned long, rocksdb::Logger*)+0x175) [0x27a88130a5] 15: (()+0xa1e5f3) [0x27a884d5f3] 16: (()+0xa1edcd) [0x27a884ddcd] 17: (rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, rocksdb::Slice const&, rocksdb::Slice const&, bool, rocksdb::MemTablePostProcessInfo*)+0x109) [0x27a880e569] 18: (rocksdb::MemTableInserter::PutCF(unsigned int, rocksdb::Slice const&, rocksdb::Slice const&)+0x39c) [0x27a884660c] 19: (rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) const+0x5b7) [0x27a883fee7] 20: (rocksdb::WriteBatchInternal::InsertInto(rocksdb::autovector<rocksdb::WriteThread::Writer*, 8ul> const&, unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*, bool, unsigned long, rocksdb::DB*, bool)+0x14b) [0x27a8843d0b] 21: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool)+0x14a7) [0x27a8905367] 22: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x2a) [0x27a8905dca] 23: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0xaf) [0x27a82dbfaf] 24: (MonitorDBStore::apply_transaction(std::shared_ptr<MonitorDBStore::Transaction>)+0xc1f) [0x27a819e00f] 25: (Paxos::store_state(MMonPaxos*)+0x2f2) [0x27a8301422] 26: (Paxos::handle_commit(boost::intrusive_ptr<MonOpRequest>)+0x31c) [0x27a830291c] 27: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x30b) [0x27a830913b] 28: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xebe) [0x27a81dcafe] </pre> ceph-post-file: aa192d54-4d1f-4b60-8879-695c618a9598 ( crashing mon's log ) ceph-post-file: 280c3558-803e-4133-b67a-4739e0f1736c ( debug pack ) Thanks