Project

General

Profile

Support #22917

mon keeps on crashing ( 12.2.2 )

Added by yair mackenzi about 6 years ago. Updated about 6 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Correctness/Safety
Target version:
% Done:

0%

Tags:
aa192d54-4d1f-4b60-8879-695c618a9598
Reviewed:
Affected Versions:
Component(RADOS):
Pull request ID:

Description

mon keeps on crashing ( 0> 2018-02-05 00:22:49.915541 7f6d0a781700 -1 ** Caught signal (Aborted) *
in thread 7f6d0a781700 thread_name:ms_dispatch )

i thought it was resolved on 12.2.2 ( BUG #22041 ) but it doesn't .
i'm able to restart it but it keeps crashing again after a while :

    -2> 2018-02-05 00:22:49.860326 7f6d0a781700  1 -- 10.63.4.55:6789/0 <== mon.0 10.63.4.53:6789/0 1279690678 ==== paxos(commit lc 11606204 fc 0 pn 12000 opn 0) v4 ==== 13258+0+0 (2462847767 0 0) 0x27b9b28300 con 0x27b298f800
    -1> 2018-02-05 00:22:49.861068 7f6d06779700  5 -- 10.63.4.55:6789/0 >> 10.63.4.53:6789/0 conn(0x27b298f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25778395 cs=1 l=0). rx mon.0 seq 1279690679 0x27baf52400 paxos(lease lc 11606204 fc 11605488 pn 0 opn 0) v4
     0> 2018-02-05 00:22:49.915541 7f6d0a781700 -1 *** Caught signal (Aborted) **
 in thread 7f6d0a781700 thread_name:ms_dispatch

 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
 1: (()+0x95c864) [0x27a878b864]
 2: (()+0x11390) [0x7f6d124d4390]
 3: (gsignal()+0x38) [0x7f6d10c22428]
 4: (abort()+0x16a) [0x7f6d10c2402a]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x16d) [0x7f6d1156584d]
 6: (()+0x8d6b6) [0x7f6d115636b6]
 7: (()+0x8d701) [0x7f6d11563701]
 8: (()+0x8d919) [0x7f6d11563919]
 9: (()+0x1228f) [0x7f6d11a8428f]
 10: (operator new[](unsigned long)+0x4e7) [0x7f6d11aa8417]
 11: (rocksdb::Arena::AllocateNewBlock(unsigned long)+0x70) [0x27a88a4a10]
 12: (rocksdb::Arena::AllocateFallback(unsigned long, bool)+0x45) [0x27a88a4b95]
 13: (rocksdb::Arena::AllocateAligned(unsigned long, unsigned long, rocksdb::Logger*)+0x100) [0x27a88a4d20]
 14: (rocksdb::ConcurrentArena::AllocateAligned(unsigned long, unsigned long, rocksdb::Logger*)+0x175) [0x27a88130a5]
 15: (()+0xa1e5f3) [0x27a884d5f3]
 16: (()+0xa1edcd) [0x27a884ddcd]
 17: (rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, rocksdb::Slice const&, rocksdb::Slice const&, bool, rocksdb::MemTablePostProcessInfo*)+0x109) [0x27a880e569]
 18: (rocksdb::MemTableInserter::PutCF(unsigned int, rocksdb::Slice const&, rocksdb::Slice const&)+0x39c) [0x27a884660c]
 19: (rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) const+0x5b7) [0x27a883fee7]
 20: (rocksdb::WriteBatchInternal::InsertInto(rocksdb::autovector<rocksdb::WriteThread::Writer*, 8ul> const&, unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*, bool, unsigned long, rocksdb::DB*, bool)+0x14b) [0x27a8843d0b]
 21: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool)+0x14a7) [0x27a8905367]
 22: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x2a) [0x27a8905dca]
 23: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0xaf) [0x27a82dbfaf]
 24: (MonitorDBStore::apply_transaction(std::shared_ptr<MonitorDBStore::Transaction>)+0xc1f) [0x27a819e00f]
 25: (Paxos::store_state(MMonPaxos*)+0x2f2) [0x27a8301422]
 26: (Paxos::handle_commit(boost::intrusive_ptr<MonOpRequest>)+0x31c) [0x27a830291c]
 27: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x30b) [0x27a830913b]
 28: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xebe) [0x27a81dcafe]

ceph-post-file: aa192d54-4d1f-4b60-8879-695c618a9598 ( crashing mon's log )
ceph-post-file: 280c3558-803e-4133-b67a-4739e0f1736c ( debug pack )

Thanks

History

#1 Updated by Kefu Chai about 6 years ago

  • Description updated (diff)
  • Category changed from Introspection/Control to Correctness/Safety

Also available in: Atom PDF