Project

General

Profile

Support #22917

Updated by Kefu Chai over 3 years ago

mon keeps on crashing ( 0> 2018-02-05 00:22:49.915541 7f6d0a781700 -1 *** Caught signal (Aborted) **
in thread 7f6d0a781700 thread_name:ms_dispatch )

i thought it was resolved on 12.2.2 ( BUG #22041 ) but it doesn't .
i'm able to restart it but it keeps crashing again after a while :

<pre>

-2> 2018-02-05 00:22:49.860326 7f6d0a781700 1 -- 10.63.4.55:6789/0 <== mon.0 10.63.4.53:6789/0 1279690678 ==== paxos(commit lc 11606204 fc 0 pn 12000 opn 0) v4 ==== 13258+0+0 (2462847767 0 0) 0x27b9b28300 con 0x27b298f800
-1> 2018-02-05 00:22:49.861068 7f6d06779700 5 -- 10.63.4.55:6789/0 >> 10.63.4.53:6789/0 conn(0x27b298f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25778395 cs=1 l=0). rx mon.0 seq 1279690679 0x27baf52400 paxos(lease lc 11606204 fc 11605488 pn 0 opn 0) v4
0> 2018-02-05 00:22:49.915541 7f6d0a781700 -1 *** Caught signal (Aborted) **
in thread 7f6d0a781700 thread_name:ms_dispatch

ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
1: (()+0x95c864) [0x27a878b864]
2: (()+0x11390) [0x7f6d124d4390]
3: (gsignal()+0x38) [0x7f6d10c22428]
4: (abort()+0x16a) [0x7f6d10c2402a]
5: (__gnu_cxx::__verbose_terminate_handler()+0x16d) [0x7f6d1156584d]
6: (()+0x8d6b6) [0x7f6d115636b6]
7: (()+0x8d701) [0x7f6d11563701]
8: (()+0x8d919) [0x7f6d11563919]
9: (()+0x1228f) [0x7f6d11a8428f]
10: (operator new[](unsigned long)+0x4e7) [0x7f6d11aa8417]
11: (rocksdb::Arena::AllocateNewBlock(unsigned long)+0x70) [0x27a88a4a10]
12: (rocksdb::Arena::AllocateFallback(unsigned long, bool)+0x45) [0x27a88a4b95]
13: (rocksdb::Arena::AllocateAligned(unsigned long, unsigned long, rocksdb::Logger*)+0x100) [0x27a88a4d20]
14: (rocksdb::ConcurrentArena::AllocateAligned(unsigned long, unsigned long, rocksdb::Logger*)+0x175) [0x27a88130a5]
15: (()+0xa1e5f3) [0x27a884d5f3]
16: (()+0xa1edcd) [0x27a884ddcd]
17: (rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, rocksdb::Slice const&, rocksdb::Slice const&, bool, rocksdb::MemTablePostProcessInfo*)+0x109) [0x27a880e569]
18: (rocksdb::MemTableInserter::PutCF(unsigned int, rocksdb::Slice const&, rocksdb::Slice const&)+0x39c) [0x27a884660c]
19: (rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) const+0x5b7) [0x27a883fee7]
20: (rocksdb::WriteBatchInternal::InsertInto(rocksdb::autovector<rocksdb::WriteThread::Writer*, 8ul> const&, unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*, bool, unsigned long, rocksdb::DB*, bool)+0x14b) [0x27a8843d0b]
21: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool)+0x14a7) [0x27a8905367]
22: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x2a) [0x27a8905dca]
23: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0xaf) [0x27a82dbfaf]
24: (MonitorDBStore::apply_transaction(std::shared_ptr<MonitorDBStore::Transaction>)+0xc1f) [0x27a819e00f]
25: (Paxos::store_state(MMonPaxos*)+0x2f2) [0x27a8301422]
26: (Paxos::handle_commit(boost::intrusive_ptr<MonOpRequest>)+0x31c) [0x27a830291c]
27: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x30b) [0x27a830913b]
28: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xebe) [0x27a81dcafe]

</pre>



ceph-post-file: aa192d54-4d1f-4b60-8879-695c618a9598 ( crashing mon's log )
ceph-post-file: 280c3558-803e-4133-b67a-4739e0f1736c ( debug pack )

Thanks

Back