Bug #11680
Updated by Kefu Chai almost 9 years ago
<pre>
ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
1: /usr/bin/ceph-mon() [0x9017e2]
2: (()+0xf130) [0x7f30fab97130]
3: (gsignal()+0x37) [0x7f30f95b15d7]
4: (abort()+0x148) [0x7f30f95b2cc8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f30f9eb59b5]
6: (()+0x5e926) [0x7f30f9eb3926]
7: (()+0x5e953) [0x7f30f9eb3953]
8: (()+0x5eb73) [0x7f30f9eb3b73]
9: (std::__throw_logic_error(char const*)+0x77) [0x7f30f9f08717]
10: (char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag)+0xa1) [0x7f30f9f14561]
11: (std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)+0x38) [0x7f30f9f14918]
12: (CrushTreeDumper::dump_item_fields(CrushWrapper const*, CrushTreeDumper::Item const&, ceph::Formatter*)+0xb9) [0x6175c9]
13: (OSDMap::print_tree(std::ostream*, ceph::Formatter*) const+0x10d9) [0x787ab9]
14: (OSDMonitor::preprocess_command(MMonCommand*)+0xe55) [0x60a0b5]
15: (OSDMonitor::preprocess_query(PaxosServiceMessage*)+0x20b) [0x60f75b]
16: (PaxosService::dispatch(PaxosServiceMessage*)+0x833) [0x5cacd3]
17: (Monitor::handle_command(MMonCommand*)+0x147c) [0x591a4c]
18: (Monitor::dispatch(MonSession*, Message*, bool)+0xf9) [0x594c89]
19: (Monitor::_ms_dispatch(Message*)+0x1a6) [0x595936]
20: (Monitor::ms_dispatch(Message*)+0x23) [0x5b5403]
21: (DispatchQueue::entry()+0x64a) [0x8a1d9a]
22: (DispatchQueue::DispatchThread::entry()+0xd) [0x79bd9d]
23: (()+0x7df5) [0x7f30fab8fdf5]
24: (clone()+0x6d) [0x7f30f96721ad]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
</pre>
the most recent log messages
<pre>
-25> 2015-05-18 06:20:09.143639 7f09817a0700 10 log_client log_queue is 5 last_log 24 sent 23 num 5 unsent 1 sending 1
-24> 2015-05-18 06:20:09.143642 7f09817a0700 10 log_client will send 2015-05-18 06:20:09.143637 mon.0 10.8.128.33:6789/0 24 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
-23> 2015-05-18 06:20:09.143652 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600
-22> 2015-05-18 06:20:09.143795 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 -- ?+1108 0x466b000 con 0x460aaa0
-21> 2015-05-18 06:20:09.143817 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600
-20> 2015-05-18 06:20:09.143827 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.143828 lease_expire=0.000000 has v0 lc 78631
-19> 2015-05-18 06:20:09.145454 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 8 ==== mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 ==== 82+0+0 (3917390200 0 0) 0x466a000 con 0x460aaa0
-18> 2015-05-18 06:20:09.145494 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd dump", "format": "json"} v 0) v1
-17> 2015-05-18 06:20:09.145520 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch
-16> 2015-05-18 06:20:09.145522 7f09817a0700 10 log_client _send_to_monlog to self
-15> 2015-05-18 06:20:09.145523 7f09817a0700 10 log_client log_queue is 6 last_log 25 sent 24 num 6 unsent 1 sending 1
-14> 2015-05-18 06:20:09.145527 7f09817a0700 10 log_client will send 2015-05-18 06:20:09.145522 mon.0 10.8.128.33:6789/0 25 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch
-13> 2015-05-18 06:20:09.145535 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f6540 con 0x4609600
-12> 2015-05-18 06:20:09.145546 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.145547 lease_expire=0.000000 has v0 lc 78631
-11> 2015-05-18 06:20:09.145779 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 -- ?+3436 0x4669800 con 0x460aaa0
-10> 2015-05-18 06:20:09.145805 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f6540 con 0x4609600
-9> 2015-05-18 06:20:09.145814 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.145815 lease_expire=0.000000 has v0 lc 78631
-8> 2015-05-18 06:20:09.147249 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 9 ==== mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1 ==== 95+0+0 (942654298 0 0) 0x466b800 con 0x460aaa0
-7> 2015-05-18 06:20:09.147309 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1
-6> 2015-05-18 06:20:09.147337 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch
-5> 2015-05-18 06:20:09.147343 7f09817a0700 10 log_client _send_to_monlog to self
-4> 2015-05-18 06:20:09.147345 7f09817a0700 10 log_client log_queue is 7 last_log 26 sent 25 num 7 unsent 1 sending 1
-3> 2015-05-18 06:20:09.147347 7f09817a0700 10 log_client will send 2015-05-18 06:20:09.147343 mon.0 10.8.128.33:6789/0 26 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch
-2> 2015-05-18 06:20:09.147355 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f3cc0 con 0x4609600
-1> 2015-05-18 06:20:09.147365 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.147366 lease_expire=0.000000 has v0 lc 78631
0> 2015-05-18 06:20:09.149437 7f09817a0700 -1 *** Caught signal (Aborted) **
in thread 7f09817a0700
</pre>
originally reported at https://bugzilla.redhat.com/show_bug.cgi?id=1222509