2015-05-18 06:20:01.644938 7f09890a87c0 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-mon, pid 3992 2015-05-18 06:20:01.759779 7f09890a87c0 0 starting mon.Mon rank 0 at 10.8.128.33:6789/0 mon_data /var/lib/ceph/mon/ceph-Mon fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36 2015-05-18 06:20:01.759988 7f09890a87c0 1 mon.Mon@-1(probing) e1 preinit fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36 2015-05-18 06:20:01.760303 7f09890a87c0 1 mon.Mon@-1(probing).paxosservice(pgmap 1..567) refresh upgraded, format 0 -> 1 2015-05-18 06:20:01.760314 7f09890a87c0 1 mon.Mon@-1(probing).pg v0 on_upgrade discarding in-core PGMap 2015-05-18 06:20:01.760927 7f09890a87c0 0 mon.Mon@-1(probing).mds e1 print_map epoch 1 flags 0 created 0.000000 modified 2015-05-06 15:39:42.874456 tableserver 0 root 0 session_timeout 0 session_autoclose 0 max_file_size 0 last_failure 0 last_failure_osd_epoch 0 compat compat={},rocompat={},incompat={} max_mds 0 in up {} failed stopped data_pools metadata_pool 0 inline_data disabled 2015-05-18 06:20:01.761127 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires 2015-05-18 06:20:01.761135 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires 2015-05-18 06:20:01.761138 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires 2015-05-18 06:20:01.761141 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires 2015-05-18 06:20:01.761445 7f09890a87c0 1 mon.Mon@-1(probing).paxosservice(auth 251..328) refresh upgraded, format 0 -> 1 2015-05-18 06:20:01.761932 7f09890a87c0 0 mon.Mon@-1(probing) e1 my rank is now 0 (was -1) 2015-05-18 06:20:01.761953 7f09890a87c0 1 mon.Mon@0(probing) e1 win_standalone_election 2015-05-18 06:20:01.801222 7f09890a87c0 0 log_channel(cluster) log [INF] : mon.Mon@0 won leader election with quorum 0 2015-05-18 06:20:01.801309 7f09890a87c0 0 log_channel(cluster) log [INF] : monmap e1: 1 mons at {Mon=10.8.128.33:6789/0} 2015-05-18 06:20:01.801372 7f09890a87c0 0 log_channel(cluster) log [INF] : pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail 2015-05-18 06:20:01.801422 7f09890a87c0 0 log_channel(cluster) log [INF] : mdsmap e1: 0/0/0 up 2015-05-18 06:20:01.801492 7f09890a87c0 0 log_channel(cluster) log [INF] : osdmap e84: 4 osds: 4 up, 4 in 2015-05-18 06:20:02.475336 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 2015-05-18 06:20:02.475350 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 2015-05-18 06:20:02.475405 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 2015-05-18 06:20:02.475418 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 2015-05-18 06:20:02.823300 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch 2015-05-18 06:20:02.823374 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished 2015-05-18 06:20:02.832689 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 2015-05-18 06:20:02.832704 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 2015-05-18 06:20:02.832760 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 2015-05-18 06:20:02.832773 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 2015-05-18 06:20:07.802963 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch 2015-05-18 06:20:07.803031 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished 2015-05-18 06:20:07.835601 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 2015-05-18 06:20:07.835627 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 2015-05-18 06:20:07.835680 7f09817a0700 1 mon.Mon@0(leader).osd e84 we have enough reports/reporters to mark osd.0 down 2015-05-18 06:20:07.835700 7f09817a0700 0 log_channel(cluster) log [INF] : osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211) 2015-05-18 06:20:07.835868 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 2015-05-18 06:20:07.835888 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 2015-05-18 06:20:07.835925 7f09817a0700 1 mon.Mon@0(leader).osd e84 we have enough reports/reporters to mark osd.3 down 2015-05-18 06:20:07.835939 7f09817a0700 0 log_channel(cluster) log [INF] : osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015) 2015-05-18 06:20:07.937432 7f098370b700 1 mon.Mon@0(leader).osd e85 e85: 4 osds: 2 up, 4 in 2015-05-18 06:20:07.987480 7f098370b700 0 log_channel(cluster) log [INF] : osdmap e85: 4 osds: 2 up, 4 in 2015-05-18 06:20:08.013617 7f098370b700 0 log_channel(cluster) log [INF] : pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail 2015-05-18 06:20:08.949801 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "mon_status", "format": "json"} v 0) v1 2015-05-18 06:20:08.949836 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "mon_status", "format": "json"}]: dispatch 2015-05-18 06:20:08.951357 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1 2015-05-18 06:20:08.951380 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2015-05-18 06:20:08.952829 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "health", "detail": "", "format": "json"} v 0) v1 2015-05-18 06:20:08.952852 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "health", "detail": "", "format": "json"}]: dispatch 2015-05-18 06:20:08.954227 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"} v 0) v1 2015-05-18 06:20:08.954251 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]: dispatch 2015-05-18 06:20:09.143603 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1 2015-05-18 06:20:09.143635 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2015-05-18 06:20:09.145494 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 2015-05-18 06:20:09.145520 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2015-05-18 06:20:09.147309 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1 2015-05-18 06:20:09.147337 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch 2015-05-18 06:20:09.149437 7f09817a0700 -1 *** Caught signal (Aborted) ** in thread 7f09817a0700 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-mon() [0x9017e2] 2: (()+0xf130) [0x7f098826e130] 3: (gsignal()+0x37) [0x7f0986c885d7] 4: (abort()+0x148) [0x7f0986c89cc8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f098758c9b5] 6: (()+0x5e926) [0x7f098758a926] 7: (()+0x5e953) [0x7f098758a953] 8: (()+0x5eb73) [0x7f098758ab73] 9: (std::__throw_logic_error(char const*)+0x77) [0x7f09875df717] 10: (char* std::string::_S_construct(char const*, char const*, std::allocator const&, std::forward_iterator_tag)+0xa1) [0x7f09875eb561] 11: (std::basic_string, std::allocator >::basic_string(char const*, std::allocator const&)+0x38) [0x7f09875eb918] 12: (CrushTreeDumper::dump_item_fields(CrushWrapper const*, CrushTreeDumper::Item const&, ceph::Formatter*)+0xb9) [0x6175c9] 13: (OSDMap::print_tree(std::ostream*, ceph::Formatter*) const+0x10d9) [0x787ab9] 14: (OSDMonitor::preprocess_command(MMonCommand*)+0xe55) [0x60a0b5] 15: (OSDMonitor::preprocess_query(PaxosServiceMessage*)+0x20b) [0x60f75b] 16: (PaxosService::dispatch(PaxosServiceMessage*)+0x833) [0x5cacd3] 17: (Monitor::handle_command(MMonCommand*)+0x147c) [0x591a4c] 18: (Monitor::dispatch(MonSession*, Message*, bool)+0xf9) [0x594c89] 19: (Monitor::_ms_dispatch(Message*)+0x1a6) [0x595936] 20: (Monitor::ms_dispatch(Message*)+0x23) [0x5b5403] 21: (DispatchQueue::entry()+0x64a) [0x8a1d9a] 22: (DispatchQueue::DispatchThread::entry()+0xd) [0x79bd9d] 23: (()+0x7df5) [0x7f0988266df5] 24: (clone()+0x6d) [0x7f0986d491ad] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -448> 2015-05-18 06:20:01.642177 7f09890a87c0 5 asok(0x4650000) register_command perfcounters_dump hook 0x45c0050 -447> 2015-05-18 06:20:01.642205 7f09890a87c0 5 asok(0x4650000) register_command 1 hook 0x45c0050 -446> 2015-05-18 06:20:01.642212 7f09890a87c0 5 asok(0x4650000) register_command perf dump hook 0x45c0050 -445> 2015-05-18 06:20:01.642219 7f09890a87c0 5 asok(0x4650000) register_command perfcounters_schema hook 0x45c0050 -444> 2015-05-18 06:20:01.642224 7f09890a87c0 5 asok(0x4650000) register_command 2 hook 0x45c0050 -443> 2015-05-18 06:20:01.642228 7f09890a87c0 5 asok(0x4650000) register_command perf schema hook 0x45c0050 -442> 2015-05-18 06:20:01.642233 7f09890a87c0 5 asok(0x4650000) register_command perf reset hook 0x45c0050 -441> 2015-05-18 06:20:01.642237 7f09890a87c0 5 asok(0x4650000) register_command config show hook 0x45c0050 -440> 2015-05-18 06:20:01.642242 7f09890a87c0 5 asok(0x4650000) register_command config set hook 0x45c0050 -439> 2015-05-18 06:20:01.642247 7f09890a87c0 5 asok(0x4650000) register_command config get hook 0x45c0050 -438> 2015-05-18 06:20:01.642251 7f09890a87c0 5 asok(0x4650000) register_command config diff hook 0x45c0050 -437> 2015-05-18 06:20:01.642255 7f09890a87c0 5 asok(0x4650000) register_command log flush hook 0x45c0050 -436> 2015-05-18 06:20:01.642260 7f09890a87c0 5 asok(0x4650000) register_command log dump hook 0x45c0050 -435> 2015-05-18 06:20:01.642264 7f09890a87c0 5 asok(0x4650000) register_command log reopen hook 0x45c0050 -434> 2015-05-18 06:20:01.644938 7f09890a87c0 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-mon, pid 3992 -433> 2015-05-18 06:20:01.667692 7f09890a87c0 5 asok(0x4650000) init /var/run/ceph/ceph-mon.Mon.asok -432> 2015-05-18 06:20:01.667706 7f09890a87c0 5 asok(0x4650000) bind_and_listen /var/run/ceph/ceph-mon.Mon.asok -431> 2015-05-18 06:20:01.667808 7f09890a87c0 5 asok(0x4650000) register_command 0 hook 0x45b80b8 -430> 2015-05-18 06:20:01.667817 7f09890a87c0 5 asok(0x4650000) register_command version hook 0x45b80b8 -429> 2015-05-18 06:20:01.667823 7f09890a87c0 5 asok(0x4650000) register_command git_version hook 0x45b80b8 -428> 2015-05-18 06:20:01.667828 7f09890a87c0 5 asok(0x4650000) register_command help hook 0x45c00b0 -427> 2015-05-18 06:20:01.667833 7f09890a87c0 5 asok(0x4650000) register_command get_command_descriptions hook 0x45c0150 -426> 2015-05-18 06:20:01.667864 7f0984a8b700 5 asok(0x4650000) entry start -425> 2015-05-18 06:20:01.742013 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 1496 bytes -424> 2015-05-18 06:20:01.759779 7f09890a87c0 0 starting mon.Mon rank 0 at 10.8.128.33:6789/0 mon_data /var/lib/ceph/mon/ceph-Mon fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36 -423> 2015-05-18 06:20:01.759829 7f09890a87c0 1 -- 10.8.128.33:6789/0 learned my addr 10.8.128.33:6789/0 -422> 2015-05-18 06:20:01.759835 7f09890a87c0 1 accepter.accepter.bind my_inst.addr is 10.8.128.33:6789/0 need_addr=0 -421> 2015-05-18 06:20:01.759884 7f09890a87c0 5 adding auth protocol: cephx -420> 2015-05-18 06:20:01.759890 7f09890a87c0 5 adding auth protocol: cephx -419> 2015-05-18 06:20:01.759909 7f09890a87c0 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info) -418> 2015-05-18 06:20:01.759913 7f09890a87c0 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info) -417> 2015-05-18 06:20:01.759988 7f09890a87c0 1 mon.Mon@-1(probing) e1 preinit fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36 -416> 2015-05-18 06:20:01.760303 7f09890a87c0 1 mon.Mon@-1(probing).paxosservice(pgmap 1..567) refresh upgraded, format 0 -> 1 -415> 2015-05-18 06:20:01.760314 7f09890a87c0 1 mon.Mon@-1(probing).pg v0 on_upgrade discarding in-core PGMap -414> 2015-05-18 06:20:01.760919 7f09890a87c0 4 mon.Mon@-1(probing).mds e1 new map -413> 2015-05-18 06:20:01.760927 7f09890a87c0 0 mon.Mon@-1(probing).mds e1 print_map epoch 1 flags 0 created 0.000000 modified 2015-05-06 15:39:42.874456 tableserver 0 root 0 session_timeout 0 session_autoclose 0 max_file_size 0 last_failure 0 last_failure_osd_epoch 0 compat compat={},rocompat={},incompat={} max_mds 0 in up {} failed stopped data_pools metadata_pool 0 inline_data disabled -412> 2015-05-18 06:20:01.761127 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires -411> 2015-05-18 06:20:01.761135 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires -410> 2015-05-18 06:20:01.761138 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires -409> 2015-05-18 06:20:01.761141 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires -408> 2015-05-18 06:20:01.761445 7f09890a87c0 1 mon.Mon@-1(probing).paxosservice(auth 251..328) refresh upgraded, format 0 -> 1 -407> 2015-05-18 06:20:01.761792 7f09890a87c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-Mon/keyring -406> 2015-05-18 06:20:01.761799 7f09890a87c0 5 asok(0x4650000) register_command mon_status hook 0x45c0330 -405> 2015-05-18 06:20:01.761804 7f09890a87c0 5 asok(0x4650000) register_command quorum_status hook 0x45c0330 -404> 2015-05-18 06:20:01.761808 7f09890a87c0 5 asok(0x4650000) register_command sync_force hook 0x45c0330 -403> 2015-05-18 06:20:01.761811 7f09890a87c0 5 asok(0x4650000) register_command add_bootstrap_peer_hint hook 0x45c0330 -402> 2015-05-18 06:20:01.761814 7f09890a87c0 5 asok(0x4650000) register_command quorum enter hook 0x45c0330 -401> 2015-05-18 06:20:01.761817 7f09890a87c0 5 asok(0x4650000) register_command quorum exit hook 0x45c0330 -400> 2015-05-18 06:20:01.761825 7f09890a87c0 1 -- 10.8.128.33:6789/0 messenger.start -399> 2015-05-18 06:20:01.761854 7f09890a87c0 2 mon.Mon@-1(probing) e1 init -398> 2015-05-18 06:20:01.761912 7f09890a87c0 1 accepter.accepter.start -397> 2015-05-18 06:20:01.761932 7f09890a87c0 0 mon.Mon@-1(probing) e1 my rank is now 0 (was -1) -396> 2015-05-18 06:20:01.761939 7f09890a87c0 1 -- 10.8.128.33:6789/0 mark_down_all -395> 2015-05-18 06:20:01.761953 7f09890a87c0 1 mon.Mon@0(probing) e1 win_standalone_election -394> 2015-05-18 06:20:01.801222 7f09890a87c0 0 log_channel(cluster) log [INF] : mon.Mon@0 won leader election with quorum 0 -393> 2015-05-18 06:20:01.801230 7f09890a87c0 10 log_client _send_to_monlog to self -392> 2015-05-18 06:20:01.801231 7f09890a87c0 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1 -391> 2015-05-18 06:20:01.801235 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801229 mon.0 10.8.128.33:6789/0 1 : cluster [INF] mon.Mon@0 won leader election with quorum 0 -390> 2015-05-18 06:20:01.801247 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2f40 con 0x4609600 -389> 2015-05-18 06:20:01.801309 7f09890a87c0 0 log_channel(cluster) log [INF] : monmap e1: 1 mons at {Mon=10.8.128.33:6789/0} -388> 2015-05-18 06:20:01.801316 7f09890a87c0 10 log_client _send_to_monlog to self -387> 2015-05-18 06:20:01.801317 7f09890a87c0 10 log_client log_queue is 2 last_log 2 sent 1 num 2 unsent 1 sending 1 -386> 2015-05-18 06:20:01.801298 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2f40 con 0x4609600 -385> 2015-05-18 06:20:01.801320 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801314 mon.0 10.8.128.33:6789/0 2 : cluster [INF] monmap e1: 1 mons at {Mon=10.8.128.33:6789/0} -384> 2015-05-18 06:20:01.801327 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2ac0 con 0x4609600 -383> 2015-05-18 06:20:01.801372 7f09890a87c0 0 log_channel(cluster) log [INF] : pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail -382> 2015-05-18 06:20:01.801376 7f09890a87c0 10 log_client _send_to_monlog to self -381> 2015-05-18 06:20:01.801377 7f09890a87c0 10 log_client log_queue is 3 last_log 3 sent 2 num 3 unsent 1 sending 1 -380> 2015-05-18 06:20:01.801380 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801376 mon.0 10.8.128.33:6789/0 3 : cluster [INF] pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail -379> 2015-05-18 06:20:01.801387 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2880 con 0x4609600 -378> 2015-05-18 06:20:01.801422 7f09890a87c0 0 log_channel(cluster) log [INF] : mdsmap e1: 0/0/0 up -377> 2015-05-18 06:20:01.801426 7f09890a87c0 10 log_client _send_to_monlog to self -376> 2015-05-18 06:20:01.801427 7f09890a87c0 10 log_client log_queue is 4 last_log 4 sent 3 num 4 unsent 1 sending 1 -375> 2015-05-18 06:20:01.801428 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801425 mon.0 10.8.128.33:6789/0 4 : cluster [INF] mdsmap e1: 0/0/0 up -374> 2015-05-18 06:20:01.801435 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2640 con 0x4609600 -373> 2015-05-18 06:20:01.801492 7f09890a87c0 0 log_channel(cluster) log [INF] : osdmap e84: 4 osds: 4 up, 4 in -372> 2015-05-18 06:20:01.801496 7f09890a87c0 10 log_client _send_to_monlog to self -371> 2015-05-18 06:20:01.801498 7f09890a87c0 10 log_client log_queue is 5 last_log 5 sent 4 num 5 unsent 1 sending 1 -370> 2015-05-18 06:20:01.801499 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801496 mon.0 10.8.128.33:6789/0 5 : cluster [INF] osdmap e84: 4 osds: 4 up, 4 in -369> 2015-05-18 06:20:01.801506 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600 -368> 2015-05-18 06:20:01.801773 7f09817a0700 5 mon.Mon@0(leader) e1 setting monitor caps on this connection -367> 2015-05-18 06:20:01.801790 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801791 lease_expire=0.000000 has v0 lc 78623 -366> 2015-05-18 06:20:01.801820 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2ac0 con 0x4609600 -365> 2015-05-18 06:20:01.801832 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801833 lease_expire=0.000000 has v0 lc 78623 -364> 2015-05-18 06:20:01.801846 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2880 con 0x4609600 -363> 2015-05-18 06:20:01.801853 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801854 lease_expire=0.000000 has v0 lc 78623 -362> 2015-05-18 06:20:01.801866 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2640 con 0x4609600 -361> 2015-05-18 06:20:01.801874 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801874 lease_expire=0.000000 has v0 lc 78623 -360> 2015-05-18 06:20:01.801886 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600 -359> 2015-05-18 06:20:01.801894 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801894 lease_expire=0.000000 has v0 lc 78623 -358> 2015-05-18 06:20:01.851959 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) queue_pending_finisher 0x45c04d0 -357> 2015-05-18 06:20:01.918852 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 1) v1 -- ?+0 0x47b81e0 con 0x4609600 -356> 2015-05-18 06:20:01.918871 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 2) v1 -- ?+0 0x47b85a0 con 0x4609600 -355> 2015-05-18 06:20:01.918879 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 3) v1 -- ?+0 0x47b83c0 con 0x4609600 -354> 2015-05-18 06:20:01.918886 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 4) v1 -- ?+0 0x47b8b40 con 0x4609600 -353> 2015-05-18 06:20:01.918894 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 5) v1 -- ?+0 0x47b8960 con 0x4609600 -352> 2015-05-18 06:20:01.918930 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 1) v1 ==== 0+0+0 (0 0 0) 0x47b81e0 con 0x4609600 -351> 2015-05-18 06:20:01.918953 7f09817a0700 10 log_client handle_log_ack log(last 1) v1 -350> 2015-05-18 06:20:01.918956 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801229 mon.0 10.8.128.33:6789/0 1 : cluster [INF] mon.Mon@0 won leader election with quorum 0 -349> 2015-05-18 06:20:01.918977 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 2) v1 ==== 0+0+0 (0 0 0) 0x47b85a0 con 0x4609600 -348> 2015-05-18 06:20:01.918986 7f09817a0700 10 log_client handle_log_ack log(last 2) v1 -347> 2015-05-18 06:20:01.918989 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801314 mon.0 10.8.128.33:6789/0 2 : cluster [INF] monmap e1: 1 mons at {Mon=10.8.128.33:6789/0} -346> 2015-05-18 06:20:01.918997 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 3) v1 ==== 0+0+0 (0 0 0) 0x47b83c0 con 0x4609600 -345> 2015-05-18 06:20:01.919005 7f09817a0700 10 log_client handle_log_ack log(last 3) v1 -344> 2015-05-18 06:20:01.919007 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801376 mon.0 10.8.128.33:6789/0 3 : cluster [INF] pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail -343> 2015-05-18 06:20:01.919015 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 4) v1 ==== 0+0+0 (0 0 0) 0x47b8b40 con 0x4609600 -342> 2015-05-18 06:20:01.919022 7f09817a0700 10 log_client handle_log_ack log(last 4) v1 -341> 2015-05-18 06:20:01.919025 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801425 mon.0 10.8.128.33:6789/0 4 : cluster [INF] mdsmap e1: 0/0/0 up -340> 2015-05-18 06:20:01.919034 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 5) v1 ==== 0+0+0 (0 0 0) 0x47b8960 con 0x4609600 -339> 2015-05-18 06:20:01.919041 7f09817a0700 10 log_client handle_log_ack log(last 5) v1 -338> 2015-05-18 06:20:01.919043 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801496 mon.0 10.8.128.33:6789/0 5 : cluster [INF] osdmap e84: 4 osds: 4 up, 4 in -337> 2015-05-18 06:20:02.473956 7f0989044700 1 -- 10.8.128.33:6789/0 >> :/0 pipe(0x48d0000 sd=20 :6789 s=0 pgs=0 cs=0 l=0 c=0x460a260).accept sd=20 10.8.128.86:48717/0 -336> 2015-05-18 06:20:02.474299 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 1 ==== auth(proto 0 26 bytes epoch 1) v1 ==== 56+0+0 (1708591557 0 0) 0x47f3a80 con 0x460a260 -335> 2015-05-18 06:20:02.474322 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.474323 lease_expire=0.000000 has v0 lc 78624 -334> 2015-05-18 06:20:02.474362 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f2f40 con 0x460a260 -333> 2015-05-18 06:20:02.474629 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (1575692119 0 0) 0x47f4140 con 0x460a260 -332> 2015-05-18 06:20:02.474645 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.474646 lease_expire=0.000000 has v0 lc 78624 -331> 2015-05-18 06:20:02.474885 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f3a80 con 0x460a260 -330> 2015-05-18 06:20:02.475200 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 3 ==== mon_subscribe({monmap=2+,osd_pg_creates=0}) v2 ==== 50+0+0 (3889722309 0 0) 0x466a600 con 0x460a260 -329> 2015-05-18 06:20:02.475267 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8960 con 0x460a260 -328> 2015-05-18 06:20:02.475300 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 4 ==== osd_failure(failed osd.0 10.8.128.40:6800/25974 for 765sec e84 v84) v3 ==== 188+0+0 (2601468610 0 0) 0x46798c0 con 0x460a260 -327> 2015-05-18 06:20:02.475316 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475317 lease_expire=0.000000 has v0 lc 78624 -326> 2015-05-18 06:20:02.475336 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 -325> 2015-05-18 06:20:02.475350 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 -324> 2015-05-18 06:20:02.475353 7f09817a0700 10 log_client _send_to_monlog to self -323> 2015-05-18 06:20:02.475354 7f09817a0700 10 log_client log_queue is 1 last_log 6 sent 5 num 1 unsent 1 sending 1 -322> 2015-05-18 06:20:02.475356 7f09817a0700 10 log_client will send 2015-05-18 06:20:02.475352 mon.0 10.8.128.33:6789/0 6 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 -321> 2015-05-18 06:20:02.475362 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f3cc0 con 0x4609600 -320> 2015-05-18 06:20:02.475385 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 5 ==== osd_failure(failed osd.3 10.8.128.86:6804/19849 for 723sec e84 v84) v3 ==== 188+0+0 (967876087 0 0) 0x4679e40 con 0x460a260 -319> 2015-05-18 06:20:02.475396 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475397 lease_expire=0.000000 has v0 lc 78624 -318> 2015-05-18 06:20:02.475405 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 -317> 2015-05-18 06:20:02.475418 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 -316> 2015-05-18 06:20:02.475420 7f09817a0700 10 log_client _send_to_monlog to self -315> 2015-05-18 06:20:02.475421 7f09817a0700 10 log_client log_queue is 2 last_log 7 sent 6 num 2 unsent 1 sending 1 -314> 2015-05-18 06:20:02.475423 7f09817a0700 10 log_client will send 2015-05-18 06:20:02.475419 mon.0 10.8.128.33:6789/0 7 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 -313> 2015-05-18 06:20:02.475428 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2ac0 con 0x4609600 -312> 2015-05-18 06:20:02.475444 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f3cc0 con 0x4609600 -311> 2015-05-18 06:20:02.475452 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475453 lease_expire=0.000000 has v0 lc 78624 -310> 2015-05-18 06:20:02.475470 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2ac0 con 0x4609600 -309> 2015-05-18 06:20:02.475479 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475480 lease_expire=0.000000 has v0 lc 78624 -308> 2015-05-18 06:20:02.475496 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 6 ==== mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=84}) v2 ==== 69+0+0 (2997052209 0 0) 0x466b000 con 0x460a260 -307> 2015-05-18 06:20:02.475517 7f09817a0700 5 mon.Mon@0(leader).osd e84 send_incremental [84..84] to osd.1 10.8.128.86:6800/19549 -306> 2015-05-18 06:20:02.475587 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- osd_map(84..84 src has 1..84) v3 -- ?+0 0x47f4140 con 0x460a260 -305> 2015-05-18 06:20:02.475605 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b83c0 con 0x460a260 -304> 2015-05-18 06:20:02.815632 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes -303> 2015-05-18 06:20:02.823300 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch -302> 2015-05-18 06:20:02.823308 7f0984a8b700 10 log_client _send_to_monlog to self -301> 2015-05-18 06:20:02.823309 7f0984a8b700 10 log_client log_queue is 3 last_log 8 sent 7 num 3 unsent 1 sending 1 -300> 2015-05-18 06:20:02.823312 7f0984a8b700 10 log_client will send 2015-05-18 06:20:02.823306 mon.0 10.8.128.33:6789/0 8 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch -299> 2015-05-18 06:20:02.823333 7f0984a8b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x4648d80 con 0x4609600 -298> 2015-05-18 06:20:02.823374 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished -297> 2015-05-18 06:20:02.823377 7f0984a8b700 10 log_client _send_to_monlog to self -296> 2015-05-18 06:20:02.823378 7f0984a8b700 10 log_client log_queue is 4 last_log 9 sent 8 num 4 unsent 1 sending 1 -295> 2015-05-18 06:20:02.823380 7f0984a8b700 10 log_client will send 2015-05-18 06:20:02.823376 mon.0 10.8.128.33:6789/0 9 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished -294> 2015-05-18 06:20:02.823386 7f0984a8b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4800 con 0x4609600 -293> 2015-05-18 06:20:02.823393 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4648d80 con 0x4609600 -292> 2015-05-18 06:20:02.823407 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'mon_status' '' to 0x45c0330 returned 520 bytes -291> 2015-05-18 06:20:02.823421 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.823422 lease_expire=0.000000 has v0 lc 78624 -290> 2015-05-18 06:20:02.823450 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4800 con 0x4609600 -289> 2015-05-18 06:20:02.823467 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.823468 lease_expire=0.000000 has v0 lc 78624 -288> 2015-05-18 06:20:02.832644 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 7 ==== osd_failure(failed osd.0 10.8.128.40:6800/25974 for 766sec e84 v84) v3 ==== 188+0+0 (4180528123 0 0) 0x4679b80 con 0x460a260 -287> 2015-05-18 06:20:02.832671 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832672 lease_expire=0.000000 has v0 lc 78624 -286> 2015-05-18 06:20:02.832689 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 -285> 2015-05-18 06:20:02.832704 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 -284> 2015-05-18 06:20:02.832707 7f09817a0700 10 log_client _send_to_monlog to self -283> 2015-05-18 06:20:02.832708 7f09817a0700 10 log_client log_queue is 5 last_log 10 sent 9 num 5 unsent 1 sending 1 -282> 2015-05-18 06:20:02.832710 7f09817a0700 10 log_client will send 2015-05-18 06:20:02.832706 mon.0 10.8.128.33:6789/0 10 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 -281> 2015-05-18 06:20:02.832717 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2f40 con 0x4609600 -280> 2015-05-18 06:20:02.832739 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 8 ==== osd_failure(failed osd.3 10.8.128.86:6804/19849 for 723sec e84 v84) v3 ==== 188+0+0 (967876087 0 0) 0x467a680 con 0x460a260 -279> 2015-05-18 06:20:02.832751 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832751 lease_expire=0.000000 has v0 lc 78624 -278> 2015-05-18 06:20:02.832760 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 -277> 2015-05-18 06:20:02.832773 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 -276> 2015-05-18 06:20:02.832775 7f09817a0700 10 log_client _send_to_monlog to self -275> 2015-05-18 06:20:02.832777 7f09817a0700 10 log_client log_queue is 6 last_log 11 sent 10 num 6 unsent 1 sending 1 -274> 2015-05-18 06:20:02.832779 7f09817a0700 10 log_client will send 2015-05-18 06:20:02.832774 mon.0 10.8.128.33:6789/0 11 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 -273> 2015-05-18 06:20:02.832785 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5100 con 0x4609600 -272> 2015-05-18 06:20:02.832807 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2f40 con 0x4609600 -271> 2015-05-18 06:20:02.832817 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832817 lease_expire=0.000000 has v0 lc 78624 -270> 2015-05-18 06:20:02.832833 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5100 con 0x4609600 -269> 2015-05-18 06:20:02.832842 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832844 lease_expire=0.000000 has v0 lc 78624 -268> 2015-05-18 06:20:02.918393 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) queue_pending_finisher 0x45c04c0 -267> 2015-05-18 06:20:02.977214 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 6) v1 -- ?+0 0x47b8780 con 0x4609600 -266> 2015-05-18 06:20:02.977246 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 7) v1 -- ?+0 0x47b81e0 con 0x4609600 -265> 2015-05-18 06:20:02.977254 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 8) v1 -- ?+0 0x47b8b40 con 0x4609600 -264> 2015-05-18 06:20:02.977259 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 9) v1 -- ?+0 0x47b83c0 con 0x4609600 -263> 2015-05-18 06:20:02.977267 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 10) v1 -- ?+0 0x47b8960 con 0x4609600 -262> 2015-05-18 06:20:02.977291 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 11) v1 -- ?+0 0x47b92c0 con 0x4609600 -261> 2015-05-18 06:20:02.977297 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 6) v1 ==== 0+0+0 (0 0 0) 0x47b8780 con 0x4609600 -260> 2015-05-18 06:20:02.977319 7f09817a0700 10 log_client handle_log_ack log(last 6) v1 -259> 2015-05-18 06:20:02.977322 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.475352 mon.0 10.8.128.33:6789/0 6 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 -258> 2015-05-18 06:20:02.977334 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 7) v1 ==== 0+0+0 (0 0 0) 0x47b81e0 con 0x4609600 -257> 2015-05-18 06:20:02.977341 7f09817a0700 10 log_client handle_log_ack log(last 7) v1 -256> 2015-05-18 06:20:02.977343 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.475419 mon.0 10.8.128.33:6789/0 7 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 -255> 2015-05-18 06:20:02.977352 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 8) v1 ==== 0+0+0 (0 0 0) 0x47b8b40 con 0x4609600 -254> 2015-05-18 06:20:02.977375 7f09817a0700 10 log_client handle_log_ack log(last 8) v1 -253> 2015-05-18 06:20:02.977379 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.823306 mon.0 10.8.128.33:6789/0 8 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch -252> 2015-05-18 06:20:02.977388 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 9) v1 ==== 0+0+0 (0 0 0) 0x47b83c0 con 0x4609600 -251> 2015-05-18 06:20:02.977395 7f09817a0700 10 log_client handle_log_ack log(last 9) v1 -250> 2015-05-18 06:20:02.977398 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.823376 mon.0 10.8.128.33:6789/0 9 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished -249> 2015-05-18 06:20:02.977405 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 10) v1 ==== 0+0+0 (0 0 0) 0x47b8960 con 0x4609600 -248> 2015-05-18 06:20:02.977412 7f09817a0700 10 log_client handle_log_ack log(last 10) v1 -247> 2015-05-18 06:20:02.977414 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.832706 mon.0 10.8.128.33:6789/0 10 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 -246> 2015-05-18 06:20:02.977422 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 11) v1 ==== 0+0+0 (0 0 0) 0x47b92c0 con 0x4609600 -245> 2015-05-18 06:20:02.977430 7f09817a0700 10 log_client handle_log_ack log(last 11) v1 -244> 2015-05-18 06:20:02.977431 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.832774 mon.0 10.8.128.33:6789/0 11 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 -243> 2015-05-18 06:20:06.761957 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) is_readable = 1 - now=2015-05-18 06:20:06.761960 lease_expire=0.000000 has v0 lc 78625 -242> 2015-05-18 06:20:06.761973 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) is_readable = 1 - now=2015-05-18 06:20:06.761974 lease_expire=0.000000 has v0 lc 78625 -241> 2015-05-18 06:20:06.761993 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) is_readable = 1 - now=2015-05-18 06:20:06.761993 lease_expire=0.000000 has v0 lc 78625 -240> 2015-05-18 06:20:06.762078 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) queue_pending_finisher 0x45c04f0 -239> 2015-05-18 06:20:07.793435 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes -238> 2015-05-18 06:20:07.802963 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch -237> 2015-05-18 06:20:07.802972 7f0984a8b700 10 log_client _send_to_monlog to self -236> 2015-05-18 06:20:07.802973 7f0984a8b700 10 log_client log_queue is 1 last_log 12 sent 11 num 1 unsent 1 sending 1 -235> 2015-05-18 06:20:07.802977 7f0984a8b700 10 log_client will send 2015-05-18 06:20:07.802970 mon.0 10.8.128.33:6789/0 12 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch -234> 2015-05-18 06:20:07.802995 7f0984a8b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x4648900 con 0x4609600 -233> 2015-05-18 06:20:07.803031 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished -232> 2015-05-18 06:20:07.803033 7f0984a8b700 10 log_client _send_to_monlog to self -231> 2015-05-18 06:20:07.803034 7f0984a8b700 10 log_client log_queue is 2 last_log 13 sent 12 num 2 unsent 1 sending 1 -230> 2015-05-18 06:20:07.803037 7f0984a8b700 10 log_client will send 2015-05-18 06:20:07.803032 mon.0 10.8.128.33:6789/0 13 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished -229> 2015-05-18 06:20:07.803043 7f0984a8b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600 -228> 2015-05-18 06:20:07.803059 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'mon_status' '' to 0x45c0330 returned 307 bytes -227> 2015-05-18 06:20:07.803053 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4648900 con 0x4609600 -226> 2015-05-18 06:20:07.803083 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.803084 lease_expire=0.000000 has v0 lc 78626 -225> 2015-05-18 06:20:07.803114 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600 -224> 2015-05-18 06:20:07.803123 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.803125 lease_expire=0.000000 has v0 lc 78626 -223> 2015-05-18 06:20:07.803590 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes -222> 2015-05-18 06:20:07.810261 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78626) queue_pending_finisher 0x45c0320 -221> 2015-05-18 06:20:07.812704 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'version' '' to 0x45b80b8 returned 20 bytes -220> 2015-05-18 06:20:07.833354 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 9 ==== osd_failure(failed osd.0 10.8.128.40:6800/25974 for 771sec e84 v84) v3 ==== 188+0+0 (3406336582 0 0) 0x467a3c0 con 0x460a260 -219> 2015-05-18 06:20:07.835570 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835574 lease_expire=0.000000 has v0 lc 78626 -218> 2015-05-18 06:20:07.835601 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 -217> 2015-05-18 06:20:07.835627 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 -216> 2015-05-18 06:20:07.835632 7f09817a0700 10 log_client _send_to_monlog to self -215> 2015-05-18 06:20:07.835633 7f09817a0700 10 log_client log_queue is 3 last_log 14 sent 13 num 3 unsent 1 sending 1 -214> 2015-05-18 06:20:07.835639 7f09817a0700 10 log_client will send 2015-05-18 06:20:07.835630 mon.0 10.8.128.33:6789/0 14 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 -213> 2015-05-18 06:20:07.835652 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4ec0 con 0x4609600 -212> 2015-05-18 06:20:07.835680 7f09817a0700 1 mon.Mon@0(leader).osd e84 we have enough reports/reporters to mark osd.0 down -211> 2015-05-18 06:20:07.835700 7f09817a0700 0 log_channel(cluster) log [INF] : osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211) -210> 2015-05-18 06:20:07.835703 7f09817a0700 10 log_client _send_to_monlog to self -209> 2015-05-18 06:20:07.835704 7f09817a0700 10 log_client log_queue is 4 last_log 15 sent 14 num 4 unsent 1 sending 1 -208> 2015-05-18 06:20:07.835707 7f09817a0700 10 log_client will send 2015-05-18 06:20:07.835702 mon.0 10.8.128.33:6789/0 15 : cluster [INF] osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211) -207> 2015-05-18 06:20:07.835722 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4c80 con 0x4609600 -206> 2015-05-18 06:20:07.835744 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4ec0 con 0x4609600 -205> 2015-05-18 06:20:07.835759 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835761 lease_expire=0.000000 has v0 lc 78626 -204> 2015-05-18 06:20:07.835780 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4c80 con 0x4609600 -203> 2015-05-18 06:20:07.835809 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835810 lease_expire=0.000000 has v0 lc 78626 -202> 2015-05-18 06:20:07.835826 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 10 ==== osd_failure(failed osd.3 10.8.128.86:6804/19849 for 728sec e84 v84) v3 ==== 188+0+0 (2916513576 0 0) 0x467a100 con 0x460a260 -201> 2015-05-18 06:20:07.835853 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835854 lease_expire=0.000000 has v0 lc 78626 -200> 2015-05-18 06:20:07.835868 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 -199> 2015-05-18 06:20:07.835888 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 -198> 2015-05-18 06:20:07.835892 7f09817a0700 10 log_client _send_to_monlog to self -197> 2015-05-18 06:20:07.835893 7f09817a0700 10 log_client log_queue is 5 last_log 16 sent 15 num 5 unsent 1 sending 1 -196> 2015-05-18 06:20:07.835897 7f09817a0700 10 log_client will send 2015-05-18 06:20:07.835891 mon.0 10.8.128.33:6789/0 16 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 -195> 2015-05-18 06:20:07.835907 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4a40 con 0x4609600 -194> 2015-05-18 06:20:07.835925 7f09817a0700 1 mon.Mon@0(leader).osd e84 we have enough reports/reporters to mark osd.3 down -193> 2015-05-18 06:20:07.835939 7f09817a0700 0 log_channel(cluster) log [INF] : osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015) -192> 2015-05-18 06:20:07.835942 7f09817a0700 10 log_client _send_to_monlog to self -191> 2015-05-18 06:20:07.835943 7f09817a0700 10 log_client log_queue is 6 last_log 17 sent 16 num 6 unsent 1 sending 1 -190> 2015-05-18 06:20:07.835946 7f09817a0700 10 log_client will send 2015-05-18 06:20:07.835941 mon.0 10.8.128.33:6789/0 17 : cluster [INF] osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015) -189> 2015-05-18 06:20:07.835956 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5c40 con 0x4609600 -188> 2015-05-18 06:20:07.835984 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4a40 con 0x4609600 -187> 2015-05-18 06:20:07.835997 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835998 lease_expire=0.000000 has v0 lc 78626 -186> 2015-05-18 06:20:07.836011 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5c40 con 0x4609600 -185> 2015-05-18 06:20:07.836021 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.836022 lease_expire=0.000000 has v0 lc 78626 -184> 2015-05-18 06:20:07.860966 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 12) v1 -- ?+0 0x47b90e0 con 0x4609600 -183> 2015-05-18 06:20:07.860985 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 13) v1 -- ?+0 0x47b8f00 con 0x4609600 -182> 2015-05-18 06:20:07.860993 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.860994 lease_expire=0.000000 has v0 lc 78627 -181> 2015-05-18 06:20:07.861010 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.861011 lease_expire=0.000000 has v0 lc 78627 -180> 2015-05-18 06:20:07.861023 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.861024 lease_expire=0.000000 has v0 lc 78627 -179> 2015-05-18 06:20:07.861034 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.861035 lease_expire=0.000000 has v0 lc 78627 -178> 2015-05-18 06:20:07.861029 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 12) v1 ==== 0+0+0 (0 0 0) 0x47b90e0 con 0x4609600 -177> 2015-05-18 06:20:07.861052 7f09817a0700 10 log_client handle_log_ack log(last 12) v1 -176> 2015-05-18 06:20:07.861057 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.802970 mon.0 10.8.128.33:6789/0 12 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch -175> 2015-05-18 06:20:07.861074 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 13) v1 ==== 0+0+0 (0 0 0) 0x47b8f00 con 0x4609600 -174> 2015-05-18 06:20:07.861086 7f09817a0700 10 log_client handle_log_ack log(last 13) v1 -173> 2015-05-18 06:20:07.861090 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.803032 mon.0 10.8.128.33:6789/0 13 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished -172> 2015-05-18 06:20:07.885804 7f0981fa1700 2 mon.Mon@0(leader).osd e84 osd.0 DOWN -171> 2015-05-18 06:20:07.885814 7f0981fa1700 2 mon.Mon@0(leader).osd e84 osd.3 DOWN -170> 2015-05-18 06:20:07.885893 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) queue_pending_finisher 0x45c04d0 -169> 2015-05-18 06:20:07.937432 7f098370b700 1 mon.Mon@0(leader).osd e85 e85: 4 osds: 2 up, 4 in -168> 2015-05-18 06:20:07.962214 7f098370b700 5 mon.Mon@0(leader).paxos(paxos refresh c 78062..78628) is_readable = 0 - now=2015-05-18 06:20:07.962218 lease_expire=0.000000 has v0 lc 78628 -167> 2015-05-18 06:20:07.962309 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- osd_map(84..85 src has 1..85) v3 -- ?+0 0x47f2400 con 0x460a260 -166> 2015-05-18 06:20:07.962334 7f098370b700 5 mon.Mon@0(leader).osd e85 send_latest to osd.1 10.8.128.86:6800/19549 start 84 -165> 2015-05-18 06:20:07.962341 7f098370b700 5 mon.Mon@0(leader).osd e85 send_incremental [84..85] to osd.1 10.8.128.86:6800/19549 -164> 2015-05-18 06:20:07.962379 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f2f40 con 0x460a260 -163> 2015-05-18 06:20:07.962394 7f098370b700 5 mon.Mon@0(leader).osd e85 send_latest to osd.1 10.8.128.86:6800/19549 start 84 -162> 2015-05-18 06:20:07.962399 7f098370b700 5 mon.Mon@0(leader).osd e85 send_incremental [84..85] to osd.1 10.8.128.86:6800/19549 -161> 2015-05-18 06:20:07.962590 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78628) is_readable = 1 - now=2015-05-18 06:20:07.962592 lease_expire=0.000000 has v0 lc 78628 -160> 2015-05-18 06:20:07.962922 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78628) queue_pending_finisher 0x45c04d0 -159> 2015-05-18 06:20:07.987480 7f098370b700 0 log_channel(cluster) log [INF] : osdmap e85: 4 osds: 2 up, 4 in -158> 2015-05-18 06:20:07.987489 7f098370b700 10 log_client _send_to_monlog to self -157> 2015-05-18 06:20:07.987492 7f098370b700 10 log_client log_queue is 5 last_log 18 sent 17 num 5 unsent 1 sending 1 -156> 2015-05-18 06:20:07.987495 7f098370b700 10 log_client will send 2015-05-18 06:20:07.987487 mon.0 10.8.128.33:6789/0 18 : cluster [INF] osdmap e85: 4 osds: 2 up, 4 in -155> 2015-05-18 06:20:07.987506 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4800 con 0x4609600 -154> 2015-05-18 06:20:07.987564 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4800 con 0x4609600 -153> 2015-05-18 06:20:07.987582 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78628) is_readable = 1 - now=2015-05-18 06:20:07.987583 lease_expire=0.000000 has v0 lc 78628 -152> 2015-05-18 06:20:08.005319 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 11 ==== mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=86}) v2 ==== 69+0+0 (1597159760 0 0) 0x466b400 con 0x460a260 -151> 2015-05-18 06:20:08.005352 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8f00 con 0x460a260 -150> 2015-05-18 06:20:08.013617 7f098370b700 0 log_channel(cluster) log [INF] : pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail -149> 2015-05-18 06:20:08.013624 7f098370b700 10 log_client _send_to_monlog to self -148> 2015-05-18 06:20:08.013625 7f098370b700 10 log_client log_queue is 6 last_log 19 sent 18 num 6 unsent 1 sending 1 -147> 2015-05-18 06:20:08.013627 7f098370b700 10 log_client will send 2015-05-18 06:20:08.013623 mon.0 10.8.128.33:6789/0 19 : cluster [INF] pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail -146> 2015-05-18 06:20:08.013637 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5100 con 0x4609600 -145> 2015-05-18 06:20:08.013689 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5100 con 0x4609600 -144> 2015-05-18 06:20:08.013702 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78629) is_readable = 1 - now=2015-05-18 06:20:08.013703 lease_expire=0.000000 has v0 lc 78629 -143> 2015-05-18 06:20:08.860774 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78629) queue_pending_finisher 0x45c0510 -142> 2015-05-18 06:20:08.909048 7f097f69b700 1 -- 10.8.128.33:6789/0 >> :/0 pipe(0x48e2000 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x460a3c0).accept sd=21 10.8.128.33:60116/0 -141> 2015-05-18 06:20:08.909248 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x47f4140 con 0x460a3c0 -140> 2015-05-18 06:20:08.909277 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78629) is_readable = 1 - now=2015-05-18 06:20:08.909278 lease_expire=0.000000 has v0 lc 78629 -139> 2015-05-18 06:20:08.909363 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78629) queue_pending_finisher 0x45c0500 -138> 2015-05-18 06:20:08.913369 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.913371 lease_expire=0.000000 has v0 lc 78630 -137> 2015-05-18 06:20:08.913393 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_map magic: 0 v1 -- ?+0 0x47f2ac0 con 0x460a3c0 -136> 2015-05-18 06:20:08.913421 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4648900 con 0x460a3c0 -135> 2015-05-18 06:20:08.913486 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 14) v1 -- ?+0 0x47b85a0 con 0x4609600 -134> 2015-05-18 06:20:08.913503 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 15) v1 -- ?+0 0x47b8960 con 0x4609600 -133> 2015-05-18 06:20:08.913512 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 16) v1 -- ?+0 0x47b83c0 con 0x4609600 -132> 2015-05-18 06:20:08.913519 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 17) v1 -- ?+0 0x47b8b40 con 0x4609600 -131> 2015-05-18 06:20:08.913528 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 18) v1 -- ?+0 0x47b81e0 con 0x4609600 -130> 2015-05-18 06:20:08.913536 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 19) v1 -- ?+0 0x47b8780 con 0x4609600 -129> 2015-05-18 06:20:08.913547 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 14) v1 ==== 0+0+0 (0 0 0) 0x47b85a0 con 0x4609600 -128> 2015-05-18 06:20:08.938024 7f09817a0700 10 log_client handle_log_ack log(last 14) v1 -127> 2015-05-18 06:20:08.938034 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.835630 mon.0 10.8.128.33:6789/0 14 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 -126> 2015-05-18 06:20:08.938051 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 15) v1 ==== 0+0+0 (0 0 0) 0x47b8960 con 0x4609600 -125> 2015-05-18 06:20:08.938060 7f09817a0700 10 log_client handle_log_ack log(last 15) v1 -124> 2015-05-18 06:20:08.938063 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.835702 mon.0 10.8.128.33:6789/0 15 : cluster [INF] osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211) -123> 2015-05-18 06:20:08.938071 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 16) v1 ==== 0+0+0 (0 0 0) 0x47b83c0 con 0x4609600 -122> 2015-05-18 06:20:08.938079 7f09817a0700 10 log_client handle_log_ack log(last 16) v1 -121> 2015-05-18 06:20:08.938083 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.835891 mon.0 10.8.128.33:6789/0 16 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 -120> 2015-05-18 06:20:08.938092 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 17) v1 ==== 0+0+0 (0 0 0) 0x47b8b40 con 0x4609600 -119> 2015-05-18 06:20:08.938099 7f09817a0700 10 log_client handle_log_ack log(last 17) v1 -118> 2015-05-18 06:20:08.938101 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.835941 mon.0 10.8.128.33:6789/0 17 : cluster [INF] osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015) -117> 2015-05-18 06:20:08.938109 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 18) v1 ==== 0+0+0 (0 0 0) 0x47b81e0 con 0x4609600 -116> 2015-05-18 06:20:08.938116 7f09817a0700 10 log_client handle_log_ack log(last 18) v1 -115> 2015-05-18 06:20:08.938119 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.987487 mon.0 10.8.128.33:6789/0 18 : cluster [INF] osdmap e85: 4 osds: 2 up, 4 in -114> 2015-05-18 06:20:08.938127 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 19) v1 ==== 0+0+0 (0 0 0) 0x47b8780 con 0x4609600 -113> 2015-05-18 06:20:08.938135 7f09817a0700 10 log_client handle_log_ack log(last 19) v1 -112> 2015-05-18 06:20:08.938137 7f09817a0700 10 log_client logged 2015-05-18 06:20:08.013623 mon.0 10.8.128.33:6789/0 19 : cluster [INF] pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail -111> 2015-05-18 06:20:08.938145 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (309592415 0 0) 0x47f2ac0 con 0x460a3c0 -110> 2015-05-18 06:20:08.938154 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.938155 lease_expire=0.000000 has v0 lc 78630 -109> 2015-05-18 06:20:08.938432 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f4380 con 0x460a3c0 -108> 2015-05-18 06:20:08.938705 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (3478599268 0 0) 0x4648900 con 0x460a3c0 -107> 2015-05-18 06:20:08.938717 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.938718 lease_expire=0.000000 has v0 lc 78630 -106> 2015-05-18 06:20:08.938852 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f2ac0 con 0x460a3c0 -105> 2015-05-18 06:20:08.939109 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x466b000 con 0x460a3c0 -104> 2015-05-18 06:20:08.939138 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_map magic: 0 v1 -- ?+0 0x4648900 con 0x460a3c0 -103> 2015-05-18 06:20:08.939153 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8780 con 0x460a3c0 -102> 2015-05-18 06:20:08.939168 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 5 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x466a600 con 0x460a3c0 -101> 2015-05-18 06:20:08.939244 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f5a00 con 0x460a3c0 -100> 2015-05-18 06:20:08.939266 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b81e0 con 0x460a3c0 -99> 2015-05-18 06:20:08.939293 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 6 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x466b400 con 0x460a3c0 -98> 2015-05-18 06:20:08.939342 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f57c0 con 0x460a3c0 -97> 2015-05-18 06:20:08.939354 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8960 con 0x460a3c0 -96> 2015-05-18 06:20:08.949743 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 7 ==== mon_command({"prefix": "mon_status", "format": "json"} v 0) v1 ==== 84+0+0 (110991942 0 0) 0x466b400 con 0x460a3c0 -95> 2015-05-18 06:20:08.949801 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "mon_status", "format": "json"} v 0) v1 -94> 2015-05-18 06:20:08.949836 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "mon_status", "format": "json"}]: dispatch -93> 2015-05-18 06:20:08.949839 7f09817a0700 10 log_client _send_to_monlog to self -92> 2015-05-18 06:20:08.949840 7f09817a0700 10 log_client log_queue is 1 last_log 20 sent 19 num 1 unsent 1 sending 1 -91> 2015-05-18 06:20:08.949843 7f09817a0700 10 log_client will send 2015-05-18 06:20:08.949838 mon.0 10.8.128.33:6789/0 20 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "mon_status", "format": "json"}]: dispatch -90> 2015-05-18 06:20:08.949851 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5340 con 0x4609600 -89> 2015-05-18 06:20:08.949891 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"prefix": "mon_status", "format": "json"}]=0 v0) v1 -- ?+307 0x4668e00 con 0x460a3c0 -88> 2015-05-18 06:20:08.949910 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5340 con 0x4609600 -87> 2015-05-18 06:20:08.949919 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.949920 lease_expire=0.000000 has v0 lc 78630 -86> 2015-05-18 06:20:08.951317 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 8 ==== mon_command({"prefix": "status", "format": "json"} v 0) v1 ==== 80+0+0 (1009655278 0 0) 0x466a600 con 0x460a3c0 -85> 2015-05-18 06:20:08.951357 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1 -84> 2015-05-18 06:20:08.951380 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch -83> 2015-05-18 06:20:08.951382 7f09817a0700 10 log_client _send_to_monlog to self -82> 2015-05-18 06:20:08.951384 7f09817a0700 10 log_client log_queue is 2 last_log 21 sent 20 num 2 unsent 1 sending 1 -81> 2015-05-18 06:20:08.951386 7f09817a0700 10 log_client will send 2015-05-18 06:20:08.951382 mon.0 10.8.128.33:6789/0 21 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch -80> 2015-05-18 06:20:08.951394 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5580 con 0x4609600 -79> 2015-05-18 06:20:08.951519 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 -- ?+1108 0x466b400 con 0x460a3c0 -78> 2015-05-18 06:20:08.951538 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5580 con 0x4609600 -77> 2015-05-18 06:20:08.951548 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.951548 lease_expire=0.000000 has v0 lc 78630 -76> 2015-05-18 06:20:08.952793 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 9 ==== mon_command({"prefix": "health", "detail": "", "format": "json"} v 0) v1 ==== 94+0+0 (1843246703 0 0) 0x466b000 con 0x460a3c0 -75> 2015-05-18 06:20:08.952829 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "health", "detail": "", "format": "json"} v 0) v1 -74> 2015-05-18 06:20:08.952852 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "health", "detail": "", "format": "json"}]: dispatch -73> 2015-05-18 06:20:08.952855 7f09817a0700 10 log_client _send_to_monlog to self -72> 2015-05-18 06:20:08.952856 7f09817a0700 10 log_client log_queue is 3 last_log 22 sent 21 num 3 unsent 1 sending 1 -71> 2015-05-18 06:20:08.952858 7f09817a0700 10 log_client will send 2015-05-18 06:20:08.952854 mon.0 10.8.128.33:6789/0 22 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "health", "detail": "", "format": "json"}]: dispatch -70> 2015-05-18 06:20:08.952864 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2f40 con 0x4609600 -69> 2015-05-18 06:20:08.952941 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"prefix": "health", "detail": "", "format": "json"}]=0 v0) v1 -- ?+401 0x466a600 con 0x460a3c0 -68> 2015-05-18 06:20:08.952960 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2f40 con 0x4609600 -67> 2015-05-18 06:20:08.952970 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.952970 lease_expire=0.000000 has v0 lc 78630 -66> 2015-05-18 06:20:08.954184 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 10 ==== mon_command({"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"} v 0) v1 ==== 112+0+0 (3334637970 0 0) 0x4669800 con 0x460a3c0 -65> 2015-05-18 06:20:08.954227 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"} v 0) v1 -64> 2015-05-18 06:20:08.954251 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]: dispatch -63> 2015-05-18 06:20:08.954255 7f09817a0700 10 log_client _send_to_monlog to self -62> 2015-05-18 06:20:08.954256 7f09817a0700 10 log_client log_queue is 4 last_log 23 sent 22 num 4 unsent 1 sending 1 -61> 2015-05-18 06:20:08.954258 7f09817a0700 10 log_client will send 2015-05-18 06:20:08.954254 mon.0 10.8.128.33:6789/0 23 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]: dispatch -60> 2015-05-18 06:20:08.954266 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f60c0 con 0x4609600 -59> 2015-05-18 06:20:08.954285 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.954286 lease_expire=0.000000 has v0 lc 78630 -58> 2015-05-18 06:20:08.954605 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]=0 dumped pgs_brief in format json v568) v1 -- ?+6616 0x466b000 con 0x460a3c0 -57> 2015-05-18 06:20:08.954626 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f60c0 con 0x4609600 -56> 2015-05-18 06:20:08.954636 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.954636 lease_expire=0.000000 has v0 lc 78630 -55> 2015-05-18 06:20:08.957218 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes -54> 2015-05-18 06:20:08.965521 7f0984a8b700 1 do_command 'config show' 'format:json -53> 2015-05-18 06:20:08.970043 7f0984a8b700 1 do_command 'config show' 'format:json result is 26602 bytes -52> 2015-05-18 06:20:08.970050 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'config show' '' to 0x45c0050 returned 26602 bytes -51> 2015-05-18 06:20:09.029915 7f097f69b700 2 -- 10.8.128.33:6789/0 >> 10.8.128.33:0/1004034 pipe(0x48e2000 sd=21 :6789 s=2 pgs=1 cs=1 l=1 c=0x460a3c0).reader couldn't read tag, (0) Success -50> 2015-05-18 06:20:09.029958 7f097f69b700 2 -- 10.8.128.33:6789/0 >> 10.8.128.33:0/1004034 pipe(0x48e2000 sd=21 :6789 s=2 pgs=1 cs=1 l=1 c=0x460a3c0).fault (0) Success -49> 2015-05-18 06:20:09.134600 7f097f69b700 1 -- 10.8.128.33:6789/0 >> :/0 pipe(0x48e2000 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x460aaa0).accept sd=21 10.8.128.33:60119/0 -48> 2015-05-18 06:20:09.134790 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x47f5a00 con 0x460aaa0 -47> 2015-05-18 06:20:09.134816 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.134817 lease_expire=0.000000 has v0 lc 78631 -46> 2015-05-18 06:20:09.134841 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_map magic: 0 v1 -- ?+0 0x47f7740 con 0x460aaa0 -45> 2015-05-18 06:20:09.134870 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f7980 con 0x460aaa0 -44> 2015-05-18 06:20:09.135204 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (2513201549 0 0) 0x47f7980 con 0x460aaa0 -43> 2015-05-18 06:20:09.135222 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.135223 lease_expire=0.000000 has v0 lc 78631 -42> 2015-05-18 06:20:09.135370 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f5a00 con 0x460aaa0 -41> 2015-05-18 06:20:09.135590 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (692588905 0 0) 0x47f7740 con 0x460aaa0 -40> 2015-05-18 06:20:09.135605 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.135606 lease_expire=0.000000 has v0 lc 78631 -39> 2015-05-18 06:20:09.135738 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f7980 con 0x460aaa0 -38> 2015-05-18 06:20:09.135936 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x466a600 con 0x460aaa0 -37> 2015-05-18 06:20:09.135967 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_map magic: 0 v1 -- ?+0 0x47f7740 con 0x460aaa0 -36> 2015-05-18 06:20:09.135984 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b85a0 con 0x460aaa0 -35> 2015-05-18 06:20:09.136000 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 5 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x4668e00 con 0x460aaa0 -34> 2015-05-18 06:20:09.136101 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f2ac0 con 0x460aaa0 -33> 2015-05-18 06:20:09.136120 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b90e0 con 0x460aaa0 -32> 2015-05-18 06:20:09.136139 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 6 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x466b000 con 0x460aaa0 -31> 2015-05-18 06:20:09.136183 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f5e80 con 0x460aaa0 -30> 2015-05-18 06:20:09.136195 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b92c0 con 0x460aaa0 -29> 2015-05-18 06:20:09.143538 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 7 ==== mon_command({"prefix": "status", "format": "json"} v 0) v1 ==== 80+0+0 (1009655278 0 0) 0x4669800 con 0x460aaa0 -28> 2015-05-18 06:20:09.143603 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1 -27> 2015-05-18 06:20:09.143635 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch -26> 2015-05-18 06:20:09.143638 7f09817a0700 10 log_client _send_to_monlog to self -25> 2015-05-18 06:20:09.143639 7f09817a0700 10 log_client log_queue is 5 last_log 24 sent 23 num 5 unsent 1 sending 1 -24> 2015-05-18 06:20:09.143642 7f09817a0700 10 log_client will send 2015-05-18 06:20:09.143637 mon.0 10.8.128.33:6789/0 24 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch -23> 2015-05-18 06:20:09.143652 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600 -22> 2015-05-18 06:20:09.143795 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 -- ?+1108 0x466b000 con 0x460aaa0 -21> 2015-05-18 06:20:09.143817 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600 -20> 2015-05-18 06:20:09.143827 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.143828 lease_expire=0.000000 has v0 lc 78631 -19> 2015-05-18 06:20:09.145454 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 8 ==== mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 ==== 82+0+0 (3917390200 0 0) 0x466a000 con 0x460aaa0 -18> 2015-05-18 06:20:09.145494 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -17> 2015-05-18 06:20:09.145520 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch -16> 2015-05-18 06:20:09.145522 7f09817a0700 10 log_client _send_to_monlog to self -15> 2015-05-18 06:20:09.145523 7f09817a0700 10 log_client log_queue is 6 last_log 25 sent 24 num 6 unsent 1 sending 1 -14> 2015-05-18 06:20:09.145527 7f09817a0700 10 log_client will send 2015-05-18 06:20:09.145522 mon.0 10.8.128.33:6789/0 25 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch -13> 2015-05-18 06:20:09.145535 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f6540 con 0x4609600 -12> 2015-05-18 06:20:09.145546 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.145547 lease_expire=0.000000 has v0 lc 78631 -11> 2015-05-18 06:20:09.145779 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 -- ?+3436 0x4669800 con 0x460aaa0 -10> 2015-05-18 06:20:09.145805 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f6540 con 0x4609600 -9> 2015-05-18 06:20:09.145814 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.145815 lease_expire=0.000000 has v0 lc 78631 -8> 2015-05-18 06:20:09.147249 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 9 ==== mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1 ==== 95+0+0 (942654298 0 0) 0x466b800 con 0x460aaa0 -7> 2015-05-18 06:20:09.147309 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1 -6> 2015-05-18 06:20:09.147337 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch -5> 2015-05-18 06:20:09.147343 7f09817a0700 10 log_client _send_to_monlog to self -4> 2015-05-18 06:20:09.147345 7f09817a0700 10 log_client log_queue is 7 last_log 26 sent 25 num 7 unsent 1 sending 1 -3> 2015-05-18 06:20:09.147347 7f09817a0700 10 log_client will send 2015-05-18 06:20:09.147343 mon.0 10.8.128.33:6789/0 26 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch -2> 2015-05-18 06:20:09.147355 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f3cc0 con 0x4609600 -1> 2015-05-18 06:20:09.147365 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.147366 lease_expire=0.000000 has v0 lc 78631 0> 2015-05-18 06:20:09.149437 7f09817a0700 -1 *** Caught signal (Aborted) ** in thread 7f09817a0700 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-mon() [0x9017e2] 2: (()+0xf130) [0x7f098826e130] 3: (gsignal()+0x37) [0x7f0986c885d7] 4: (abort()+0x148) [0x7f0986c89cc8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f098758c9b5] 6: (()+0x5e926) [0x7f098758a926] 7: (()+0x5e953) [0x7f098758a953] 8: (()+0x5eb73) [0x7f098758ab73] 9: (std::__throw_logic_error(char const*)+0x77) [0x7f09875df717] 10: (char* std::string::_S_construct(char const*, char const*, std::allocator const&, std::forward_iterator_tag)+0xa1) [0x7f09875eb561] 11: (std::basic_string, std::allocator >::basic_string(char const*, std::allocator const&)+0x38) [0x7f09875eb918] 12: (CrushTreeDumper::dump_item_fields(CrushWrapper const*, CrushTreeDumper::Item const&, ceph::Formatter*)+0xb9) [0x6175c9] 13: (OSDMap::print_tree(std::ostream*, ceph::Formatter*) const+0x10d9) [0x787ab9] 14: (OSDMonitor::preprocess_command(MMonCommand*)+0xe55) [0x60a0b5] 15: (OSDMonitor::preprocess_query(PaxosServiceMessage*)+0x20b) [0x60f75b] 16: (PaxosService::dispatch(PaxosServiceMessage*)+0x833) [0x5cacd3] 17: (Monitor::handle_command(MMonCommand*)+0x147c) [0x591a4c] 18: (Monitor::dispatch(MonSession*, Message*, bool)+0xf9) [0x594c89] 19: (Monitor::_ms_dispatch(Message*)+0x1a6) [0x595936] 20: (Monitor::ms_dispatch(Message*)+0x23) [0x5b5403] 21: (DispatchQueue::entry()+0x64a) [0x8a1d9a] 22: (DispatchQueue::DispatchThread::entry()+0xd) [0x79bd9d] 23: (()+0x7df5) [0x7f0988266df5] 24: (clone()+0x6d) [0x7f0986d491ad] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mon.Mon.log --- end dump of recent events --- ===================================