Project

General

Profile

Bug #11680 » file_1222509.log.txt

/var/log/ceph/ceph-mon.Mon.log - Kefu Chai, 05/19/2015 05:13 AM

 
2015-05-18 06:20:01.644938 7f09890a87c0 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-mon, pid 3992
2015-05-18 06:20:01.759779 7f09890a87c0 0 starting mon.Mon rank 0 at 10.8.128.33:6789/0 mon_data /var/lib/ceph/mon/ceph-Mon fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36
2015-05-18 06:20:01.759988 7f09890a87c0 1 mon.Mon@-1(probing) e1 preinit fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36
2015-05-18 06:20:01.760303 7f09890a87c0 1 mon.Mon@-1(probing).paxosservice(pgmap 1..567) refresh upgraded, format 0 -> 1
2015-05-18 06:20:01.760314 7f09890a87c0 1 mon.Mon@-1(probing).pg v0 on_upgrade discarding in-core PGMap
2015-05-18 06:20:01.760927 7f09890a87c0 0 mon.Mon@-1(probing).mds e1 print_map
epoch 1
flags 0
created 0.000000
modified 2015-05-06 15:39:42.874456
tableserver 0
root 0
session_timeout 0
session_autoclose 0
max_file_size 0
last_failure 0
last_failure_osd_epoch 0
compat compat={},rocompat={},incompat={}
max_mds 0
in
up {}
failed
stopped
data_pools
metadata_pool 0
inline_data disabled

2015-05-18 06:20:01.761127 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
2015-05-18 06:20:01.761135 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
2015-05-18 06:20:01.761138 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
2015-05-18 06:20:01.761141 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
2015-05-18 06:20:01.761445 7f09890a87c0 1 mon.Mon@-1(probing).paxosservice(auth 251..328) refresh upgraded, format 0 -> 1
2015-05-18 06:20:01.761932 7f09890a87c0 0 mon.Mon@-1(probing) e1 my rank is now 0 (was -1)
2015-05-18 06:20:01.761953 7f09890a87c0 1 mon.Mon@0(probing) e1 win_standalone_election
2015-05-18 06:20:01.801222 7f09890a87c0 0 log_channel(cluster) log [INF] : mon.Mon@0 won leader election with quorum 0
2015-05-18 06:20:01.801309 7f09890a87c0 0 log_channel(cluster) log [INF] : monmap e1: 1 mons at {Mon=10.8.128.33:6789/0}
2015-05-18 06:20:01.801372 7f09890a87c0 0 log_channel(cluster) log [INF] : pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
2015-05-18 06:20:01.801422 7f09890a87c0 0 log_channel(cluster) log [INF] : mdsmap e1: 0/0/0 up
2015-05-18 06:20:01.801492 7f09890a87c0 0 log_channel(cluster) log [INF] : osdmap e84: 4 osds: 4 up, 4 in
2015-05-18 06:20:02.475336 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
2015-05-18 06:20:02.475350 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
2015-05-18 06:20:02.475405 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
2015-05-18 06:20:02.475418 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
2015-05-18 06:20:02.823300 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2015-05-18 06:20:02.823374 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2015-05-18 06:20:02.832689 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
2015-05-18 06:20:02.832704 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
2015-05-18 06:20:02.832760 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
2015-05-18 06:20:02.832773 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
2015-05-18 06:20:07.802963 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch
2015-05-18 06:20:07.803031 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished
2015-05-18 06:20:07.835601 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
2015-05-18 06:20:07.835627 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
2015-05-18 06:20:07.835680 7f09817a0700 1 mon.Mon@0(leader).osd e84 we have enough reports/reporters to mark osd.0 down
2015-05-18 06:20:07.835700 7f09817a0700 0 log_channel(cluster) log [INF] : osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211)
2015-05-18 06:20:07.835868 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
2015-05-18 06:20:07.835888 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
2015-05-18 06:20:07.835925 7f09817a0700 1 mon.Mon@0(leader).osd e84 we have enough reports/reporters to mark osd.3 down
2015-05-18 06:20:07.835939 7f09817a0700 0 log_channel(cluster) log [INF] : osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015)
2015-05-18 06:20:07.937432 7f098370b700 1 mon.Mon@0(leader).osd e85 e85: 4 osds: 2 up, 4 in
2015-05-18 06:20:07.987480 7f098370b700 0 log_channel(cluster) log [INF] : osdmap e85: 4 osds: 2 up, 4 in
2015-05-18 06:20:08.013617 7f098370b700 0 log_channel(cluster) log [INF] : pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
2015-05-18 06:20:08.949801 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "mon_status", "format": "json"} v 0) v1
2015-05-18 06:20:08.949836 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "mon_status", "format": "json"}]: dispatch
2015-05-18 06:20:08.951357 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1
2015-05-18 06:20:08.951380 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
2015-05-18 06:20:08.952829 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "health", "detail": "", "format": "json"} v 0) v1
2015-05-18 06:20:08.952852 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "health", "detail": "", "format": "json"}]: dispatch
2015-05-18 06:20:08.954227 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"} v 0) v1
2015-05-18 06:20:08.954251 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]: dispatch
2015-05-18 06:20:09.143603 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1
2015-05-18 06:20:09.143635 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
2015-05-18 06:20:09.145494 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd dump", "format": "json"} v 0) v1
2015-05-18 06:20:09.145520 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch
2015-05-18 06:20:09.147309 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1
2015-05-18 06:20:09.147337 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch
2015-05-18 06:20:09.149437 7f09817a0700 -1 *** Caught signal (Aborted) **
in thread 7f09817a0700

ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
1: /usr/bin/ceph-mon() [0x9017e2]
2: (()+0xf130) [0x7f098826e130]
3: (gsignal()+0x37) [0x7f0986c885d7]
4: (abort()+0x148) [0x7f0986c89cc8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f098758c9b5]
6: (()+0x5e926) [0x7f098758a926]
7: (()+0x5e953) [0x7f098758a953]
8: (()+0x5eb73) [0x7f098758ab73]
9: (std::__throw_logic_error(char const*)+0x77) [0x7f09875df717]
10: (char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag)+0xa1) [0x7f09875eb561]
11: (std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)+0x38) [0x7f09875eb918]
12: (CrushTreeDumper::dump_item_fields(CrushWrapper const*, CrushTreeDumper::Item const&, ceph::Formatter*)+0xb9) [0x6175c9]
13: (OSDMap::print_tree(std::ostream*, ceph::Formatter*) const+0x10d9) [0x787ab9]
14: (OSDMonitor::preprocess_command(MMonCommand*)+0xe55) [0x60a0b5]
15: (OSDMonitor::preprocess_query(PaxosServiceMessage*)+0x20b) [0x60f75b]
16: (PaxosService::dispatch(PaxosServiceMessage*)+0x833) [0x5cacd3]
17: (Monitor::handle_command(MMonCommand*)+0x147c) [0x591a4c]
18: (Monitor::dispatch(MonSession*, Message*, bool)+0xf9) [0x594c89]
19: (Monitor::_ms_dispatch(Message*)+0x1a6) [0x595936]
20: (Monitor::ms_dispatch(Message*)+0x23) [0x5b5403]
21: (DispatchQueue::entry()+0x64a) [0x8a1d9a]
22: (DispatchQueue::DispatchThread::entry()+0xd) [0x79bd9d]
23: (()+0x7df5) [0x7f0988266df5]
24: (clone()+0x6d) [0x7f0986d491ad]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-448> 2015-05-18 06:20:01.642177 7f09890a87c0 5 asok(0x4650000) register_command perfcounters_dump hook 0x45c0050
-447> 2015-05-18 06:20:01.642205 7f09890a87c0 5 asok(0x4650000) register_command 1 hook 0x45c0050
-446> 2015-05-18 06:20:01.642212 7f09890a87c0 5 asok(0x4650000) register_command perf dump hook 0x45c0050
-445> 2015-05-18 06:20:01.642219 7f09890a87c0 5 asok(0x4650000) register_command perfcounters_schema hook 0x45c0050
-444> 2015-05-18 06:20:01.642224 7f09890a87c0 5 asok(0x4650000) register_command 2 hook 0x45c0050
-443> 2015-05-18 06:20:01.642228 7f09890a87c0 5 asok(0x4650000) register_command perf schema hook 0x45c0050
-442> 2015-05-18 06:20:01.642233 7f09890a87c0 5 asok(0x4650000) register_command perf reset hook 0x45c0050
-441> 2015-05-18 06:20:01.642237 7f09890a87c0 5 asok(0x4650000) register_command config show hook 0x45c0050
-440> 2015-05-18 06:20:01.642242 7f09890a87c0 5 asok(0x4650000) register_command config set hook 0x45c0050
-439> 2015-05-18 06:20:01.642247 7f09890a87c0 5 asok(0x4650000) register_command config get hook 0x45c0050
-438> 2015-05-18 06:20:01.642251 7f09890a87c0 5 asok(0x4650000) register_command config diff hook 0x45c0050
-437> 2015-05-18 06:20:01.642255 7f09890a87c0 5 asok(0x4650000) register_command log flush hook 0x45c0050
-436> 2015-05-18 06:20:01.642260 7f09890a87c0 5 asok(0x4650000) register_command log dump hook 0x45c0050
-435> 2015-05-18 06:20:01.642264 7f09890a87c0 5 asok(0x4650000) register_command log reopen hook 0x45c0050
-434> 2015-05-18 06:20:01.644938 7f09890a87c0 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-mon, pid 3992
-433> 2015-05-18 06:20:01.667692 7f09890a87c0 5 asok(0x4650000) init /var/run/ceph/ceph-mon.Mon.asok
-432> 2015-05-18 06:20:01.667706 7f09890a87c0 5 asok(0x4650000) bind_and_listen /var/run/ceph/ceph-mon.Mon.asok
-431> 2015-05-18 06:20:01.667808 7f09890a87c0 5 asok(0x4650000) register_command 0 hook 0x45b80b8
-430> 2015-05-18 06:20:01.667817 7f09890a87c0 5 asok(0x4650000) register_command version hook 0x45b80b8
-429> 2015-05-18 06:20:01.667823 7f09890a87c0 5 asok(0x4650000) register_command git_version hook 0x45b80b8
-428> 2015-05-18 06:20:01.667828 7f09890a87c0 5 asok(0x4650000) register_command help hook 0x45c00b0
-427> 2015-05-18 06:20:01.667833 7f09890a87c0 5 asok(0x4650000) register_command get_command_descriptions hook 0x45c0150
-426> 2015-05-18 06:20:01.667864 7f0984a8b700 5 asok(0x4650000) entry start
-425> 2015-05-18 06:20:01.742013 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 1496 bytes
-424> 2015-05-18 06:20:01.759779 7f09890a87c0 0 starting mon.Mon rank 0 at 10.8.128.33:6789/0 mon_data /var/lib/ceph/mon/ceph-Mon fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36
-423> 2015-05-18 06:20:01.759829 7f09890a87c0 1 -- 10.8.128.33:6789/0 learned my addr 10.8.128.33:6789/0
-422> 2015-05-18 06:20:01.759835 7f09890a87c0 1 accepter.accepter.bind my_inst.addr is 10.8.128.33:6789/0 need_addr=0
-421> 2015-05-18 06:20:01.759884 7f09890a87c0 5 adding auth protocol: cephx
-420> 2015-05-18 06:20:01.759890 7f09890a87c0 5 adding auth protocol: cephx
-419> 2015-05-18 06:20:01.759909 7f09890a87c0 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
-418> 2015-05-18 06:20:01.759913 7f09890a87c0 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info)
-417> 2015-05-18 06:20:01.759988 7f09890a87c0 1 mon.Mon@-1(probing) e1 preinit fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36
-416> 2015-05-18 06:20:01.760303 7f09890a87c0 1 mon.Mon@-1(probing).paxosservice(pgmap 1..567) refresh upgraded, format 0 -> 1
-415> 2015-05-18 06:20:01.760314 7f09890a87c0 1 mon.Mon@-1(probing).pg v0 on_upgrade discarding in-core PGMap
-414> 2015-05-18 06:20:01.760919 7f09890a87c0 4 mon.Mon@-1(probing).mds e1 new map
-413> 2015-05-18 06:20:01.760927 7f09890a87c0 0 mon.Mon@-1(probing).mds e1 print_map
epoch 1
flags 0
created 0.000000
modified 2015-05-06 15:39:42.874456
tableserver 0
root 0
session_timeout 0
session_autoclose 0
max_file_size 0
last_failure 0
last_failure_osd_epoch 0
compat compat={},rocompat={},incompat={}
max_mds 0
in
up {}
failed
stopped
data_pools
metadata_pool 0
inline_data disabled

-412> 2015-05-18 06:20:01.761127 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
-411> 2015-05-18 06:20:01.761135 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
-410> 2015-05-18 06:20:01.761138 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
-409> 2015-05-18 06:20:01.761141 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
-408> 2015-05-18 06:20:01.761445 7f09890a87c0 1 mon.Mon@-1(probing).paxosservice(auth 251..328) refresh upgraded, format 0 -> 1
-407> 2015-05-18 06:20:01.761792 7f09890a87c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-Mon/keyring
-406> 2015-05-18 06:20:01.761799 7f09890a87c0 5 asok(0x4650000) register_command mon_status hook 0x45c0330
-405> 2015-05-18 06:20:01.761804 7f09890a87c0 5 asok(0x4650000) register_command quorum_status hook 0x45c0330
-404> 2015-05-18 06:20:01.761808 7f09890a87c0 5 asok(0x4650000) register_command sync_force hook 0x45c0330
-403> 2015-05-18 06:20:01.761811 7f09890a87c0 5 asok(0x4650000) register_command add_bootstrap_peer_hint hook 0x45c0330
-402> 2015-05-18 06:20:01.761814 7f09890a87c0 5 asok(0x4650000) register_command quorum enter hook 0x45c0330
-401> 2015-05-18 06:20:01.761817 7f09890a87c0 5 asok(0x4650000) register_command quorum exit hook 0x45c0330
-400> 2015-05-18 06:20:01.761825 7f09890a87c0 1 -- 10.8.128.33:6789/0 messenger.start
-399> 2015-05-18 06:20:01.761854 7f09890a87c0 2 mon.Mon@-1(probing) e1 init
-398> 2015-05-18 06:20:01.761912 7f09890a87c0 1 accepter.accepter.start
-397> 2015-05-18 06:20:01.761932 7f09890a87c0 0 mon.Mon@-1(probing) e1 my rank is now 0 (was -1)
-396> 2015-05-18 06:20:01.761939 7f09890a87c0 1 -- 10.8.128.33:6789/0 mark_down_all
-395> 2015-05-18 06:20:01.761953 7f09890a87c0 1 mon.Mon@0(probing) e1 win_standalone_election
-394> 2015-05-18 06:20:01.801222 7f09890a87c0 0 log_channel(cluster) log [INF] : mon.Mon@0 won leader election with quorum 0
-393> 2015-05-18 06:20:01.801230 7f09890a87c0 10 log_client _send_to_monlog to self
-392> 2015-05-18 06:20:01.801231 7f09890a87c0 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1
-391> 2015-05-18 06:20:01.801235 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801229 mon.0 10.8.128.33:6789/0 1 : cluster [INF] mon.Mon@0 won leader election with quorum 0
-390> 2015-05-18 06:20:01.801247 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2f40 con 0x4609600
-389> 2015-05-18 06:20:01.801309 7f09890a87c0 0 log_channel(cluster) log [INF] : monmap e1: 1 mons at {Mon=10.8.128.33:6789/0}
-388> 2015-05-18 06:20:01.801316 7f09890a87c0 10 log_client _send_to_monlog to self
-387> 2015-05-18 06:20:01.801317 7f09890a87c0 10 log_client log_queue is 2 last_log 2 sent 1 num 2 unsent 1 sending 1
-386> 2015-05-18 06:20:01.801298 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2f40 con 0x4609600
-385> 2015-05-18 06:20:01.801320 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801314 mon.0 10.8.128.33:6789/0 2 : cluster [INF] monmap e1: 1 mons at {Mon=10.8.128.33:6789/0}
-384> 2015-05-18 06:20:01.801327 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2ac0 con 0x4609600
-383> 2015-05-18 06:20:01.801372 7f09890a87c0 0 log_channel(cluster) log [INF] : pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
-382> 2015-05-18 06:20:01.801376 7f09890a87c0 10 log_client _send_to_monlog to self
-381> 2015-05-18 06:20:01.801377 7f09890a87c0 10 log_client log_queue is 3 last_log 3 sent 2 num 3 unsent 1 sending 1
-380> 2015-05-18 06:20:01.801380 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801376 mon.0 10.8.128.33:6789/0 3 : cluster [INF] pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
-379> 2015-05-18 06:20:01.801387 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2880 con 0x4609600
-378> 2015-05-18 06:20:01.801422 7f09890a87c0 0 log_channel(cluster) log [INF] : mdsmap e1: 0/0/0 up
-377> 2015-05-18 06:20:01.801426 7f09890a87c0 10 log_client _send_to_monlog to self
-376> 2015-05-18 06:20:01.801427 7f09890a87c0 10 log_client log_queue is 4 last_log 4 sent 3 num 4 unsent 1 sending 1
-375> 2015-05-18 06:20:01.801428 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801425 mon.0 10.8.128.33:6789/0 4 : cluster [INF] mdsmap e1: 0/0/0 up
-374> 2015-05-18 06:20:01.801435 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2640 con 0x4609600
-373> 2015-05-18 06:20:01.801492 7f09890a87c0 0 log_channel(cluster) log [INF] : osdmap e84: 4 osds: 4 up, 4 in
-372> 2015-05-18 06:20:01.801496 7f09890a87c0 10 log_client _send_to_monlog to self
-371> 2015-05-18 06:20:01.801498 7f09890a87c0 10 log_client log_queue is 5 last_log 5 sent 4 num 5 unsent 1 sending 1
-370> 2015-05-18 06:20:01.801499 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801496 mon.0 10.8.128.33:6789/0 5 : cluster [INF] osdmap e84: 4 osds: 4 up, 4 in
-369> 2015-05-18 06:20:01.801506 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600
-368> 2015-05-18 06:20:01.801773 7f09817a0700 5 mon.Mon@0(leader) e1 setting monitor caps on this connection
-367> 2015-05-18 06:20:01.801790 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801791 lease_expire=0.000000 has v0 lc 78623
-366> 2015-05-18 06:20:01.801820 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2ac0 con 0x4609600
-365> 2015-05-18 06:20:01.801832 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801833 lease_expire=0.000000 has v0 lc 78623
-364> 2015-05-18 06:20:01.801846 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2880 con 0x4609600
-363> 2015-05-18 06:20:01.801853 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801854 lease_expire=0.000000 has v0 lc 78623
-362> 2015-05-18 06:20:01.801866 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2640 con 0x4609600
-361> 2015-05-18 06:20:01.801874 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801874 lease_expire=0.000000 has v0 lc 78623
-360> 2015-05-18 06:20:01.801886 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600
-359> 2015-05-18 06:20:01.801894 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801894 lease_expire=0.000000 has v0 lc 78623
-358> 2015-05-18 06:20:01.851959 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) queue_pending_finisher 0x45c04d0
-357> 2015-05-18 06:20:01.918852 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 1) v1 -- ?+0 0x47b81e0 con 0x4609600
-356> 2015-05-18 06:20:01.918871 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 2) v1 -- ?+0 0x47b85a0 con 0x4609600
-355> 2015-05-18 06:20:01.918879 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 3) v1 -- ?+0 0x47b83c0 con 0x4609600
-354> 2015-05-18 06:20:01.918886 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 4) v1 -- ?+0 0x47b8b40 con 0x4609600
-353> 2015-05-18 06:20:01.918894 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 5) v1 -- ?+0 0x47b8960 con 0x4609600
-352> 2015-05-18 06:20:01.918930 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 1) v1 ==== 0+0+0 (0 0 0) 0x47b81e0 con 0x4609600
-351> 2015-05-18 06:20:01.918953 7f09817a0700 10 log_client handle_log_ack log(last 1) v1
-350> 2015-05-18 06:20:01.918956 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801229 mon.0 10.8.128.33:6789/0 1 : cluster [INF] mon.Mon@0 won leader election with quorum 0
-349> 2015-05-18 06:20:01.918977 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 2) v1 ==== 0+0+0 (0 0 0) 0x47b85a0 con 0x4609600
-348> 2015-05-18 06:20:01.918986 7f09817a0700 10 log_client handle_log_ack log(last 2) v1
-347> 2015-05-18 06:20:01.918989 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801314 mon.0 10.8.128.33:6789/0 2 : cluster [INF] monmap e1: 1 mons at {Mon=10.8.128.33:6789/0}
-346> 2015-05-18 06:20:01.918997 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 3) v1 ==== 0+0+0 (0 0 0) 0x47b83c0 con 0x4609600
-345> 2015-05-18 06:20:01.919005 7f09817a0700 10 log_client handle_log_ack log(last 3) v1
-344> 2015-05-18 06:20:01.919007 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801376 mon.0 10.8.128.33:6789/0 3 : cluster [INF] pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
-343> 2015-05-18 06:20:01.919015 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 4) v1 ==== 0+0+0 (0 0 0) 0x47b8b40 con 0x4609600
-342> 2015-05-18 06:20:01.919022 7f09817a0700 10 log_client handle_log_ack log(last 4) v1
-341> 2015-05-18 06:20:01.919025 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801425 mon.0 10.8.128.33:6789/0 4 : cluster [INF] mdsmap e1: 0/0/0 up
-340> 2015-05-18 06:20:01.919034 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 5) v1 ==== 0+0+0 (0 0 0) 0x47b8960 con 0x4609600
-339> 2015-05-18 06:20:01.919041 7f09817a0700 10 log_client handle_log_ack log(last 5) v1
-338> 2015-05-18 06:20:01.919043 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801496 mon.0 10.8.128.33:6789/0 5 : cluster [INF] osdmap e84: 4 osds: 4 up, 4 in
-337> 2015-05-18 06:20:02.473956 7f0989044700 1 -- 10.8.128.33:6789/0 >> :/0 pipe(0x48d0000 sd=20 :6789 s=0 pgs=0 cs=0 l=0 c=0x460a260).accept sd=20 10.8.128.86:48717/0
-336> 2015-05-18 06:20:02.474299 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 1 ==== auth(proto 0 26 bytes epoch 1) v1 ==== 56+0+0 (1708591557 0 0) 0x47f3a80 con 0x460a260
-335> 2015-05-18 06:20:02.474322 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.474323 lease_expire=0.000000 has v0 lc 78624
-334> 2015-05-18 06:20:02.474362 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f2f40 con 0x460a260
-333> 2015-05-18 06:20:02.474629 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (1575692119 0 0) 0x47f4140 con 0x460a260
-332> 2015-05-18 06:20:02.474645 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.474646 lease_expire=0.000000 has v0 lc 78624
-331> 2015-05-18 06:20:02.474885 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f3a80 con 0x460a260
-330> 2015-05-18 06:20:02.475200 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 3 ==== mon_subscribe({monmap=2+,osd_pg_creates=0}) v2 ==== 50+0+0 (3889722309 0 0) 0x466a600 con 0x460a260
-329> 2015-05-18 06:20:02.475267 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8960 con 0x460a260
-328> 2015-05-18 06:20:02.475300 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 4 ==== osd_failure(failed osd.0 10.8.128.40:6800/25974 for 765sec e84 v84) v3 ==== 188+0+0 (2601468610 0 0) 0x46798c0 con 0x460a260
-327> 2015-05-18 06:20:02.475316 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475317 lease_expire=0.000000 has v0 lc 78624
-326> 2015-05-18 06:20:02.475336 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
-325> 2015-05-18 06:20:02.475350 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
-324> 2015-05-18 06:20:02.475353 7f09817a0700 10 log_client _send_to_monlog to self
-323> 2015-05-18 06:20:02.475354 7f09817a0700 10 log_client log_queue is 1 last_log 6 sent 5 num 1 unsent 1 sending 1
-322> 2015-05-18 06:20:02.475356 7f09817a0700 10 log_client will send 2015-05-18 06:20:02.475352 mon.0 10.8.128.33:6789/0 6 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
-321> 2015-05-18 06:20:02.475362 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f3cc0 con 0x4609600
-320> 2015-05-18 06:20:02.475385 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 5 ==== osd_failure(failed osd.3 10.8.128.86:6804/19849 for 723sec e84 v84) v3 ==== 188+0+0 (967876087 0 0) 0x4679e40 con 0x460a260
-319> 2015-05-18 06:20:02.475396 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475397 lease_expire=0.000000 has v0 lc 78624
-318> 2015-05-18 06:20:02.475405 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
-317> 2015-05-18 06:20:02.475418 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
-316> 2015-05-18 06:20:02.475420 7f09817a0700 10 log_client _send_to_monlog to self
-315> 2015-05-18 06:20:02.475421 7f09817a0700 10 log_client log_queue is 2 last_log 7 sent 6 num 2 unsent 1 sending 1
-314> 2015-05-18 06:20:02.475423 7f09817a0700 10 log_client will send 2015-05-18 06:20:02.475419 mon.0 10.8.128.33:6789/0 7 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
-313> 2015-05-18 06:20:02.475428 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2ac0 con 0x4609600
-312> 2015-05-18 06:20:02.475444 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f3cc0 con 0x4609600
-311> 2015-05-18 06:20:02.475452 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475453 lease_expire=0.000000 has v0 lc 78624
-310> 2015-05-18 06:20:02.475470 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2ac0 con 0x4609600
-309> 2015-05-18 06:20:02.475479 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475480 lease_expire=0.000000 has v0 lc 78624
-308> 2015-05-18 06:20:02.475496 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 6 ==== mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=84}) v2 ==== 69+0+0 (2997052209 0 0) 0x466b000 con 0x460a260
-307> 2015-05-18 06:20:02.475517 7f09817a0700 5 mon.Mon@0(leader).osd e84 send_incremental [84..84] to osd.1 10.8.128.86:6800/19549
-306> 2015-05-18 06:20:02.475587 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- osd_map(84..84 src has 1..84) v3 -- ?+0 0x47f4140 con 0x460a260
-305> 2015-05-18 06:20:02.475605 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b83c0 con 0x460a260
-304> 2015-05-18 06:20:02.815632 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes
-303> 2015-05-18 06:20:02.823300 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
-302> 2015-05-18 06:20:02.823308 7f0984a8b700 10 log_client _send_to_monlog to self
-301> 2015-05-18 06:20:02.823309 7f0984a8b700 10 log_client log_queue is 3 last_log 8 sent 7 num 3 unsent 1 sending 1
-300> 2015-05-18 06:20:02.823312 7f0984a8b700 10 log_client will send 2015-05-18 06:20:02.823306 mon.0 10.8.128.33:6789/0 8 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
-299> 2015-05-18 06:20:02.823333 7f0984a8b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x4648d80 con 0x4609600
-298> 2015-05-18 06:20:02.823374 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
-297> 2015-05-18 06:20:02.823377 7f0984a8b700 10 log_client _send_to_monlog to self
-296> 2015-05-18 06:20:02.823378 7f0984a8b700 10 log_client log_queue is 4 last_log 9 sent 8 num 4 unsent 1 sending 1
-295> 2015-05-18 06:20:02.823380 7f0984a8b700 10 log_client will send 2015-05-18 06:20:02.823376 mon.0 10.8.128.33:6789/0 9 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
-294> 2015-05-18 06:20:02.823386 7f0984a8b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4800 con 0x4609600
-293> 2015-05-18 06:20:02.823393 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4648d80 con 0x4609600
-292> 2015-05-18 06:20:02.823407 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'mon_status' '' to 0x45c0330 returned 520 bytes
-291> 2015-05-18 06:20:02.823421 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.823422 lease_expire=0.000000 has v0 lc 78624
-290> 2015-05-18 06:20:02.823450 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4800 con 0x4609600
-289> 2015-05-18 06:20:02.823467 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.823468 lease_expire=0.000000 has v0 lc 78624
-288> 2015-05-18 06:20:02.832644 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 7 ==== osd_failure(failed osd.0 10.8.128.40:6800/25974 for 766sec e84 v84) v3 ==== 188+0+0 (4180528123 0 0) 0x4679b80 con 0x460a260
-287> 2015-05-18 06:20:02.832671 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832672 lease_expire=0.000000 has v0 lc 78624
-286> 2015-05-18 06:20:02.832689 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
-285> 2015-05-18 06:20:02.832704 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
-284> 2015-05-18 06:20:02.832707 7f09817a0700 10 log_client _send_to_monlog to self
-283> 2015-05-18 06:20:02.832708 7f09817a0700 10 log_client log_queue is 5 last_log 10 sent 9 num 5 unsent 1 sending 1
-282> 2015-05-18 06:20:02.832710 7f09817a0700 10 log_client will send 2015-05-18 06:20:02.832706 mon.0 10.8.128.33:6789/0 10 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
-281> 2015-05-18 06:20:02.832717 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2f40 con 0x4609600
-280> 2015-05-18 06:20:02.832739 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 8 ==== osd_failure(failed osd.3 10.8.128.86:6804/19849 for 723sec e84 v84) v3 ==== 188+0+0 (967876087 0 0) 0x467a680 con 0x460a260
-279> 2015-05-18 06:20:02.832751 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832751 lease_expire=0.000000 has v0 lc 78624
-278> 2015-05-18 06:20:02.832760 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
-277> 2015-05-18 06:20:02.832773 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
-276> 2015-05-18 06:20:02.832775 7f09817a0700 10 log_client _send_to_monlog to self
-275> 2015-05-18 06:20:02.832777 7f09817a0700 10 log_client log_queue is 6 last_log 11 sent 10 num 6 unsent 1 sending 1
-274> 2015-05-18 06:20:02.832779 7f09817a0700 10 log_client will send 2015-05-18 06:20:02.832774 mon.0 10.8.128.33:6789/0 11 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
-273> 2015-05-18 06:20:02.832785 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5100 con 0x4609600
-272> 2015-05-18 06:20:02.832807 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2f40 con 0x4609600
-271> 2015-05-18 06:20:02.832817 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832817 lease_expire=0.000000 has v0 lc 78624
-270> 2015-05-18 06:20:02.832833 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5100 con 0x4609600
-269> 2015-05-18 06:20:02.832842 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832844 lease_expire=0.000000 has v0 lc 78624
-268> 2015-05-18 06:20:02.918393 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) queue_pending_finisher 0x45c04c0
-267> 2015-05-18 06:20:02.977214 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 6) v1 -- ?+0 0x47b8780 con 0x4609600
-266> 2015-05-18 06:20:02.977246 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 7) v1 -- ?+0 0x47b81e0 con 0x4609600
-265> 2015-05-18 06:20:02.977254 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 8) v1 -- ?+0 0x47b8b40 con 0x4609600
-264> 2015-05-18 06:20:02.977259 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 9) v1 -- ?+0 0x47b83c0 con 0x4609600
-263> 2015-05-18 06:20:02.977267 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 10) v1 -- ?+0 0x47b8960 con 0x4609600
-262> 2015-05-18 06:20:02.977291 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 11) v1 -- ?+0 0x47b92c0 con 0x4609600
-261> 2015-05-18 06:20:02.977297 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 6) v1 ==== 0+0+0 (0 0 0) 0x47b8780 con 0x4609600
-260> 2015-05-18 06:20:02.977319 7f09817a0700 10 log_client handle_log_ack log(last 6) v1
-259> 2015-05-18 06:20:02.977322 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.475352 mon.0 10.8.128.33:6789/0 6 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
-258> 2015-05-18 06:20:02.977334 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 7) v1 ==== 0+0+0 (0 0 0) 0x47b81e0 con 0x4609600
-257> 2015-05-18 06:20:02.977341 7f09817a0700 10 log_client handle_log_ack log(last 7) v1
-256> 2015-05-18 06:20:02.977343 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.475419 mon.0 10.8.128.33:6789/0 7 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
-255> 2015-05-18 06:20:02.977352 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 8) v1 ==== 0+0+0 (0 0 0) 0x47b8b40 con 0x4609600
-254> 2015-05-18 06:20:02.977375 7f09817a0700 10 log_client handle_log_ack log(last 8) v1
-253> 2015-05-18 06:20:02.977379 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.823306 mon.0 10.8.128.33:6789/0 8 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
-252> 2015-05-18 06:20:02.977388 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 9) v1 ==== 0+0+0 (0 0 0) 0x47b83c0 con 0x4609600
-251> 2015-05-18 06:20:02.977395 7f09817a0700 10 log_client handle_log_ack log(last 9) v1
-250> 2015-05-18 06:20:02.977398 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.823376 mon.0 10.8.128.33:6789/0 9 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
-249> 2015-05-18 06:20:02.977405 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 10) v1 ==== 0+0+0 (0 0 0) 0x47b8960 con 0x4609600
-248> 2015-05-18 06:20:02.977412 7f09817a0700 10 log_client handle_log_ack log(last 10) v1
-247> 2015-05-18 06:20:02.977414 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.832706 mon.0 10.8.128.33:6789/0 10 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
-246> 2015-05-18 06:20:02.977422 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 11) v1 ==== 0+0+0 (0 0 0) 0x47b92c0 con 0x4609600
-245> 2015-05-18 06:20:02.977430 7f09817a0700 10 log_client handle_log_ack log(last 11) v1
-244> 2015-05-18 06:20:02.977431 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.832774 mon.0 10.8.128.33:6789/0 11 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
-243> 2015-05-18 06:20:06.761957 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) is_readable = 1 - now=2015-05-18 06:20:06.761960 lease_expire=0.000000 has v0 lc 78625
-242> 2015-05-18 06:20:06.761973 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) is_readable = 1 - now=2015-05-18 06:20:06.761974 lease_expire=0.000000 has v0 lc 78625
-241> 2015-05-18 06:20:06.761993 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) is_readable = 1 - now=2015-05-18 06:20:06.761993 lease_expire=0.000000 has v0 lc 78625
-240> 2015-05-18 06:20:06.762078 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) queue_pending_finisher 0x45c04f0
-239> 2015-05-18 06:20:07.793435 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes
-238> 2015-05-18 06:20:07.802963 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch
-237> 2015-05-18 06:20:07.802972 7f0984a8b700 10 log_client _send_to_monlog to self
-236> 2015-05-18 06:20:07.802973 7f0984a8b700 10 log_client log_queue is 1 last_log 12 sent 11 num 1 unsent 1 sending 1
-235> 2015-05-18 06:20:07.802977 7f0984a8b700 10 log_client will send 2015-05-18 06:20:07.802970 mon.0 10.8.128.33:6789/0 12 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch
-234> 2015-05-18 06:20:07.802995 7f0984a8b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x4648900 con 0x4609600
-233> 2015-05-18 06:20:07.803031 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished
-232> 2015-05-18 06:20:07.803033 7f0984a8b700 10 log_client _send_to_monlog to self
-231> 2015-05-18 06:20:07.803034 7f0984a8b700 10 log_client log_queue is 2 last_log 13 sent 12 num 2 unsent 1 sending 1
-230> 2015-05-18 06:20:07.803037 7f0984a8b700 10 log_client will send 2015-05-18 06:20:07.803032 mon.0 10.8.128.33:6789/0 13 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished
-229> 2015-05-18 06:20:07.803043 7f0984a8b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600
-228> 2015-05-18 06:20:07.803059 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'mon_status' '' to 0x45c0330 returned 307 bytes
-227> 2015-05-18 06:20:07.803053 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4648900 con 0x4609600
-226> 2015-05-18 06:20:07.803083 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.803084 lease_expire=0.000000 has v0 lc 78626
-225> 2015-05-18 06:20:07.803114 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600
-224> 2015-05-18 06:20:07.803123 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.803125 lease_expire=0.000000 has v0 lc 78626
-223> 2015-05-18 06:20:07.803590 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes
-222> 2015-05-18 06:20:07.810261 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78626) queue_pending_finisher 0x45c0320
-221> 2015-05-18 06:20:07.812704 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'version' '' to 0x45b80b8 returned 20 bytes
-220> 2015-05-18 06:20:07.833354 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 9 ==== osd_failure(failed osd.0 10.8.128.40:6800/25974 for 771sec e84 v84) v3 ==== 188+0+0 (3406336582 0 0) 0x467a3c0 con 0x460a260
-219> 2015-05-18 06:20:07.835570 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835574 lease_expire=0.000000 has v0 lc 78626
-218> 2015-05-18 06:20:07.835601 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
-217> 2015-05-18 06:20:07.835627 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
-216> 2015-05-18 06:20:07.835632 7f09817a0700 10 log_client _send_to_monlog to self
-215> 2015-05-18 06:20:07.835633 7f09817a0700 10 log_client log_queue is 3 last_log 14 sent 13 num 3 unsent 1 sending 1
-214> 2015-05-18 06:20:07.835639 7f09817a0700 10 log_client will send 2015-05-18 06:20:07.835630 mon.0 10.8.128.33:6789/0 14 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
-213> 2015-05-18 06:20:07.835652 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4ec0 con 0x4609600
-212> 2015-05-18 06:20:07.835680 7f09817a0700 1 mon.Mon@0(leader).osd e84 we have enough reports/reporters to mark osd.0 down
-211> 2015-05-18 06:20:07.835700 7f09817a0700 0 log_channel(cluster) log [INF] : osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211)
-210> 2015-05-18 06:20:07.835703 7f09817a0700 10 log_client _send_to_monlog to self
-209> 2015-05-18 06:20:07.835704 7f09817a0700 10 log_client log_queue is 4 last_log 15 sent 14 num 4 unsent 1 sending 1
-208> 2015-05-18 06:20:07.835707 7f09817a0700 10 log_client will send 2015-05-18 06:20:07.835702 mon.0 10.8.128.33:6789/0 15 : cluster [INF] osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211)
-207> 2015-05-18 06:20:07.835722 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4c80 con 0x4609600
-206> 2015-05-18 06:20:07.835744 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4ec0 con 0x4609600
-205> 2015-05-18 06:20:07.835759 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835761 lease_expire=0.000000 has v0 lc 78626
-204> 2015-05-18 06:20:07.835780 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4c80 con 0x4609600
-203> 2015-05-18 06:20:07.835809 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835810 lease_expire=0.000000 has v0 lc 78626
-202> 2015-05-18 06:20:07.835826 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 10 ==== osd_failure(failed osd.3 10.8.128.86:6804/19849 for 728sec e84 v84) v3 ==== 188+0+0 (2916513576 0 0) 0x467a100 con 0x460a260
-201> 2015-05-18 06:20:07.835853 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835854 lease_expire=0.000000 has v0 lc 78626
-200> 2015-05-18 06:20:07.835868 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
-199> 2015-05-18 06:20:07.835888 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
-198> 2015-05-18 06:20:07.835892 7f09817a0700 10 log_client _send_to_monlog to self
-197> 2015-05-18 06:20:07.835893 7f09817a0700 10 log_client log_queue is 5 last_log 16 sent 15 num 5 unsent 1 sending 1
-196> 2015-05-18 06:20:07.835897 7f09817a0700 10 log_client will send 2015-05-18 06:20:07.835891 mon.0 10.8.128.33:6789/0 16 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
-195> 2015-05-18 06:20:07.835907 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4a40 con 0x4609600
-194> 2015-05-18 06:20:07.835925 7f09817a0700 1 mon.Mon@0(leader).osd e84 we have enough reports/reporters to mark osd.3 down
-193> 2015-05-18 06:20:07.835939 7f09817a0700 0 log_channel(cluster) log [INF] : osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015)
-192> 2015-05-18 06:20:07.835942 7f09817a0700 10 log_client _send_to_monlog to self
-191> 2015-05-18 06:20:07.835943 7f09817a0700 10 log_client log_queue is 6 last_log 17 sent 16 num 6 unsent 1 sending 1
-190> 2015-05-18 06:20:07.835946 7f09817a0700 10 log_client will send 2015-05-18 06:20:07.835941 mon.0 10.8.128.33:6789/0 17 : cluster [INF] osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015)
-189> 2015-05-18 06:20:07.835956 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5c40 con 0x4609600
-188> 2015-05-18 06:20:07.835984 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4a40 con 0x4609600
-187> 2015-05-18 06:20:07.835997 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835998 lease_expire=0.000000 has v0 lc 78626
-186> 2015-05-18 06:20:07.836011 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5c40 con 0x4609600
-185> 2015-05-18 06:20:07.836021 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.836022 lease_expire=0.000000 has v0 lc 78626
-184> 2015-05-18 06:20:07.860966 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 12) v1 -- ?+0 0x47b90e0 con 0x4609600
-183> 2015-05-18 06:20:07.860985 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 13) v1 -- ?+0 0x47b8f00 con 0x4609600
-182> 2015-05-18 06:20:07.860993 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.860994 lease_expire=0.000000 has v0 lc 78627
-181> 2015-05-18 06:20:07.861010 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.861011 lease_expire=0.000000 has v0 lc 78627
-180> 2015-05-18 06:20:07.861023 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.861024 lease_expire=0.000000 has v0 lc 78627
-179> 2015-05-18 06:20:07.861034 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.861035 lease_expire=0.000000 has v0 lc 78627
-178> 2015-05-18 06:20:07.861029 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 12) v1 ==== 0+0+0 (0 0 0) 0x47b90e0 con 0x4609600
-177> 2015-05-18 06:20:07.861052 7f09817a0700 10 log_client handle_log_ack log(last 12) v1
-176> 2015-05-18 06:20:07.861057 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.802970 mon.0 10.8.128.33:6789/0 12 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch
-175> 2015-05-18 06:20:07.861074 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 13) v1 ==== 0+0+0 (0 0 0) 0x47b8f00 con 0x4609600
-174> 2015-05-18 06:20:07.861086 7f09817a0700 10 log_client handle_log_ack log(last 13) v1
-173> 2015-05-18 06:20:07.861090 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.803032 mon.0 10.8.128.33:6789/0 13 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished
-172> 2015-05-18 06:20:07.885804 7f0981fa1700 2 mon.Mon@0(leader).osd e84 osd.0 DOWN
-171> 2015-05-18 06:20:07.885814 7f0981fa1700 2 mon.Mon@0(leader).osd e84 osd.3 DOWN
-170> 2015-05-18 06:20:07.885893 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) queue_pending_finisher 0x45c04d0
-169> 2015-05-18 06:20:07.937432 7f098370b700 1 mon.Mon@0(leader).osd e85 e85: 4 osds: 2 up, 4 in
-168> 2015-05-18 06:20:07.962214 7f098370b700 5 mon.Mon@0(leader).paxos(paxos refresh c 78062..78628) is_readable = 0 - now=2015-05-18 06:20:07.962218 lease_expire=0.000000 has v0 lc 78628
-167> 2015-05-18 06:20:07.962309 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- osd_map(84..85 src has 1..85) v3 -- ?+0 0x47f2400 con 0x460a260
-166> 2015-05-18 06:20:07.962334 7f098370b700 5 mon.Mon@0(leader).osd e85 send_latest to osd.1 10.8.128.86:6800/19549 start 84
-165> 2015-05-18 06:20:07.962341 7f098370b700 5 mon.Mon@0(leader).osd e85 send_incremental [84..85] to osd.1 10.8.128.86:6800/19549
-164> 2015-05-18 06:20:07.962379 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f2f40 con 0x460a260
-163> 2015-05-18 06:20:07.962394 7f098370b700 5 mon.Mon@0(leader).osd e85 send_latest to osd.1 10.8.128.86:6800/19549 start 84
-162> 2015-05-18 06:20:07.962399 7f098370b700 5 mon.Mon@0(leader).osd e85 send_incremental [84..85] to osd.1 10.8.128.86:6800/19549
-161> 2015-05-18 06:20:07.962590 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78628) is_readable = 1 - now=2015-05-18 06:20:07.962592 lease_expire=0.000000 has v0 lc 78628
-160> 2015-05-18 06:20:07.962922 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78628) queue_pending_finisher 0x45c04d0
-159> 2015-05-18 06:20:07.987480 7f098370b700 0 log_channel(cluster) log [INF] : osdmap e85: 4 osds: 2 up, 4 in
-158> 2015-05-18 06:20:07.987489 7f098370b700 10 log_client _send_to_monlog to self
-157> 2015-05-18 06:20:07.987492 7f098370b700 10 log_client log_queue is 5 last_log 18 sent 17 num 5 unsent 1 sending 1
-156> 2015-05-18 06:20:07.987495 7f098370b700 10 log_client will send 2015-05-18 06:20:07.987487 mon.0 10.8.128.33:6789/0 18 : cluster [INF] osdmap e85: 4 osds: 2 up, 4 in
-155> 2015-05-18 06:20:07.987506 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4800 con 0x4609600
-154> 2015-05-18 06:20:07.987564 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4800 con 0x4609600
-153> 2015-05-18 06:20:07.987582 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78628) is_readable = 1 - now=2015-05-18 06:20:07.987583 lease_expire=0.000000 has v0 lc 78628
-152> 2015-05-18 06:20:08.005319 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 11 ==== mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=86}) v2 ==== 69+0+0 (1597159760 0 0) 0x466b400 con 0x460a260
-151> 2015-05-18 06:20:08.005352 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8f00 con 0x460a260
-150> 2015-05-18 06:20:08.013617 7f098370b700 0 log_channel(cluster) log [INF] : pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
-149> 2015-05-18 06:20:08.013624 7f098370b700 10 log_client _send_to_monlog to self
-148> 2015-05-18 06:20:08.013625 7f098370b700 10 log_client log_queue is 6 last_log 19 sent 18 num 6 unsent 1 sending 1
-147> 2015-05-18 06:20:08.013627 7f098370b700 10 log_client will send 2015-05-18 06:20:08.013623 mon.0 10.8.128.33:6789/0 19 : cluster [INF] pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
-146> 2015-05-18 06:20:08.013637 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5100 con 0x4609600
-145> 2015-05-18 06:20:08.013689 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5100 con 0x4609600
-144> 2015-05-18 06:20:08.013702 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78629) is_readable = 1 - now=2015-05-18 06:20:08.013703 lease_expire=0.000000 has v0 lc 78629
-143> 2015-05-18 06:20:08.860774 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78629) queue_pending_finisher 0x45c0510
-142> 2015-05-18 06:20:08.909048 7f097f69b700 1 -- 10.8.128.33:6789/0 >> :/0 pipe(0x48e2000 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x460a3c0).accept sd=21 10.8.128.33:60116/0
-141> 2015-05-18 06:20:08.909248 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x47f4140 con 0x460a3c0
-140> 2015-05-18 06:20:08.909277 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78629) is_readable = 1 - now=2015-05-18 06:20:08.909278 lease_expire=0.000000 has v0 lc 78629
-139> 2015-05-18 06:20:08.909363 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78629) queue_pending_finisher 0x45c0500
-138> 2015-05-18 06:20:08.913369 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.913371 lease_expire=0.000000 has v0 lc 78630
-137> 2015-05-18 06:20:08.913393 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_map magic: 0 v1 -- ?+0 0x47f2ac0 con 0x460a3c0
-136> 2015-05-18 06:20:08.913421 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4648900 con 0x460a3c0
-135> 2015-05-18 06:20:08.913486 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 14) v1 -- ?+0 0x47b85a0 con 0x4609600
-134> 2015-05-18 06:20:08.913503 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 15) v1 -- ?+0 0x47b8960 con 0x4609600
-133> 2015-05-18 06:20:08.913512 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 16) v1 -- ?+0 0x47b83c0 con 0x4609600
-132> 2015-05-18 06:20:08.913519 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 17) v1 -- ?+0 0x47b8b40 con 0x4609600
-131> 2015-05-18 06:20:08.913528 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 18) v1 -- ?+0 0x47b81e0 con 0x4609600
-130> 2015-05-18 06:20:08.913536 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 19) v1 -- ?+0 0x47b8780 con 0x4609600
-129> 2015-05-18 06:20:08.913547 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 14) v1 ==== 0+0+0 (0 0 0) 0x47b85a0 con 0x4609600
-128> 2015-05-18 06:20:08.938024 7f09817a0700 10 log_client handle_log_ack log(last 14) v1
-127> 2015-05-18 06:20:08.938034 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.835630 mon.0 10.8.128.33:6789/0 14 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
-126> 2015-05-18 06:20:08.938051 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 15) v1 ==== 0+0+0 (0 0 0) 0x47b8960 con 0x4609600
-125> 2015-05-18 06:20:08.938060 7f09817a0700 10 log_client handle_log_ack log(last 15) v1
-124> 2015-05-18 06:20:08.938063 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.835702 mon.0 10.8.128.33:6789/0 15 : cluster [INF] osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211)
-123> 2015-05-18 06:20:08.938071 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 16) v1 ==== 0+0+0 (0 0 0) 0x47b83c0 con 0x4609600
-122> 2015-05-18 06:20:08.938079 7f09817a0700 10 log_client handle_log_ack log(last 16) v1
-121> 2015-05-18 06:20:08.938083 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.835891 mon.0 10.8.128.33:6789/0 16 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
-120> 2015-05-18 06:20:08.938092 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 17) v1 ==== 0+0+0 (0 0 0) 0x47b8b40 con 0x4609600
-119> 2015-05-18 06:20:08.938099 7f09817a0700 10 log_client handle_log_ack log(last 17) v1
-118> 2015-05-18 06:20:08.938101 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.835941 mon.0 10.8.128.33:6789/0 17 : cluster [INF] osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015)
-117> 2015-05-18 06:20:08.938109 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 18) v1 ==== 0+0+0 (0 0 0) 0x47b81e0 con 0x4609600
-116> 2015-05-18 06:20:08.938116 7f09817a0700 10 log_client handle_log_ack log(last 18) v1
-115> 2015-05-18 06:20:08.938119 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.987487 mon.0 10.8.128.33:6789/0 18 : cluster [INF] osdmap e85: 4 osds: 2 up, 4 in
-114> 2015-05-18 06:20:08.938127 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 19) v1 ==== 0+0+0 (0 0 0) 0x47b8780 con 0x4609600
-113> 2015-05-18 06:20:08.938135 7f09817a0700 10 log_client handle_log_ack log(last 19) v1
-112> 2015-05-18 06:20:08.938137 7f09817a0700 10 log_client logged 2015-05-18 06:20:08.013623 mon.0 10.8.128.33:6789/0 19 : cluster [INF] pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
-111> 2015-05-18 06:20:08.938145 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (309592415 0 0) 0x47f2ac0 con 0x460a3c0
-110> 2015-05-18 06:20:08.938154 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.938155 lease_expire=0.000000 has v0 lc 78630
-109> 2015-05-18 06:20:08.938432 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f4380 con 0x460a3c0
-108> 2015-05-18 06:20:08.938705 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (3478599268 0 0) 0x4648900 con 0x460a3c0
-107> 2015-05-18 06:20:08.938717 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.938718 lease_expire=0.000000 has v0 lc 78630
-106> 2015-05-18 06:20:08.938852 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f2ac0 con 0x460a3c0
-105> 2015-05-18 06:20:08.939109 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x466b000 con 0x460a3c0
-104> 2015-05-18 06:20:08.939138 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_map magic: 0 v1 -- ?+0 0x4648900 con 0x460a3c0
-103> 2015-05-18 06:20:08.939153 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8780 con 0x460a3c0
-102> 2015-05-18 06:20:08.939168 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 5 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x466a600 con 0x460a3c0
-101> 2015-05-18 06:20:08.939244 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f5a00 con 0x460a3c0
-100> 2015-05-18 06:20:08.939266 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b81e0 con 0x460a3c0
-99> 2015-05-18 06:20:08.939293 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 6 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x466b400 con 0x460a3c0
-98> 2015-05-18 06:20:08.939342 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f57c0 con 0x460a3c0
-97> 2015-05-18 06:20:08.939354 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8960 con 0x460a3c0
-96> 2015-05-18 06:20:08.949743 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 7 ==== mon_command({"prefix": "mon_status", "format": "json"} v 0) v1 ==== 84+0+0 (110991942 0 0) 0x466b400 con 0x460a3c0
-95> 2015-05-18 06:20:08.949801 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "mon_status", "format": "json"} v 0) v1
-94> 2015-05-18 06:20:08.949836 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "mon_status", "format": "json"}]: dispatch
-93> 2015-05-18 06:20:08.949839 7f09817a0700 10 log_client _send_to_monlog to self
-92> 2015-05-18 06:20:08.949840 7f09817a0700 10 log_client log_queue is 1 last_log 20 sent 19 num 1 unsent 1 sending 1
-91> 2015-05-18 06:20:08.949843 7f09817a0700 10 log_client will send 2015-05-18 06:20:08.949838 mon.0 10.8.128.33:6789/0 20 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "mon_status", "format": "json"}]: dispatch
-90> 2015-05-18 06:20:08.949851 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5340 con 0x4609600
-89> 2015-05-18 06:20:08.949891 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"prefix": "mon_status", "format": "json"}]=0 v0) v1 -- ?+307 0x4668e00 con 0x460a3c0
-88> 2015-05-18 06:20:08.949910 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5340 con 0x4609600
-87> 2015-05-18 06:20:08.949919 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.949920 lease_expire=0.000000 has v0 lc 78630
-86> 2015-05-18 06:20:08.951317 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 8 ==== mon_command({"prefix": "status", "format": "json"} v 0) v1 ==== 80+0+0 (1009655278 0 0) 0x466a600 con 0x460a3c0
-85> 2015-05-18 06:20:08.951357 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1
-84> 2015-05-18 06:20:08.951380 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
-83> 2015-05-18 06:20:08.951382 7f09817a0700 10 log_client _send_to_monlog to self
-82> 2015-05-18 06:20:08.951384 7f09817a0700 10 log_client log_queue is 2 last_log 21 sent 20 num 2 unsent 1 sending 1
-81> 2015-05-18 06:20:08.951386 7f09817a0700 10 log_client will send 2015-05-18 06:20:08.951382 mon.0 10.8.128.33:6789/0 21 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
-80> 2015-05-18 06:20:08.951394 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5580 con 0x4609600
-79> 2015-05-18 06:20:08.951519 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 -- ?+1108 0x466b400 con 0x460a3c0
-78> 2015-05-18 06:20:08.951538 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5580 con 0x4609600
-77> 2015-05-18 06:20:08.951548 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.951548 lease_expire=0.000000 has v0 lc 78630
-76> 2015-05-18 06:20:08.952793 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 9 ==== mon_command({"prefix": "health", "detail": "", "format": "json"} v 0) v1 ==== 94+0+0 (1843246703 0 0) 0x466b000 con 0x460a3c0
-75> 2015-05-18 06:20:08.952829 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "health", "detail": "", "format": "json"} v 0) v1
-74> 2015-05-18 06:20:08.952852 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "health", "detail": "", "format": "json"}]: dispatch
-73> 2015-05-18 06:20:08.952855 7f09817a0700 10 log_client _send_to_monlog to self
-72> 2015-05-18 06:20:08.952856 7f09817a0700 10 log_client log_queue is 3 last_log 22 sent 21 num 3 unsent 1 sending 1
-71> 2015-05-18 06:20:08.952858 7f09817a0700 10 log_client will send 2015-05-18 06:20:08.952854 mon.0 10.8.128.33:6789/0 22 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "health", "detail": "", "format": "json"}]: dispatch
-70> 2015-05-18 06:20:08.952864 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2f40 con 0x4609600
-69> 2015-05-18 06:20:08.952941 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"prefix": "health", "detail": "", "format": "json"}]=0 v0) v1 -- ?+401 0x466a600 con 0x460a3c0
-68> 2015-05-18 06:20:08.952960 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2f40 con 0x4609600
-67> 2015-05-18 06:20:08.952970 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.952970 lease_expire=0.000000 has v0 lc 78630
-66> 2015-05-18 06:20:08.954184 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 10 ==== mon_command({"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"} v 0) v1 ==== 112+0+0 (3334637970 0 0) 0x4669800 con 0x460a3c0
-65> 2015-05-18 06:20:08.954227 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"} v 0) v1
-64> 2015-05-18 06:20:08.954251 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]: dispatch
-63> 2015-05-18 06:20:08.954255 7f09817a0700 10 log_client _send_to_monlog to self
-62> 2015-05-18 06:20:08.954256 7f09817a0700 10 log_client log_queue is 4 last_log 23 sent 22 num 4 unsent 1 sending 1
-61> 2015-05-18 06:20:08.954258 7f09817a0700 10 log_client will send 2015-05-18 06:20:08.954254 mon.0 10.8.128.33:6789/0 23 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]: dispatch
-60> 2015-05-18 06:20:08.954266 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f60c0 con 0x4609600
-59> 2015-05-18 06:20:08.954285 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.954286 lease_expire=0.000000 has v0 lc 78630
-58> 2015-05-18 06:20:08.954605 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]=0 dumped pgs_brief in format json v568) v1 -- ?+6616 0x466b000 con 0x460a3c0
-57> 2015-05-18 06:20:08.954626 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f60c0 con 0x4609600
-56> 2015-05-18 06:20:08.954636 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.954636 lease_expire=0.000000 has v0 lc 78630
-55> 2015-05-18 06:20:08.957218 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes
-54> 2015-05-18 06:20:08.965521 7f0984a8b700 1 do_command 'config show' 'format:json
-53> 2015-05-18 06:20:08.970043 7f0984a8b700 1 do_command 'config show' 'format:json result is 26602 bytes
-52> 2015-05-18 06:20:08.970050 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'config show' '' to 0x45c0050 returned 26602 bytes
-51> 2015-05-18 06:20:09.029915 7f097f69b700 2 -- 10.8.128.33:6789/0 >> 10.8.128.33:0/1004034 pipe(0x48e2000 sd=21 :6789 s=2 pgs=1 cs=1 l=1 c=0x460a3c0).reader couldn't read tag, (0) Success
-50> 2015-05-18 06:20:09.029958 7f097f69b700 2 -- 10.8.128.33:6789/0 >> 10.8.128.33:0/1004034 pipe(0x48e2000 sd=21 :6789 s=2 pgs=1 cs=1 l=1 c=0x460a3c0).fault (0) Success
-49> 2015-05-18 06:20:09.134600 7f097f69b700 1 -- 10.8.128.33:6789/0 >> :/0 pipe(0x48e2000 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x460aaa0).accept sd=21 10.8.128.33:60119/0
-48> 2015-05-18 06:20:09.134790 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x47f5a00 con 0x460aaa0
-47> 2015-05-18 06:20:09.134816 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.134817 lease_expire=0.000000 has v0 lc 78631
-46> 2015-05-18 06:20:09.134841 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_map magic: 0 v1 -- ?+0 0x47f7740 con 0x460aaa0
-45> 2015-05-18 06:20:09.134870 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f7980 con 0x460aaa0
-44> 2015-05-18 06:20:09.135204 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (2513201549 0 0) 0x47f7980 con 0x460aaa0
-43> 2015-05-18 06:20:09.135222 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.135223 lease_expire=0.000000 has v0 lc 78631
-42> 2015-05-18 06:20:09.135370 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f5a00 con 0x460aaa0
-41> 2015-05-18 06:20:09.135590 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (692588905 0 0) 0x47f7740 con 0x460aaa0
-40> 2015-05-18 06:20:09.135605 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.135606 lease_expire=0.000000 has v0 lc 78631
-39> 2015-05-18 06:20:09.135738 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f7980 con 0x460aaa0
-38> 2015-05-18 06:20:09.135936 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x466a600 con 0x460aaa0
-37> 2015-05-18 06:20:09.135967 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_map magic: 0 v1 -- ?+0 0x47f7740 con 0x460aaa0
-36> 2015-05-18 06:20:09.135984 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b85a0 con 0x460aaa0
-35> 2015-05-18 06:20:09.136000 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 5 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x4668e00 con 0x460aaa0
-34> 2015-05-18 06:20:09.136101 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f2ac0 con 0x460aaa0
-33> 2015-05-18 06:20:09.136120 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b90e0 con 0x460aaa0
-32> 2015-05-18 06:20:09.136139 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 6 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x466b000 con 0x460aaa0
-31> 2015-05-18 06:20:09.136183 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f5e80 con 0x460aaa0
-30> 2015-05-18 06:20:09.136195 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b92c0 con 0x460aaa0
-29> 2015-05-18 06:20:09.143538 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 7 ==== mon_command({"prefix": "status", "format": "json"} v 0) v1 ==== 80+0+0 (1009655278 0 0) 0x4669800 con 0x460aaa0
-28> 2015-05-18 06:20:09.143603 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1
-27> 2015-05-18 06:20:09.143635 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
-26> 2015-05-18 06:20:09.143638 7f09817a0700 10 log_client _send_to_monlog to self
-25> 2015-05-18 06:20:09.143639 7f09817a0700 10 log_client log_queue is 5 last_log 24 sent 23 num 5 unsent 1 sending 1
-24> 2015-05-18 06:20:09.143642 7f09817a0700 10 log_client will send 2015-05-18 06:20:09.143637 mon.0 10.8.128.33:6789/0 24 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
-23> 2015-05-18 06:20:09.143652 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600
-22> 2015-05-18 06:20:09.143795 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 -- ?+1108 0x466b000 con 0x460aaa0
-21> 2015-05-18 06:20:09.143817 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600
-20> 2015-05-18 06:20:09.143827 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.143828 lease_expire=0.000000 has v0 lc 78631
-19> 2015-05-18 06:20:09.145454 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 8 ==== mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 ==== 82+0+0 (3917390200 0 0) 0x466a000 con 0x460aaa0
-18> 2015-05-18 06:20:09.145494 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd dump", "format": "json"} v 0) v1
-17> 2015-05-18 06:20:09.145520 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch
-16> 2015-05-18 06:20:09.145522 7f09817a0700 10 log_client _send_to_monlog to self
-15> 2015-05-18 06:20:09.145523 7f09817a0700 10 log_client log_queue is 6 last_log 25 sent 24 num 6 unsent 1 sending 1
-14> 2015-05-18 06:20:09.145527 7f09817a0700 10 log_client will send 2015-05-18 06:20:09.145522 mon.0 10.8.128.33:6789/0 25 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch
-13> 2015-05-18 06:20:09.145535 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f6540 con 0x4609600
-12> 2015-05-18 06:20:09.145546 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.145547 lease_expire=0.000000 has v0 lc 78631
-11> 2015-05-18 06:20:09.145779 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 -- ?+3436 0x4669800 con 0x460aaa0
-10> 2015-05-18 06:20:09.145805 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f6540 con 0x4609600
-9> 2015-05-18 06:20:09.145814 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.145815 lease_expire=0.000000 has v0 lc 78631
-8> 2015-05-18 06:20:09.147249 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 9 ==== mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1 ==== 95+0+0 (942654298 0 0) 0x466b800 con 0x460aaa0
-7> 2015-05-18 06:20:09.147309 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1
-6> 2015-05-18 06:20:09.147337 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch
-5> 2015-05-18 06:20:09.147343 7f09817a0700 10 log_client _send_to_monlog to self
-4> 2015-05-18 06:20:09.147345 7f09817a0700 10 log_client log_queue is 7 last_log 26 sent 25 num 7 unsent 1 sending 1
-3> 2015-05-18 06:20:09.147347 7f09817a0700 10 log_client will send 2015-05-18 06:20:09.147343 mon.0 10.8.128.33:6789/0 26 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch
-2> 2015-05-18 06:20:09.147355 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f3cc0 con 0x4609600
-1> 2015-05-18 06:20:09.147365 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.147366 lease_expire=0.000000 has v0 lc 78631
0> 2015-05-18 06:20:09.149437 7f09817a0700 -1 *** Caught signal (Aborted) **
in thread 7f09817a0700

ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
1: /usr/bin/ceph-mon() [0x9017e2]
2: (()+0xf130) [0x7f098826e130]
3: (gsignal()+0x37) [0x7f0986c885d7]
4: (abort()+0x148) [0x7f0986c89cc8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f098758c9b5]
6: (()+0x5e926) [0x7f098758a926]
7: (()+0x5e953) [0x7f098758a953]
8: (()+0x5eb73) [0x7f098758ab73]
9: (std::__throw_logic_error(char const*)+0x77) [0x7f09875df717]
10: (char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag)+0xa1) [0x7f09875eb561]
11: (std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)+0x38) [0x7f09875eb918]
12: (CrushTreeDumper::dump_item_fields(CrushWrapper const*, CrushTreeDumper::Item const&, ceph::Formatter*)+0xb9) [0x6175c9]
13: (OSDMap::print_tree(std::ostream*, ceph::Formatter*) const+0x10d9) [0x787ab9]
14: (OSDMonitor::preprocess_command(MMonCommand*)+0xe55) [0x60a0b5]
15: (OSDMonitor::preprocess_query(PaxosServiceMessage*)+0x20b) [0x60f75b]
16: (PaxosService::dispatch(PaxosServiceMessage*)+0x833) [0x5cacd3]
17: (Monitor::handle_command(MMonCommand*)+0x147c) [0x591a4c]
18: (Monitor::dispatch(MonSession*, Message*, bool)+0xf9) [0x594c89]
19: (Monitor::_ms_dispatch(Message*)+0x1a6) [0x595936]
20: (Monitor::ms_dispatch(Message*)+0x23) [0x5b5403]
21: (DispatchQueue::entry()+0x64a) [0x8a1d9a]
22: (DispatchQueue::DispatchThread::entry()+0xd) [0x79bd9d]
23: (()+0x7df5) [0x7f0988266df5]
24: (clone()+0x6d) [0x7f0986d491ad]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mon.Mon.log
--- end dump of recent events ---

===================================


(1-1/5)