2015-02-25 10:35:09.475014 7fe2b8a607a0 0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578), process ceph-mon, pid 29869 2015-02-25 10:35:09.531266 7fe2b8a607a0 0 starting mon.c rank 0 at 172.20.32.100:6789/0 mon_data /var/lib/ceph/mon/ceph-c fsid 8f5189d6-b197-4400-87c6-aeead2fc6f4b 2015-02-25 10:35:09.531714 7fe2b8a607a0 1 mon.c@-1(probing) e1 preinit fsid 8f5189d6-b197-4400-87c6-aeead2fc6f4b 2015-02-25 10:35:09.532040 7fe2b8a607a0 1 mon.c@-1(probing).paxosservice(pgmap 1..84) refresh upgraded, format 0 -> 1 2015-02-25 10:35:09.532082 7fe2b8a607a0 1 mon.c@-1(probing).pg v0 on_upgrade discarding in-core PGMap 2015-02-25 10:35:09.540601 7fe2b8a607a0 0 mon.c@-1(probing).mds e1 print_map epoch 1 flags 0 created 0.000000 modified 2015-02-25 10:24:24.156438 tableserver 0 root 0 session_timeout 0 session_autoclose 0 max_file_size 0 last_failure 0 last_failure_osd_epoch 0 compat compat={},rocompat={},incompat={} max_mds 0 in up {} failed stopped data_pools metadata_pool 0 inline_data disabled 2015-02-25 10:35:09.540884 7fe2b8a607a0 0 mon.c@-1(probing).osd e33 crush map has features 1107558400, adjusting msgr requires 2015-02-25 10:35:09.540923 7fe2b8a607a0 0 mon.c@-1(probing).osd e33 crush map has features 1107558400, adjusting msgr requires 2015-02-25 10:35:09.540954 7fe2b8a607a0 0 mon.c@-1(probing).osd e33 crush map has features 1107558400, adjusting msgr requires 2015-02-25 10:35:09.540981 7fe2b8a607a0 0 mon.c@-1(probing).osd e33 crush map has features 1107558400, adjusting msgr requires 2015-02-25 10:35:09.541386 7fe2b8a607a0 1 mon.c@-1(probing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 1 2015-02-25 10:35:09.543724 7fe2b8a607a0 0 mon.c@-1(probing) e1 my rank is now 0 (was -1) 2015-02-25 10:35:09.543793 7fe2b8a607a0 1 mon.c@0(probing) e1 win_standalone_election 2015-02-25 10:35:09.547144 7fe2b8a607a0 0 log_channel(cluster) log [INF] : mon.c@0 won leader election with quorum 0 2015-02-25 10:35:09.547243 7fe2b8a607a0 0 log_channel(cluster) log [INF] : monmap e1: 1 mons at {c=172.20.32.100:6789/0} 2015-02-25 10:35:09.547332 7fe2b8a607a0 0 log_channel(cluster) log [INF] : pgmap v84: 768 pgs: 768 active+clean; 1646 bytes data, 20353 MB used, 77580 MB / 100 GB avail 2015-02-25 10:35:09.547386 7fe2b8a607a0 0 log_channel(cluster) log [INF] : mdsmap e1: 0/0/0 up 2015-02-25 10:35:09.547482 7fe2b8a607a0 0 log_channel(cluster) log [INF] : osdmap e33: 3 osds: 3 up, 3 in 2015-02-25 10:35:09.804473 7fe2b4301700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch 2015-02-25 10:35:09.804593 7fe2b4301700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished 2015-02-25 10:35:09.957715 7fe2b075c700 0 log_channel(cluster) log [INF] : osd.2 marked itself down 2015-02-25 10:35:12.205123 7fe2b2b00700 1 mon.c@0(leader).osd e34 e34: 3 osds: 2 up, 3 in 2015-02-25 10:35:12.220212 7fe2b2b00700 0 log_channel(cluster) log [INF] : osd.0 marked itself down 2015-02-25 10:35:13.011926 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e34: 3 osds: 2 up, 3 in 2015-02-25 10:35:13.035201 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v85: 768 pgs: 253 stale+active+clean, 515 active+clean; 1646 bytes data, 20353 MB used, 77580 MB / 100 GB avail 2015-02-25 10:35:13.217069 7fe2b2b00700 1 mon.c@0(leader).osd e35 e35: 3 osds: 1 up, 3 in 2015-02-25 10:35:13.236799 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e35: 3 osds: 1 up, 3 in 2015-02-25 10:35:13.255912 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v86: 768 pgs: 527 stale+active+clean, 241 active+clean; 1646 bytes data, 20353 MB used, 77580 MB / 100 GB avail 2015-02-25 10:35:13.302322 7fe2b075c700 0 mon.c@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "args": ["host=set004", "root=default"], "id": 2, "weight": 0.040000000000000001} v 0) v1 2015-02-25 10:35:13.302449 7fe2b075c700 0 log_channel(audit) log [INF] : from='client.? 172.20.32.100:0/1030197' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "args": ["host=set004", "root=default"], "id": 2, "weight": 0.040000000000000001}]: dispatch 2015-02-25 10:35:13.302698 7fe2b075c700 0 mon.c@0(leader).osd e35 create-or-move crush item name 'osd.2' initial_weight 0.04 at location {host=set004,root=default} 2015-02-25 10:35:13.886747 7fe2b075c700 0 mon.c@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "args": ["host=set002", "root=default"], "id": 0, "weight": 0.01} v 0) v1 2015-02-25 10:35:13.886810 7fe2b075c700 0 log_channel(audit) log [INF] : from='client.? 172.20.32.5:0/1029790' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "args": ["host=set002", "root=default"], "id": 0, "weight": 0.01}]: dispatch 2015-02-25 10:35:13.886934 7fe2b075c700 0 mon.c@0(leader).osd e35 create-or-move crush item name 'osd.0' initial_weight 0.01 at location {host=set002,root=default} 2015-02-25 10:35:14.222790 7fe2b2b00700 1 mon.c@0(leader).osd e36 e36: 3 osds: 1 up, 3 in 2015-02-25 10:35:14.230069 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e36: 3 osds: 1 up, 3 in 2015-02-25 10:35:14.239282 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v87: 768 pgs: 527 stale+active+clean, 241 active+clean; 1646 bytes data, 20353 MB used, 77580 MB / 100 GB avail 2015-02-25 10:35:14.612336 7fe2b075c700 0 log_channel(cluster) log [INF] : osd.1 marked itself down 2015-02-25 10:35:15.564525 7fe2b2b00700 1 mon.c@0(leader).osd e37 e37: 3 osds: 2 up, 3 in 2015-02-25 10:35:15.569152 7fe2b2b00700 0 log_channel(cluster) log [INF] : osd.2 172.20.32.100:6804/30245 boot 2015-02-25 10:35:15.572777 7fe2b2b00700 0 log_channel(cluster) log [INF] : osd.0 172.20.32.5:6801/29835 boot 2015-02-25 10:35:15.603709 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e37: 3 osds: 2 up, 3 in 2015-02-25 10:35:15.638937 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v88: 768 pgs: 768 stale+active+clean; 1646 bytes data, 20353 MB used, 77580 MB / 100 GB avail 2015-02-25 10:35:16.571461 7fe2b2b00700 1 mon.c@0(leader).osd e38 e38: 3 osds: 2 up, 3 in 2015-02-25 10:35:16.580485 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e38: 3 osds: 2 up, 3 in 2015-02-25 10:35:16.591773 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v89: 768 pgs: 768 stale+active+clean; 1646 bytes data, 20353 MB used, 77580 MB / 100 GB avail 2015-02-25 10:35:18.241830 7fe2b075c700 0 mon.c@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "args": ["host=set003", "root=default"], "id": 1, "weight": 0.040000000000000001} v 0) v1 2015-02-25 10:35:18.241899 7fe2b075c700 0 log_channel(audit) log [INF] : from='client.? 172.20.32.17:0/1028711' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "args": ["host=set003", "root=default"], "id": 1, "weight": 0.040000000000000001}]: dispatch 2015-02-25 10:35:18.242024 7fe2b075c700 0 mon.c@0(leader).osd e38 create-or-move crush item name 'osd.1' initial_weight 0.04 at location {host=set003,root=default} 2015-02-25 10:35:22.176466 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v90: 768 pgs: 131 active+undersized+degraded, 637 peering; 1646 bytes data, 20349 MB used, 77584 MB / 100 GB avail; 3/141 objects degraded (2.128%) 2015-02-25 10:35:23.149050 7fe2b2b00700 1 mon.c@0(leader).osd e39 e39: 3 osds: 3 up, 3 in 2015-02-25 10:35:23.153168 7fe2b2b00700 0 log_channel(cluster) log [INF] : osd.1 172.20.32.17:6801/28756 boot 2015-02-25 10:35:23.157456 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e39: 3 osds: 3 up, 3 in 2015-02-25 10:35:23.179186 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v91: 768 pgs: 131 active+undersized+degraded, 637 peering; 1646 bytes data, 20349 MB used, 77584 MB / 100 GB avail; 3/141 objects degraded (2.128%) 2015-02-25 10:35:24.155374 7fe2b2b00700 1 mon.c@0(leader).osd e40 e40: 3 osds: 3 up, 3 in 2015-02-25 10:35:24.173615 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e40: 3 osds: 3 up, 3 in 2015-02-25 10:35:24.202470 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v92: 768 pgs: 131 active+undersized+degraded, 637 peering; 1646 bytes data, 20349 MB used, 77584 MB / 100 GB avail; 3/141 objects degraded (2.128%) 2015-02-25 10:35:28.392479 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v93: 768 pgs: 131 active+undersized+degraded, 27 active, 590 peering, 20 active+clean; 1646 bytes data, 20353 MB used, 77580 MB / 100 GB avail; 3/141 objects degraded (2.128%) 2015-02-25 10:35:30.871271 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v94: 768 pgs: 25 inactive, 235 active, 324 peering, 184 active+clean; 1646 bytes data, 20348 MB used, 77585 MB / 100 GB avail; 3341 B/s rd, 3 op/s 2015-02-25 10:35:35.053437 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v95: 768 pgs: 25 inactive, 208 active, 118 peering, 417 active+clean; 1646 bytes data, 20350 MB used, 77583 MB / 100 GB avail; 12696 B/s rd, 0 B/s wr, 18 op/s 2015-02-25 10:35:36.113821 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v96: 768 pgs: 768 active+clean; 1646 bytes data, 20353 MB used, 77580 MB / 100 GB avail; 14960 B/s rd, 0 B/s wr, 24 op/s 2015-02-25 10:35:40.171882 7fe2b075c700 0 cephx server client.radosgw.set002: couldn't find entity name: client.radosgw.set002 2015-02-25 10:35:48.507336 7fe2b075c700 0 mon.c@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1 2015-02-25 10:35:48.507406 7fe2b075c700 0 log_channel(audit) log [DBG] : from='client.? 172.20.32.17:0/1028905' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2015-02-25 10:36:09.541853 7fe2b115d700 0 mon.c@0(leader).data_health(1) update_stats avail 75% total 46198 MB, used 8960 MB, avail 34891 MB 2015-02-25 10:36:33.423611 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v97: 768 pgs: 768 active+clean; 1646 bytes data, 20354 MB used, 77579 MB / 100 GB avail; 214 B/s rd, 0 B/s wr, 0 op/s 2015-02-25 10:36:35.746863 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v98: 768 pgs: 768 active+clean; 1646 bytes data, 20355 MB used, 77578 MB / 100 GB avail; 561 B/s rd, 0 B/s wr, 0 op/s 2015-02-25 10:36:39.324440 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v99: 768 pgs: 768 active+clean; 1646 bytes data, 20355 MB used, 77578 MB / 100 GB avail; 9125 B/s rd, 0 B/s wr, 14 op/s 2015-02-25 10:36:48.004240 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v100: 768 pgs: 768 active+clean; 1646 bytes data, 20355 MB used, 77578 MB / 100 GB avail; 6866 B/s rd, 0 B/s wr, 11 op/s 2015-02-25 10:36:50.222363 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v101: 768 pgs: 768 active+clean; 1646 bytes data, 20355 MB used, 77578 MB / 100 GB avail; 3277 B/s rd, 0 B/s wr, 5 op/s 2015-02-25 10:37:09.542142 7fe2b115d700 0 mon.c@0(leader).data_health(1) update_stats avail 75% total 46198 MB, used 8960 MB, avail 34891 MB 2015-02-25 10:37:33.402914 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v102: 768 pgs: 768 active+clean; 1646 bytes data, 20355 MB used, 77578 MB / 100 GB avail; 830 B/s rd, 0 B/s wr, 1 op/s 2015-02-25 10:37:34.457358 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v103: 768 pgs: 768 active+clean; 1646 bytes data, 20355 MB used, 77578 MB / 100 GB avail; 1962 B/s rd, 0 B/s wr, 3 op/s 2015-02-25 10:37:38.400046 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v104: 768 pgs: 768 active+clean; 1646 bytes data, 20356 MB used, 77578 MB / 100 GB avail; 12285 B/s rd, 0 B/s wr, 19 op/s 2015-02-25 10:38:09.542398 7fe2b115d700 0 mon.c@0(leader).data_health(1) update_stats avail 75% total 46198 MB, used 8961 MB, avail 34891 MB 2015-02-25 10:38:34.624958 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v105: 768 pgs: 768 active+clean; 1646 bytes data, 20356 MB used, 77578 MB / 100 GB avail; 111 B/s rd, 0 B/s wr, 0 op/s 2015-02-25 10:38:35.641598 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v106: 768 pgs: 768 active+clean; 1646 bytes data, 20356 MB used, 77577 MB / 100 GB avail; 304 B/s rd, 0 B/s wr, 0 op/s 2015-02-25 10:38:40.492757 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v107: 768 pgs: 768 active+clean; 1646 bytes data, 20356 MB used, 77577 MB / 100 GB avail; 4568 B/s rd, 0 B/s wr, 7 op/s 2015-02-25 10:38:43.400182 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v108: 768 pgs: 768 active+clean; 1646 bytes data, 20356 MB used, 77577 MB / 100 GB avail; 6593 B/s rd, 0 B/s wr, 10 op/s 2015-02-25 10:38:44.413500 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v109: 768 pgs: 768 active+clean; 1646 bytes data, 20356 MB used, 77577 MB / 100 GB avail; 12307 B/s rd, 0 B/s wr, 19 op/s 2015-02-25 10:39:09.542690 7fe2b115d700 0 mon.c@0(leader).data_health(1) update_stats avail 75% total 46198 MB, used 8961 MB, avail 34890 MB 2015-02-25 10:39:33.412191 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v110: 768 pgs: 768 active+clean; 1646 bytes data, 20356 MB used, 77577 MB / 100 GB avail; 188 B/s rd, 0 B/s wr, 0 op/s 2015-02-25 10:39:34.426050 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v111: 768 pgs: 768 active+clean; 1646 bytes data, 20356 MB used, 77577 MB / 100 GB avail; 614 B/s rd, 0 B/s wr, 0 op/s 2015-02-25 10:39:39.710315 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v112: 768 pgs: 768 active+clean; 1646 bytes data, 20357 MB used, 77577 MB / 100 GB avail; 12080 B/s rd, 0 B/s wr, 19 op/s 2015-02-25 10:39:40.720411 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v113: 768 pgs: 768 active+clean; 1646 bytes data, 20357 MB used, 77576 MB / 100 GB avail; 11874 B/s rd, 0 B/s wr, 18 op/s 2015-02-25 10:39:47.813821 7fe2b2b00700 1 mon.c@0(leader).osd e41 e41: 3 osds: 3 up, 3 in 2015-02-25 10:39:47.823464 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e41: 3 osds: 3 up, 3 in 2015-02-25 10:39:47.832381 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v114: 896 pgs: 128 creating, 768 active+clean; 1646 bytes data, 20357 MB used, 77576 MB / 100 GB avail; 3806 B/s rd, 0 B/s wr, 5 op/s 2015-02-25 10:39:48.913135 7fe2b2b00700 1 mon.c@0(leader).osd e42 e42: 3 osds: 3 up, 3 in 2015-02-25 10:39:48.956402 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e42: 3 osds: 3 up, 3 in 2015-02-25 10:39:48.961575 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v115: 896 pgs: 128 creating, 768 active+clean; 1646 bytes data, 20357 MB used, 77576 MB / 100 GB avail 2015-02-25 10:39:53.307700 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v116: 896 pgs: 90 creating, 38 peering, 768 active+clean; 1646 bytes data, 20357 MB used, 77576 MB / 100 GB avail 2015-02-25 10:39:54.345898 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v117: 896 pgs: 18 inactive, 75 peering, 803 active+clean; 1981 bytes data, 20352 MB used, 77582 MB / 100 GB avail; 190 B/s rd, 381 B/s wr, 2 op/s 2015-02-25 10:39:58.248454 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v118: 896 pgs: 18 inactive, 37 peering, 841 active+clean; 1981 bytes data, 20353 MB used, 77580 MB / 100 GB avail; 204 B/s rd, 409 B/s wr, 2 op/s 2015-02-25 10:39:59.263513 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v119: 896 pgs: 896 active+clean; 1981 bytes data, 20349 MB used, 77584 MB / 100 GB avail 2015-02-25 10:40:01.364182 7fe2b2b00700 1 mon.c@0(leader).osd e43 e43: 3 osds: 3 up, 3 in 2015-02-25 10:40:01.375122 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e43: 3 osds: 3 up, 3 in 2015-02-25 10:40:01.403009 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v120: 1024 pgs: 128 creating, 896 active+clean; 1981 bytes data, 20349 MB used, 77584 MB / 100 GB avail 2015-02-25 10:40:06.687689 7fe2b2b00700 1 mon.c@0(leader).osd e44 e44: 3 osds: 3 up, 3 in 2015-02-25 10:40:06.711516 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e44: 3 osds: 3 up, 3 in 2015-02-25 10:40:06.769467 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v121: 1024 pgs: 84 creating, 44 peering, 896 active+clean; 1981 bytes data, 20350 MB used, 77584 MB / 100 GB avail 2015-02-25 10:40:07.812833 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v122: 1024 pgs: 128 peering, 896 active+clean; 1981 bytes data, 20350 MB used, 77583 MB / 100 GB avail 2015-02-25 10:40:09.542917 7fe2b115d700 0 mon.c@0(leader).data_health(1) update_stats avail 75% total 46198 MB, used 8961 MB, avail 34890 MB 2015-02-25 10:40:12.390159 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v123: 1024 pgs: 84 peering, 940 active+clean; 1981 bytes data, 20351 MB used, 77582 MB / 100 GB avail 2015-02-25 10:40:13.413038 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v124: 1024 pgs: 1024 active+clean; 1981 bytes data, 20355 MB used, 77578 MB / 100 GB avail; 363 B/s rd, 0 B/s wr, 1 op/s 2015-02-25 10:40:38.232938 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v125: 1024 pgs: 1024 active+clean; 1981 bytes data, 20355 MB used, 77578 MB / 100 GB avail; 85 B/s rd, 0 B/s wr, 0 op/s 2015-02-25 10:40:43.256812 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v126: 1024 pgs: 1024 active+clean; 1981 bytes data, 20356 MB used, 77577 MB / 100 GB avail; 1234 B/s rd, 0 B/s wr, 2 op/s 2015-02-25 10:40:47.380160 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v127: 1024 pgs: 1024 active+clean; 1981 bytes data, 20356 MB used, 77577 MB / 100 GB avail; 5630 B/s rd, 0 B/s wr, 9 op/s 2015-02-25 10:40:48.393731 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v128: 1024 pgs: 1024 active+clean; 1981 bytes data, 20356 MB used, 77577 MB / 100 GB avail; 11977 B/s rd, 0 B/s wr, 19 op/s 2015-02-25 10:40:50.176105 7fe2b075c700 0 cephx server client.radosgw.set002: couldn't find entity name: client.radosgw.set002 2015-02-25 10:41:05.131093 7fe2b2b00700 1 mon.c@0(leader).osd e45 e45: 3 osds: 3 up, 3 in 2015-02-25 10:41:05.158078 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e45: 3 osds: 3 up, 3 in 2015-02-25 10:41:05.188157 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v130: 1152 pgs: 128 creating, 1024 active+clean; 1981 bytes data, 20356 MB used, 77577 MB / 100 GB avail; 244 B/s rd, 0 op/s 2015-02-25 10:41:06.183348 7fe2b2b00700 1 mon.c@0(leader).osd e46 e46: 3 osds: 3 up, 3 in 2015-02-25 10:41:06.255255 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e46: 3 osds: 3 up, 3 in 2015-02-25 10:41:06.260584 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v131: 1152 pgs: 128 creating, 1024 active+clean; 1981 bytes data, 20356 MB used, 77577 MB / 100 GB avail 2015-02-25 10:41:09.543112 7fe2b115d700 0 mon.c@0(leader).data_health(1) update_stats avail 75% total 46198 MB, used 8963 MB, avail 34888 MB 2015-02-25 10:41:19.298115 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v132: 1152 pgs: 85 creating, 43 peering, 1024 active+clean; 1981 bytes data, 20357 MB used, 77576 MB / 100 GB avail; 160 B/s rd, 0 B/s wr, 0 op/s 2015-02-25 10:41:20.925590 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v133: 1152 pgs: 48 creating, 80 peering, 1024 active+clean; 1981 bytes data, 20358 MB used, 77575 MB / 100 GB avail; 140 B/s rd, 0 B/s wr, 0 op/s 2015-02-25 10:41:22.037509 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v134: 1152 pgs: 128 peering, 1024 active+clean; 1981 bytes data, 20359 MB used, 77574 MB / 100 GB avail 2015-02-25 10:41:25.151474 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v135: 1152 pgs: 90 peering, 1062 active+clean; 1981 bytes data, 20359 MB used, 77574 MB / 100 GB avail 2015-02-25 10:41:26.170601 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v136: 1152 pgs: 53 peering, 1099 active+clean; 1981 bytes data, 20361 MB used, 77572 MB / 100 GB avail 2015-02-25 10:41:26.173378 7fe2b2b00700 1 mon.c@0(leader).osd e47 e47: 3 osds: 3 up, 3 in 2015-02-25 10:41:26.185216 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e47: 3 osds: 3 up, 3 in 2015-02-25 10:41:26.259225 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v137: 1280 pgs: 128 creating, 53 peering, 1099 active+clean; 1981 bytes data, 20361 MB used, 77572 MB / 100 GB avail 2015-02-25 10:41:27.285614 7fe2b2b00700 1 mon.c@0(leader).osd e48 e48: 3 osds: 3 up, 3 in 2015-02-25 10:41:27.308612 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e48: 3 osds: 3 up, 3 in 2015-02-25 10:41:27.316532 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v138: 1280 pgs: 128 creating, 53 peering, 1099 active+clean; 1981 bytes data, 20361 MB used, 77572 MB / 100 GB avail 2015-02-25 10:41:31.924381 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v139: 1280 pgs: 83 creating, 48 peering, 1149 active+clean; 4097 kB data, 20362 MB used, 77571 MB / 100 GB avail; 359 B/s rd, 718 kB/s wr, 2 op/s 2015-02-25 10:41:33.871652 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v140: 1280 pgs: 1280 active+clean; 9217 kB data, 20358 MB used, 77576 MB / 100 GB avail; 328 B/s rd, 1476 kB/s wr, 4 op/s 2015-02-25 10:41:37.373058 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v141: 1280 pgs: 1280 active+clean; 15361 kB data, 20391 MB used, 77542 MB / 100 GB avail; 938 B/s rd, 2063 kB/s wr, 6 op/s 2015-02-25 10:41:41.870964 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v142: 1280 pgs: 1280 active+clean; 15361 kB data, 20391 MB used, 77542 MB / 100 GB avail; 1968 B/s rd, 738 kB/s wr, 5 op/s 2015-02-25 10:41:42.881620 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v143: 1280 pgs: 1280 active+clean; 24065 kB data, 20426 MB used, 77507 MB / 100 GB avail; 7029 B/s rd, 1572 kB/s wr, 15 op/s 2015-02-25 10:41:56.910514 7fe2b075c700 1 mon.c@0(leader).osd e48 prepare_failure osd.1 172.20.32.17:6801/28756 from osd.2 172.20.32.100:6804/30245 is reporting failure:1 2015-02-25 10:41:56.910530 7fe2b075c700 0 log_channel(cluster) log [DBG] : osd.1 172.20.32.17:6801/28756 reported failed by osd.2 172.20.32.100:6804/30245 2015-02-25 10:42:01.816517 7fe2b075c700 1 mon.c@0(leader).osd e48 prepare_failure osd.1 172.20.32.17:6801/28756 from osd.0 172.20.32.5:6801/29835 is reporting failure:1 2015-02-25 10:42:01.816558 7fe2b075c700 0 log_channel(cluster) log [DBG] : osd.1 172.20.32.17:6801/28756 reported failed by osd.0 172.20.32.5:6801/29835 2015-02-25 10:42:01.911758 7fe2b075c700 1 mon.c@0(leader).osd e48 prepare_failure osd.1 172.20.32.17:6801/28756 from osd.2 172.20.32.100:6804/30245 is reporting failure:1 2015-02-25 10:42:01.911775 7fe2b075c700 0 log_channel(cluster) log [DBG] : osd.1 172.20.32.17:6801/28756 reported failed by osd.2 172.20.32.100:6804/30245 2015-02-25 10:42:01.911820 7fe2b075c700 1 mon.c@0(leader).osd e48 we have enough reports/reporters to mark osd.1 down 2015-02-25 10:42:01.911828 7fe2b075c700 0 log_channel(cluster) log [INF] : osd.1 172.20.32.17:6801/28756 failed (3 reports from 2 peers after 24.095403 >= grace 20.000000) 2015-02-25 10:42:02.035893 7fe2b075c700 1 mon.c@0(leader).osd e48 prepare_failure osd.0 172.20.32.5:6801/29835 from osd.1 172.20.32.17:6801/28756 is reporting failure:1 2015-02-25 10:42:02.035907 7fe2b075c700 0 log_channel(cluster) log [DBG] : osd.0 172.20.32.5:6801/29835 reported failed by osd.1 172.20.32.17:6801/28756 2015-02-25 10:42:02.035953 7fe2b075c700 1 mon.c@0(leader).osd e48 prepare_failure osd.2 172.20.32.100:6804/30245 from osd.1 172.20.32.17:6801/28756 is reporting failure:1 2015-02-25 10:42:02.035961 7fe2b075c700 0 log_channel(cluster) log [DBG] : osd.2 172.20.32.100:6804/30245 reported failed by osd.1 172.20.32.17:6801/28756 2015-02-25 10:42:04.746055 7fe2b2b00700 1 mon.c@0(leader).osd e49 e49: 3 osds: 2 up, 3 in 2015-02-25 10:42:04.868503 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e49: 3 osds: 2 up, 3 in 2015-02-25 10:42:04.926672 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v144: 1280 pgs: 404 stale+active+clean, 876 active+clean; 24065 kB data, 20426 MB used, 77507 MB / 100 GB avail 2015-02-25 10:42:06.109963 7fe2b2b00700 1 mon.c@0(leader).osd e50 e50: 3 osds: 2 up, 3 in 2015-02-25 10:42:06.118962 7fe2b2b00700 0 log_channel(cluster) log [INF] : osdmap e50: 3 osds: 2 up, 3 in 2015-02-25 10:42:09.041905 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v145: 1280 pgs: 404 stale+active+clean, 876 active+clean; 24065 kB data, 20426 MB used, 77507 MB / 100 GB avail 2015-02-25 10:42:09.543365 7fe2b115d700 0 mon.c@0(leader).data_health(1) update_stats avail 75% total 46198 MB, used 9000 MB, avail 34851 MB 2015-02-25 10:42:10.467373 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v146: 1280 pgs: 270 stale+active+clean, 410 peering, 600 active+clean; 24065 kB data, 20443 MB used, 77490 MB / 100 GB avail 2015-02-25 10:42:11.660828 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v147: 1280 pgs: 180 stale+active+clean, 673 peering, 427 active+clean; 24065 kB data, 20449 MB used, 77484 MB / 100 GB avail; 0 B/s wr, 0 op/s 2015-02-25 10:42:19.133401 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v148: 1280 pgs: 28 active+undersized+degraded, 104 stale+active+clean, 898 peering, 250 active+clean; 24065 kB data, 20451 MB used, 77483 MB / 100 GB avail; 0 B/s wr, 0 op/s; 1/180 objects degraded (0.556%) 2015-02-25 10:42:20.285198 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v149: 1280 pgs: 28 active+undersized+degraded, 1252 peering; 30721 kB data, 20451 MB used, 77483 MB / 100 GB avail; 773 kB/s wr, 1 op/s; 1/183 objects degraded (0.546%) 2015-02-25 10:42:21.425038 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v150: 1280 pgs: 1280 active+undersized+degraded; 32257 kB data, 20455 MB used, 77479 MB / 100 GB avail; 1693 B/s rd, 2032 kB/s wr, 8 op/s; 63/186 objects degraded (33.871%) 2015-02-25 10:42:27.071043 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v151: 1280 pgs: 1280 active+undersized+degraded; 36353 kB data, 20452 MB used, 77481 MB / 100 GB avail; 9286 B/s rd, 1917 kB/s wr, 21 op/s; 67/192 objects degraded (34.896%) 2015-02-25 10:42:29.195488 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v152: 1280 pgs: 1280 active+undersized+degraded; 53761 kB data, 20472 MB used, 77462 MB / 100 GB avail; 7393 B/s rd, 2723 kB/s wr, 19 op/s; 75/204 objects degraded (36.765%) 2015-02-25 10:42:31.993510 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v153: 1280 pgs: 1280 active+undersized+degraded; 60929 kB data, 20506 MB used, 77427 MB / 100 GB avail; 6715 B/s rd, 4883 kB/s wr, 23 op/s; 79/210 objects degraded (37.619%) 2015-02-25 10:42:33.004995 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v154: 1280 pgs: 1280 active+undersized+degraded; 76289 kB data, 20525 MB used, 77409 MB / 100 GB avail; 2622 B/s rd, 5769 kB/s wr, 19 op/s; 87/222 objects degraded (39.189%) 2015-02-25 10:42:35.211384 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v155: 1280 pgs: 1280 active+undersized+degraded; 76289 kB data, 20541 MB used, 77393 MB / 100 GB avail; 10779 B/s rd, 3050 kB/s wr, 26 op/s; 85/219 objects degraded (38.813%) 2015-02-25 10:42:36.226376 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v156: 1280 pgs: 1280 active+undersized+degraded; 79079 kB data, 20548 MB used, 77386 MB / 100 GB avail; 34644 B/s rd, 865 kB/s wr, 63 op/s; 89/225 objects degraded (39.556%) 2015-02-25 10:42:40.146339 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v157: 1280 pgs: 1280 active+undersized+degraded; 79079 kB data, 20548 MB used, 77386 MB / 100 GB avail; 13899 B/s rd, 565 kB/s wr, 26 op/s; 89/225 objects degraded (39.556%) 2015-02-25 10:42:55.620024 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v158: 1280 pgs: 1280 active+undersized+degraded; 79079 kB data, 20548 MB used, 77385 MB / 100 GB avail; 527 B/s rd, 0 B/s wr, 0 op/s; 89/225 objects degraded (39.556%) 2015-02-25 10:43:05.151516 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v159: 1280 pgs: 1280 active+undersized+degraded; 79079 kB data, 20548 MB used, 77385 MB / 100 GB avail; 245 B/s rd, 0 B/s wr, 0 op/s; 89/225 objects degraded (39.556%) 2015-02-25 10:43:07.970246 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v160: 1280 pgs: 1280 active+undersized+degraded; 79079 kB data, 20548 MB used, 77385 MB / 100 GB avail; 663 B/s rd, 0 B/s wr, 1 op/s; 87/222 objects degraded (39.189%) 2015-02-25 10:43:09.543655 7fe2b115d700 0 mon.c@0(leader).data_health(1) update_stats avail 75% total 46198 MB, used 9048 MB, avail 34803 MB 2015-02-25 10:43:25.625030 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v161: 1280 pgs: 1280 active+undersized+degraded; 79079 kB data, 20548 MB used, 77385 MB / 100 GB avail; 200 B/s rd, 0 B/s wr, 0 op/s; 87/222 objects degraded (39.189%) 2015-02-25 10:43:35.630559 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v162: 1280 pgs: 1280 active+undersized+degraded; 79079 kB data, 20548 MB used, 77385 MB / 100 GB avail; 1036 B/s rd, 0 B/s wr, 1 op/s; 87/222 objects degraded (39.189%) 2015-02-25 10:43:40.190576 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v163: 1280 pgs: 1280 active+undersized+degraded; 63719 kB data, 20476 MB used, 77457 MB / 100 GB avail; 4932 B/s rd, 0 B/s wr, 8 op/s; 79/210 objects degraded (37.619%) 2015-02-25 10:43:41.204528 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v164: 1280 pgs: 1280 active+undersized+degraded; 11265 kB data, 20410 MB used, 77523 MB / 100 GB avail; 13416 B/s rd, 0 B/s wr, 24 op/s; 51/168 objects degraded (30.357%) 2015-02-25 10:43:42.110225 7fe2b075c700 1 mon.c@0(leader).pg v164 ignoring stats from non-active osd. 2015-02-25 10:44:09.543935 7fe2b115d700 0 mon.c@0(leader).data_health(1) update_stats avail 75% total 46198 MB, used 8982 MB, avail 34869 MB 2015-02-25 10:44:12.118838 7fe2b075c700 1 mon.c@0(leader).pg v164 ignoring stats from non-active osd. 2015-02-25 10:44:35.165474 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v165: 1280 pgs: 1280 active+undersized+degraded; 11265 kB data, 20410 MB used, 77523 MB / 100 GB avail; 37 B/s rd, 0 op/s; 51/168 objects degraded (30.357%) 2015-02-25 10:44:36.702653 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v166: 1280 pgs: 1280 active+undersized+degraded; 11265 kB data, 20410 MB used, 77523 MB / 100 GB avail; 74 B/s rd, 0 B/s wr, 0 op/s; 51/168 objects degraded (30.357%) 2015-02-25 10:44:40.166335 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v167: 1280 pgs: 1280 active+undersized+degraded; 11265 kB data, 20410 MB used, 77523 MB / 100 GB avail; 8394 B/s rd, 0 B/s wr, 13 op/s; 51/168 objects degraded (30.357%) 2015-02-25 10:44:41.180163 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v168: 1280 pgs: 1280 active+undersized+degraded; 11265 kB data, 20410 MB used, 77523 MB / 100 GB avail; 19645 B/s rd, 0 B/s wr, 31 op/s; 51/168 objects degraded (30.357%) 2015-02-25 10:44:42.123792 7fe2b075c700 1 mon.c@0(leader).pg v168 ignoring stats from non-active osd. 2015-02-25 10:45:09.544175 7fe2b115d700 0 mon.c@0(leader).data_health(1) update_stats avail 75% total 46198 MB, used 8982 MB, avail 34869 MB 2015-02-25 10:45:12.128954 7fe2b075c700 1 mon.c@0(leader).pg v168 ignoring stats from non-active osd. 2015-02-25 10:45:40.176366 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v169: 1280 pgs: 1280 active+undersized+degraded; 11265 kB data, 20411 MB used, 77523 MB / 100 GB avail; 676 B/s rd, 0 B/s wr, 1 op/s; 51/168 objects degraded (30.357%) 2015-02-25 10:45:41.186778 7fe2b2b00700 0 log_channel(cluster) log [INF] : pgmap v170: 1280 pgs: 1280 active+undersized+degraded; 11265 kB data, 20411 MB used, 77523 MB / 100 GB avail; 1638 B/s rd, 0 B/s wr, 2 op/s; 51/168 objects degraded (30.357%) 2015-02-25 10:45:42.132806 7fe2b075c700 1 mon.c@0(leader).pg v170 ignoring stats from non-active osd. 2015-02-25 10:46:00.180957 7fe2b075c700 0 cephx server client.radosgw.set002: couldn't find entity name: client.radosgw.set002 2015-02-25 10:46:09.544443 7fe2b115d700 0 mon.c@0(leader).data_health(1) update_stats avail 75% total 46198 MB, used 8983 MB, avail 34869 MB 2015-02-25 10:46:12.137383 7fe2b075c700 1 mon.c@0(leader).pg v170 ignoring stats from non-active osd.