admin:~ # ceph -w cluster 557c4525-cad3-371a-8e05-4b95f6accdf5 health HEALTH_OK monmap e2: 3 mons at {data1=172.16.1.21:6789/0,data2=172.16.1.22:6789/0,mon1=172.16.1.11:6789/0} election epoch 6, quorum 0,1,2 mon1,data1,data2 mgr active: mon1 standbys: data1, data2 osdmap e40: 8 osds: 8 up, 8 in flags sortbitwise,require_jewel_osds,require_kraken_osds,require_luminous_osds pgmap v96: 248 pgs, 1 pools, 0 bytes data, 0 objects 282 MB used, 10407 MB / 10690 MB avail 248 active+clean 2017-03-02 16:00:00.000236 mon.0 [INF] HEALTH_OK 2017-03-02 16:00:16.868943 mon.0 [INF] pgmap v97: 248 pgs: 248 active+clean; 32768 kB data, 282 MB use d, 10407 MB / 10690 MB avail; 1243 kB/s wr, 0 op/s 2017-03-02 16:00:25.485054 mon.0 [INF] pgmap v98: 248 pgs: 248 active+clean; 196 MB data, 1838 MB used , 8852 MB / 10690 MB avail; 5992 kB/s wr, 2 op/s 2017-03-02 16:00:34.127780 mon.0 [INF] pgmap v99: 248 pgs: 248 active+clean; 384 MB data, 3349 MB used , 7340 MB / 10690 MB avail; 9448 kB/s wr, 4 op/s 2017-03-02 16:00:45.216840 mon.0 [INF] pgmap v100: 248 pgs: 248 active+clean; 664 MB data, 4801 MB use d, 5888 MB / 10690 MB avail; 13263 kB/s wr, 6 op/s 2017-03-02 16:00:42.398014 osd.1 [WRN] OSD near full (94%) 2017-03-02 16:00:54.949233 mon.0 [INF] pgmap v101: 248 pgs: 248 active+clean; 912 MB data, 6045 MB use d, 4645 MB / 10690 MB avail; 15103 kB/s wr, 7 op/s 2017-03-02 16:00:51.798991 osd.1 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:00:52.199787 osd.0 [WRN] OSD near full (91%) 2017-03-02 16:00:57.699552 osd.1 [WRN] OSD near full (97%) 2017-03-02 16:01:00.309164 mon.0 [INF] pgmap v102: 248 pgs: 248 active+clean; 1272 MB data, 7854 MB us ed, 2836 MB / 10690 MB avail; 18507 kB/s wr, 9 op/s 2017-03-02 16:00:57.535252 osd.6 [WRN] OSD near full (96%) 2017-03-02 16:00:59.801120 osd.0 [ERR] OSD full dropping all updates 97% full 2017-03-02 16:01:00.201562 osd.3 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:01:00.702151 osd.3 [WRN] OSD near full (93%) 2017-03-02 16:01:01.800024 osd.1 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:01:02.481969 osd.5 [WRN] OSD near full (93%) 2017-03-02 16:01:03.901671 osd.0 [WRN] OSD near full (92%) 2017-03-02 16:01:04.202755 osd.3 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:01:05.081565 osd.7 [WRN] OSD near full (91%) 2017-03-02 16:01:06.644936 mon.0 [INF] HEALTH_WARN; Difference in osd space utilization 42.3649% great er than 40%; 1 near full osd(s) 2017-03-02 16:01:07.683188 osd.5 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:01:09.455542 mon.0 [INF] pgmap v103: 248 pgs: 248 active+clean; 1440 MB data, 8403 MB us ed, 2286 MB / 10690 MB avail; 28918 kB/s wr, 14 op/s 2017-03-02 16:01:09.455717 mon.0 [INF] osdmap e41: 8 osds: 8 up, 8 in nearfull 2017-03-02 16:01:09.682783 osd.7 [WRN] OSD near full (90%) 2017-03-02 16:01:10.291758 mon.0 [INF] pgmap v104: 248 pgs: 248 active+clean; 1668 MB data, 9007 MB us ed, 1682 MB / 10690 MB avail; 30373 kB/s wr, 14 op/s 2017-03-02 16:01:11.379502 mon.0 [INF] pgmap v105: 248 pgs: 248 active+clean; 1816 MB data, 9447 MB us ed, 1242 MB / 10690 MB avail; 33249 kB/s wr, 16 op/s 2017-03-02 16:01:13.279581 mon.0 [INF] osdmap e42: 8 osds: 8 up, 8 in full 2017-03-02 16:01:13.543353 mon.0 [INF] pgmap v106: 248 pgs: 248 active+clean; 1852 MB data, 9508 MB us ed, 1182 MB / 10690 MB avail; 33421 kB/s wr, 16 op/s 2017-03-02 16:01:15.442087 mon.0 [INF] pgmap v107: 248 pgs: 248 active+clean; 1856 MB data, 9496 MB us ed, 1193 MB / 10690 MB avail; 34650 kB/s wr, 16 op/s 2017-03-02 16:01:08.602416 osd.0 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:01:16.514533 mon.0 [INF] pgmap v108: 248 pgs: 248 active+clean; 1856 MB data, 9380 MB us ed, 1310 MB / 10690 MB avail; 29180 kB/s wr, 14 op/s 2017-03-02 16:01:15.304448 osd.3 [WRN] OSD near full (96%) 2017-03-02 16:01:17.584997 osd.5 [WRN] OSD near full (95%) 2017-03-02 16:01:18.771202 mon.0 [INF] pgmap v109: 248 pgs: 248 active+clean; 1856 MB data, 9229 MB us ed, 1461 MB / 10690 MB avail; 25194 kB/s wr, 12 op/s 2017-03-02 16:01:19.842066 mon.0 [INF] pgmap v110: 248 pgs: 248 active+clean; 1856 MB data, 9149 MB us ed, 1541 MB / 10690 MB avail; 17371 kB/s wr, 8 op/s 2017-03-02 16:01:20.922190 mon.0 [INF] pgmap v111: 248 pgs: 248 active+clean; 1856 MB data, 8939 MB us ed, 1751 MB / 10690 MB avail; 4287 kB/s wr, 2 op/s 2017-03-02 16:01:23.015742 mon.0 [INF] pgmap v112: 248 pgs: 248 active+clean; 1856 MB data, 8938 MB us ed, 1751 MB / 10690 MB avail; 422 kB/s wr, 0 op/s 2017-03-02 16:01:25.167305 mon.0 [INF] pgmap v113: 248 pgs: 248 active+clean; 1856 MB data, 8938 MB us ed, 1751 MB / 10690 MB avail 2017-03-02 16:01:17.908801 osd.0 [WRN] OSD near full (95%) 2017-03-02 16:01:32.606021 osd.1 [ERR] OSD full dropping all updates 97% full 2017-03-02 16:01:47.711172 osd.3 [WRN] OSD near full (96%) 2017-03-02 16:01:48.489545 osd.5 [WRN] OSD near full (95%) 2017-03-02 16:01:51.913736 osd.0 [WRN] OSD near full (95%) 2017-03-02 16:02:06.645301 mon.0 [INF] HEALTH_ERR; 4 full osd(s); 1 near full osd(s); full flag(s) set 2017-03-02 16:02:03.313213 osd.1 [ERR] OSD full dropping all updates 97% full 2017-03-02 16:02:18.318061 osd.3 [WRN] OSD near full (96%) 2017-03-02 16:02:18.393507 osd.5 [WRN] OSD near full (95%) 2017-03-02 16:02:24.618909 osd.0 [WRN] OSD near full (95%) 2017-03-02 16:02:34.718579 osd.1 [ERR] OSD full dropping all updates 97% full 2017-03-02 16:02:51.698604 osd.5 [WRN] OSD near full (95%) 2017-03-02 16:02:53.621977 osd.3 [WRN] OSD near full (96%) 2017-03-02 16:02:57.025961 osd.0 [WRN] OSD near full (95%) 2017-03-02 16:03:08.023640 osd.1 [ERR] OSD full dropping all updates 97% full 2017-03-02 16:03:21.603128 osd.5 [WRN] OSD near full (95%) 2017-03-02 16:03:24.731983 osd.3 [WRN] OSD near full (96%) 2017-03-02 16:03:35.707623 mon.0 [INF] pgmap v114: 248 pgs: 248 active+clean; 1856 MB data, 8468 MB us ed, 2221 MB / 10690 MB avail 2017-03-02 16:03:31.733471 osd.0 [WRN] OSD near full (95%) 2017-03-02 16:03:38.285830 mon.0 [INF] osdmap e43: 8 osds: 8 up, 8 in full 2017-03-02 16:03:38.361267 mon.0 [INF] pgmap v115: 248 pgs: 248 active+clean; 1856 MB data, 8468 MB us ed, 2221 MB / 10690 MB avail 2017-03-02 16:03:40.368272 mon.0 [INF] pgmap v116: 248 pgs: 248 active+clean; 1856 MB data, 8445 MB us ed, 2244 MB / 10690 MB avail 2017-03-02 16:03:41.444201 mon.0 [INF] pgmap v117: 248 pgs: 248 active+clean; 1856 MB data, 8437 MB us ed, 2252 MB / 10690 MB avail 2017-03-02 16:03:43.300288 mon.0 [INF] osdmap e44: 8 osds: 8 up, 8 in full 2017-03-02 16:03:43.379195 mon.0 [INF] pgmap v118: 248 pgs: 248 active+clean; 1856 MB data, 8433 MB us ed, 2256 MB / 10690 MB avail 2017-03-02 16:03:45.375373 mon.0 [INF] pgmap v119: 248 pgs: 248 active+clean; 1856 MB data, 8414 MB us ed, 2276 MB / 10690 MB avail 2017-03-02 16:03:46.438494 mon.0 [INF] pgmap v120: 248 pgs: 248 active+clean; 1856 MB data, 8414 MB us ed, 2276 MB / 10690 MB avail 2017-03-02 16:03:39.727425 osd.1 [ERR] OSD full dropping all updates 97% full 2017-03-02 16:03:45.027991 osd.1 [WRN] OSD near full (97%) 2017-03-02 16:03:48.644909 mon.0 [INF] pgmap v121: 248 pgs: 248 active+clean; 1856 MB data, 8410 MB us ed, 2280 MB / 10690 MB avail 2017-03-02 16:03:50.753073 mon.0 [INF] pgmap v122: 248 pgs: 248 active+clean; 1856 MB data, 8382 MB us ed, 2307 MB / 10690 MB avail 2017-03-02 16:03:53.028311 mon.0 [INF] pgmap v123: 248 pgs: 248 active+clean; 1856 MB data, 7944 MB us ed, 2745 MB / 10690 MB avail 2017-03-02 16:03:55.129215 mon.0 [INF] pgmap v124: 248 pgs: 248 active+clean; 1856 MB data, 7944 MB us ed, 2745 MB / 10690 MB avail 2017-03-02 16:03:51.411028 osd.5 [WRN] OSD near full (94%) 2017-03-02 16:03:54.338277 osd.3 [WRN] OSD near full (96%) 2017-03-02 16:04:06.645956 mon.0 [INF] HEALTH_ERR; Difference in osd space utilization 45.537% greater than 40%; 2 full osd(s); 2 near full osd(s); full flag(s) set 2017-03-02 16:04:01.339401 osd.0 [WRN] OSD near full (94%) 2017-03-02 16:04:15.234013 osd.1 [WRN] OSD near full (97%) 2017-03-02 16:04:22.416117 osd.5 [WRN] OSD near full (94%) 2017-03-02 16:04:25.342770 osd.3 [WRN] OSD near full (96%) 2017-03-02 16:04:33.846091 osd.0 [WRN] OSD near full (94%) 2017-03-02 16:04:47.839188 osd.1 [WRN] OSD near full (97%) 2017-03-02 16:04:52.721065 osd.5 [WRN] OSD near full (94%) 2017-03-02 16:04:55.246440 osd.3 [WRN] OSD near full (96%) 2017-03-02 16:05:04.151031 osd.0 [WRN] OSD near full (94%) 2017-03-02 16:05:19.983481 mon.0 [INF] pgmap v125: 248 pgs: 248 active+clean; 1856 MB data, 7642 MB us ed, 3047 MB / 10690 MB avail 2017-03-02 16:05:21.143362 mon.0 [INF] pgmap v126: 248 pgs: 248 active+clean; 1856 MB data, 7300 MB us ed, 3389 MB / 10690 MB avail 2017-03-02 16:05:25.625063 mon.0 [INF] pgmap v127: 248 pgs: 248 active+clean; 1856 MB data, 6863 MB us ed, 3826 MB / 10690 MB avail 2017-03-02 16:05:24.825684 osd.5 [WRN] OSD near full (94%) 2017-03-02 16:05:28.352873 osd.3 [WRN] OSD near full (96%) 2017-03-02 16:05:30.286750 mon.0 [INF] pgmap v128: 248 pgs: 248 active+clean; 1856 MB data, 6539 MB us ed, 4151 MB / 10690 MB avail 2017-03-02 16:05:33.326578 mon.0 [INF] osdmap e45: 8 osds: 8 up, 8 in nearfull 2017-03-02 16:05:33.767557 mon.0 [INF] pgmap v129: 248 pgs: 248 active+clean; 1856 MB data, 6539 MB us ed, 4151 MB / 10690 MB avail 2017-03-02 16:05:44.907158 mon.0 [INF] pgmap v130: 248 pgs: 248 active+clean; 1944 MB data, 6119 MB us ed, 4571 MB / 10690 MB avail; 895 kB/s wr, 0 op/s 2017-03-02 16:05:39.829279 osd.5 [WRN] OSD near full (96%) 2017-03-02 16:05:40.355809 osd.3 [WRN] OSD near full (96%) 2017-03-02 16:05:41.457745 osd.0 [WRN] OSD near full (95%) 2017-03-02 16:05:45.056642 osd.3 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:05:45.129800 osd.5 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:05:46.758205 osd.0 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:05:40.851872 osd.1 [ERR] OSD full dropping all updates 97% full 2017-03-02 16:05:50.965096 mon.0 [INF] pgmap v131: 248 pgs: 248 active+clean; 2168 MB data, 7952 MB us ed, 2738 MB / 10690 MB avail; 11646 kB/s wr, 5 op/s 2017-03-02 16:05:50.965347 mon.0 [INF] osdmap e46: 8 osds: 8 up, 8 in 2017-03-02 16:05:51.064242 mon.0 [INF] pgmap v132: 248 pgs: 248 active+clean; 2252 MB data, 8746 MB us ed, 1943 MB / 10690 MB avail; 13637 kB/s wr, 6 op/s 2017-03-02 16:05:46.285956 osd.6 [WRN] OSD near full (94%) 2017-03-02 16:05:52.138187 mon.0 [INF] pgmap v133: 248 pgs: 248 active+clean; 2424 MB data, 9444 MB us ed, 1245 MB / 10690 MB avail; 21955 kB/s wr, 10 op/s 2017-03-02 16:05:53.567886 mon.0 [INF] osdmap e47: 8 osds: 8 up, 8 in full 2017-03-02 16:05:53.663061 mon.0 [INF] pgmap v134: 248 pgs: 248 active+clean; 2424 MB data, 9444 MB us ed, 1245 MB / 10690 MB avail; 25169 kB/s wr, 12 op/s 2017-03-02 16:05:56.310635 mon.0 [INF] pgmap v135: 248 pgs: 248 active+clean; 2424 MB data, 9424 MB us ed, 1265 MB / 10690 MB avail; 25361 kB/s wr, 12 op/s 2017-03-02 16:05:57.524510 mon.0 [INF] pgmap v136: 248 pgs: 248 active+clean; 2424 MB data, 9167 MB us ed, 1522 MB / 10690 MB avail; 22673 kB/s wr, 11 op/s 2017-03-02 16:05:54.958142 osd.3 [WRN] OSD near full (97%) 2017-03-02 16:06:01.309856 mon.0 [INF] pgmap v137: 248 pgs: 248 active+clean; 2424 MB data, 9167 MB us ed, 1523 MB / 10690 MB avail; 18882 kB/s wr, 9 op/s 2017-03-02 16:06:02.391238 mon.0 [INF] pgmap v138: 248 pgs: 248 active+clean; 2424 MB data, 9098 MB us ed, 1591 MB / 10690 MB avail; 15223 kB/s wr, 7 op/s 2017-03-02 16:06:06.314569 mon.0 [INF] pgmap v139: 248 pgs: 248 active+clean; 2424 MB data, 9098 MB us ed, 1592 MB / 10690 MB avail 2017-03-02 16:06:06.646525 mon.0 [INF] HEALTH_ERR; 4 full osd(s); 1 near full osd(s); full flag(s) set 2017-03-02 16:06:12.157815 osd.1 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:06:16.034446 osd.5 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:06:20.663582 osd.0 [ERR] OSD full dropping all updates 98% full 2017-03-02 16:06:24.261788 osd.3 [WRN] OSD near full (97%) 2017-03-02 16:06:42.363745 osd.1 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:06:46.339598 osd.5 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:06:53.968147 osd.0 [ERR] OSD full dropping all updates 98% full 2017-03-02 16:06:57.469759 osd.3 [WRN] OSD near full (97%) 2017-03-02 16:07:13.268817 osd.1 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:07:25.978622 osd.0 [ERR] OSD full dropping all updates 98% full 2017-03-02 16:07:19.346106 osd.5 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:07:31.374800 osd.3 [WRN] OSD near full (97%) 2017-03-02 16:07:45.775181 osd.1 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:07:55.384920 osd.0 [ERR] OSD full dropping all updates 98% full 2017-03-02 16:07:50.951336 osd.5 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:08:01.680040 osd.3 [WRN] OSD near full (97%) 2017-03-02 16:08:16.581150 osd.1 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:08:25.490884 osd.0 [ERR] OSD full dropping all updates 98% full 2017-03-02 16:08:21.256363 osd.5 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:08:36.085325 osd.3 [WRN] OSD near full (97%) 2017-03-02 16:08:46.087673 osd.1 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:08:56.599101 osd.0 [ERR] OSD full dropping all updates 98% full 2017-03-02 16:08:53.763419 osd.5 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:09:18.391436 osd.1 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:09:09.791576 osd.3 [WRN] OSD near full (97%) 2017-03-02 16:09:23.869551 osd.5 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:09:27.305424 osd.0 [ERR] OSD full dropping all updates 98% full 2017-03-02 16:09:39.595946 osd.3 [WRN] OSD near full (97%) 2017-03-02 16:09:50.496134 osd.1 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:09:54.373991 osd.5 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:09:58.011731 osd.0 [ERR] OSD full dropping all updates 98% full 2017-03-02 16:10:09.201507 osd.3 [WRN] OSD near full (97%) 2017-03-02 16:10:22.045770 mon.0 [INF] pgmap v140: 248 pgs: 248 active+clean; 2424 MB data, 8953 MB us ed, 1736 MB / 10690 MB avail 2017-03-02 16:10:23.106920 mon.0 [INF] pgmap v141: 248 pgs: 248 active+clean; 2424 MB data, 8779 MB us ed, 1911 MB / 10690 MB avail 2017-03-02 16:10:26.881719 osd.5 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:10:36.333459 mon.0 [INF] pgmap v142: 248 pgs: 248 active+clean; 2424 MB data, 8459 MB us ed, 2230 MB / 10690 MB avail 2017-03-02 16:10:37.451547 mon.0 [INF] pgmap v143: 248 pgs: 248 active+clean; 2424 MB data, 7750 MB us ed, 2939 MB / 10690 MB avail 2017-03-02 16:10:38.369779 mon.0 [INF] osdmap e48: 8 osds: 8 up, 8 in nearfull 2017-03-02 16:10:38.476568 mon.0 [INF] pgmap v144: 248 pgs: 248 active+clean; 2424 MB data, 7750 MB us ed, 2939 MB / 10690 MB avail 2017-03-02 16:10:39.112042 osd.3 [WRN] OSD near full (92%) 2017-03-02 16:10:41.412518 osd.3 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:10:41.583273 osd.5 [ERR] OSD full dropping all updates 98% full 2017-03-02 16:10:43.618776 osd.0 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:10:47.433980 mon.0 [INF] pgmap v145: 248 pgs: 248 active+clean; 2456 MB data, 7858 MB us ed, 2831 MB / 10690 MB avail; 119 kB/s wr, 0 op/s 2017-03-02 16:10:48.135531 mon.0 [INF] osd.1 172.16.1.24:6800/9261 failed (forced) 2017-03-02 16:10:49.504743 mon.0 [INF] pgmap v146: 248 pgs: 248 active+clean; 2700 MB data, 9478 MB us ed, 1211 MB / 10690 MB avail; 10682 kB/s wr, 5 op/s 2017-03-02 16:10:49.509016 mon.0 [INF] osdmap e49: 8 osds: 7 up, 8 in nearfull 2017-03-02 16:10:49.687370 mon.0 [INF] pgmap v147: 248 pgs: 248 active+clean; 2700 MB data, 9478 MB us ed, 1211 MB / 10690 MB avail; 10802 kB/s wr, 5 op/s 2017-03-02 16:10:49.882669 mon.0 [INF] osdmap e50: 8 osds: 7 up, 8 in nearfull 2017-03-02 16:10:50.292783 mon.0 [INF] pgmap v148: 248 pgs: 20 stale+active+clean, 228 active+clean; 2 700 MB data, 9478 MB used, 1211 MB / 10690 MB avail; 20861 kB/s wr, 10 op/s 2017-03-02 16:10:50.942875 mon.0 [INF] osdmap e51: 8 osds: 7 up, 8 in nearfull 2017-03-02 16:10:51.129324 mon.0 [INF] pgmap v149: 248 pgs: 20 stale+active+clean, 228 active+clean; 2 700 MB data, 9478 MB used, 1211 MB / 10690 MB avail; 20852 kB/s wr, 10 op/s 2017-03-02 16:10:45.493219 osd.7 [WRN] OSD near full (91%) 2017-03-02 16:10:52.544006 mon.0 [INF] pgmap v150: 248 pgs: 27 active+undersized+degraded, 12 stale+ac tive+clean, 209 active+clean; 2736 MB data, 9535 MB used, 1154 MB / 10690 MB avail; 23165 kB/s wr, 11 op/s; 84/2052 objects degraded (4.094%) 2017-03-02 16:10:46.238825 osd.6 [WRN] OSD near full (97%) 2017-03-02 16:10:48.539306 osd.6 [ERR] OSD full dropping all updates 98% full 2017-03-02 16:10:56.631291 mon.0 [INF] pgmap v151: 248 pgs: 87 active+undersized+degraded, 161 active+ clean; 2820 MB data, 9710 MB used, 979 MB / 10690 MB avail; 30406 kB/s wr, 14 op/s; 261/2115 objects d egraded (12.340%) 2017-03-02 16:10:56.631605 mon.0 [INF] osdmap e52: 8 osds: 7 up, 8 in full 2017-03-02 16:10:56.759901 mon.0 [INF] pgmap v152: 248 pgs: 87 active+undersized+degraded, 161 active+ clean; 2820 MB data, 9706 MB used, 983 MB / 10690 MB avail; 15048 kB/s wr, 7 op/s; 261/2115 objects de graded (12.340%) 2017-03-02 16:10:57.840071 mon.0 [INF] pgmap v153: 248 pgs: 87 active+undersized+degraded, 161 active+ clean; 2848 MB data, 9719 MB used, 971 MB / 10690 MB avail; 17720 kB/s wr, 8 op/s; 265/2136 objects de graded (12.406%) 2017-03-02 16:11:01.376789 mon.0 [INF] pgmap v154: 248 pgs: 87 active+undersized+degraded, 161 active+ clean; 2848 MB data, 9718 MB used, 971 MB / 10690 MB avail; 13223 kB/s wr, 6 op/s; 265/2136 objects de graded (12.406%) 2017-03-02 16:10:56.740054 osd.6 [WRN] OSD near full (92%) 2017-03-02 16:11:02.545336 mon.0 [INF] pgmap v155: 248 pgs: 87 active+undersized+degraded, 161 active+ clean; 2848 MB data, 9468 MB used, 1221 MB / 10690 MB avail; 13183 kB/s wr, 6 op/s; 265/2136 objects d egraded (12.406%) 2017-03-02 16:11:06.647816 mon.0 [INF] HEALTH_ERR; 87 pgs degraded; 34 pgs stuck unclean; 87 pgs under sized; recovery 265/2136 objects degraded (12.406%); 3 full osd(s); 3 near full osd(s); 1/8 in osds ar e down; full flag(s) set 2017-03-02 16:11:07.865842 mon.0 [INF] pgmap v156: 248 pgs: 87 active+undersized+degraded, 161 active+ clean; 2848 MB data, 9468 MB used, 1222 MB / 10690 MB avail; 7327 kB/s wr, 3 op/s; 265/2136 objects de graded (12.406%) 2017-03-02 16:11:09.780896 osd.1 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:11:09.786239 mon.0 [INF] from='osd.1 172.16.1.24:6800/10572' entity='osd.1' cmd=[{"prefi x": "osd crush create-or-move", "id": 1, "weight":0.0010, "args": ["host=data4", "root=default"]}]: di spatch 2017-03-02 16:11:10.120022 mon.0 [INF] osd.1 172.16.1.24:6800/10572 boot 2017-03-02 16:11:10.192750 mon.0 [INF] osdmap e53: 8 osds: 8 up, 8 in full 2017-03-02 16:11:10.305990 mon.0 [INF] pgmap v157: 248 pgs: 87 active+undersized+degraded, 161 active+ clean; 2848 MB data, 9468 MB used, 1222 MB / 10690 MB avail; 1729 kB/s wr, 0 op/s; 265/2136 objects de graded (12.406%) 2017-03-02 16:11:10.489964 mon.0 [INF] osdmap e54: 8 osds: 8 up, 8 in full 2017-03-02 16:11:10.547510 mon.0 [INF] pgmap v158: 248 pgs: 87 active+undersized+degraded, 161 active+ clean; 2848 MB data, 9468 MB used, 1222 MB / 10690 MB avail; 2071 kB/s wr, 1 op/s; 265/2136 objects de graded (12.406%) 2017-03-02 16:11:11.604070 mon.0 [INF] osdmap e55: 8 osds: 8 up, 8 in full 2017-03-02 16:11:11.743501 mon.0 [INF] pgmap v159: 248 pgs: 5 peering, 1 remapped, 81 active+undersize d+degraded, 161 active+clean; 2848 MB data, 9468 MB used, 1222 MB / 10690 MB avail; 241/2136 objects d egraded (11.283%) 2017-03-02 16:11:11.947407 mon.0 [INF] osdmap e56: 8 osds: 8 up, 8 in full 2017-03-02 16:11:12.277552 mon.0 [INF] pgmap v160: 248 pgs: 8 activating, 24 peering, 1 remapped, 53 a ctive+undersized+degraded, 1 activating+degraded, 161 active+clean; 2848 MB data, 9460 MB used, 1229 M B / 10690 MB avail; 162/2136 objects degraded (7.584%) 2017-03-02 16:11:13.062951 mon.0 [INF] osdmap e57: 8 osds: 8 up, 8 in full 2017-03-02 16:11:13.111860 mon.0 [INF] osd.1 172.16.1.24:6800/10572 failed (forced) 2017-03-02 16:11:13.148836 mon.0 [INF] pgmap v161: 248 pgs: 20 activating, 40 peering, 1 remapped, 20 active+undersized+degraded, 6 activating+degraded, 161 active+clean; 2848 MB data, 9460 MB used, 1229 MB / 10690 MB avail; 63/2136 objects degraded (2.949%) 2017-03-02 16:11:14.224893 mon.0 [INF] osdmap e58: 8 osds: 7 up, 8 in full 2017-03-02 16:11:14.310806 mon.0 [INF] pgmap v162: 248 pgs: 20 activating, 40 peering, 1 remapped, 20 active+undersized+degraded, 6 activating+degraded, 161 active+clean; 2848 MB data, 9460 MB used, 1229 MB / 10690 MB avail; 63/2136 objects degraded (2.949%) 2017-03-02 16:11:14.518233 mon.0 [INF] osdmap e59: 8 osds: 7 up, 8 in full 2017-03-02 16:11:14.606351 mon.0 [INF] pgmap v163: 248 pgs: 20 activating, 40 peering, 1 remapped, 20 active+undersized+degraded, 6 activating+degraded, 161 active+clean; 2848 MB data, 9460 MB used, 1229 MB / 10690 MB avail; 63/2136 objects degraded (2.949%) 2017-03-02 16:11:15.569987 mon.0 [INF] osdmap e60: 8 osds: 7 up, 8 in full 2017-03-02 16:11:15.702331 mon.0 [INF] pgmap v164: 248 pgs: 20 activating, 40 peering, 1 remapped, 20 active+undersized+degraded, 6 activating+degraded, 161 active+clean; 2848 MB data, 9460 MB used, 1229 MB / 10690 MB avail; 63/2136 objects degraded (2.949%) 2017-03-02 16:11:12.921904 osd.0 [WRN] OSD near full (97%) 2017-03-02 16:11:16.790273 mon.0 [INF] pgmap v165: 248 pgs: 12 activating, 16 peering, 54 active+under sized+degraded, 5 activating+degraded, 161 active+clean; 2848 MB data, 9416 MB used, 1273 MB / 10690 M B avail; 162/2136 objects degraded (7.584%); 795 kB/s, 0 objects/s recovering 2017-03-02 16:11:17.871439 mon.0 [INF] pgmap v166: 248 pgs: 87 active+undersized+degraded, 161 active+ clean; 2848 MB data, 9383 MB used, 1307 MB / 10690 MB avail; 255/2136 objects degraded (11.938%); 695 kB/s, 0 objects/s recovering 2017-03-02 16:11:12.618251 osd.3 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:11:14.786176 osd.5 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:11:21.374454 mon.0 [INF] pgmap v167: 248 pgs: 87 active+undersized+degraded, 161 active+ clean; 2848 MB data, 9382 MB used, 1307 MB / 10690 MB avail; 255/2136 objects degraded (11.938%); 493 kB/s, 0 objects/s recovering 2017-03-02 16:11:22.537417 mon.0 [INF] pgmap v168: 248 pgs: 87 active+undersized+degraded, 161 active+ clean; 2848 MB data, 9380 MB used, 1309 MB / 10690 MB avail; 255/2136 objects degraded (11.938%); 497 kB/s, 0 objects/s recovering 2017-03-02 16:11:26.845284 osd.6 [WRN] OSD near full (90%) 2017-03-02 16:11:44.727636 osd.0 [WRN] OSD near full (97%) 2017-03-02 16:11:44.688298 osd.5 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:11:47.722370 osd.3 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:11:57.750631 osd.6 [WRN] OSD near full (90%) 2017-03-02 16:12:06.648141 mon.0 [INF] HEALTH_ERR; 87 pgs degraded; 21 pgs stuck unclean; 87 pgs under sized; recovery 255/2136 objects degraded (11.938%); 3 full osd(s); 3 near full osd(s); 1/8 in osds ar e down; full flag(s) set 2017-03-02 16:12:14.531442 osd.0 [WRN] OSD near full (97%) 2017-03-02 16:12:17.526405 osd.3 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:12:18.091959 osd.5 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:12:28.754698 osd.6 [WRN] OSD near full (90%) 2017-03-02 16:12:44.935946 osd.0 [WRN] OSD near full (97%) 2017-03-02 16:12:47.231282 osd.3 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:12:50.597548 osd.5 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:13:02.960850 osd.6 [WRN] OSD near full (90%) 2017-03-02 16:13:15.241017 osd.0 [WRN] OSD near full (97%) 2017-03-02 16:13:18.235574 osd.3 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:13:21.101305 osd.5 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:13:34.766836 osd.6 [WRN] OSD near full (90%) 2017-03-02 16:13:47.246341 osd.0 [WRN] OSD near full (97%) 2017-03-02 16:13:50.041517 osd.3 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:13:53.105995 osd.5 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:14:05.472628 osd.6 [WRN] OSD near full (90%) 2017-03-02 16:14:17.451810 osd.0 [WRN] OSD near full (97%) 2017-03-02 16:14:22.646883 osd.3 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:14:24.710631 osd.5 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:14:35.777537 osd.6 [WRN] OSD near full (90%) 2017-03-02 16:14:50.856136 osd.0 [WRN] OSD near full (97%) 2017-03-02 16:14:56.853288 osd.3 [ERR] OSD full dropping all updates 100% full 2017-03-02 16:14:57.017252 osd.5 [ERR] OSD full dropping all updates 99% full 2017-03-02 16:15:06.236347 7fd792ffd700 0 -- 172.16.1.254:0/1637311347 >> 172.16.1.11:6800/6899 conn( 0x7fd780009fb0 :-1 s=STATE_OPEN pgs=837 cs=1 l=0).fault initiating reconnect ^Cadmin:~ # ceph -s cluster 557c4525-cad3-371a-8e05-4b95f6accdf5 health HEALTH_ERR 87 pgs degraded 21 pgs stuck unclean 87 pgs undersized recovery 255/2136 objects degraded (11.938%) 3 full osd(s) 3 near full osd(s) 1/8 in osds are down full flag(s) set monmap e2: 3 mons at {data1=172.16.1.21:6789/0,data2=172.16.1.22:6789/0,mon1=172.16.1.11:6789/0} election epoch 6, quorum 0,1,2 mon1,data1,data2 mgr active: mon1 standbys: data1, data2 osdmap e60: 8 osds: 7 up, 8 in; 66 remapped pgs flags nearfull,full,sortbitwise,require_jewel_osds,require_kraken_osds,require_luminous_os ds pgmap v168: 248 pgs, 1 pools, 2848 MB data, 712 objects 9380 MB used, 1309 MB / 10690 MB avail 255/2136 objects degraded (11.938%) 161 active+clean 87 active+undersized+degraded admin:~ # ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 0.01035 root default -2 0.00198 host data3 0 0.00099 osd.0 up 1.00000 1.00000 3 0.00099 osd.3 up 1.00000 1.00000 -3 0.00339 host data2 2 0.00169 osd.2 up 1.00000 1.00000 4 0.00169 osd.4 up 1.00000 1.00000 -4 0.00198 host data4 1 0.00099 osd.1 down 1.00000 1.00000 5 0.00099 osd.5 up 1.00000 1.00000 -5 0.00299 host data1 6 0.00150 osd.6 up 1.00000 1.00000 7 0.00150 osd.7 up 1.00000 1.00000 admin:~ #