2017-09-28 12:18:14.647068 mon.compute1 mon.0 10.200.246.20:6789/0 355 : cluster [INF] osd.3 marked down after no beacon for 300.736645 seconds
2017-09-28 12:18:14.649124 mon.compute1 mon.0 10.200.246.20:6789/0 356 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 12:18:17.183940 mon.compute1 mon.0 10.200.246.20:6789/0 366 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 42 pgs peering (PG_AVAILABILITY)
2017-09-28 12:18:17.184014 mon.compute1 mon.0 10.200.246.20:6789/0 367 : cluster [WRN] Health check failed: Degraded data redundancy: 58 pgs unclean, 16 pgs degraded (PG_DEGRADED)
2017-09-28 12:18:18.316954 mon.compute1 mon.0 10.200.246.20:6789/0 368 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 12:18:18.390223 mon.compute1 mon.0 10.200.246.20:6789/0 369 : cluster [INF] osd.3 10.200.246.61:6800/1478 boot
2017-09-28 12:18:17.536317 osd.3 osd.3 10.200.246.61:6800/1478 5 : cluster [WRN] Monitor daemon marked osd.3 down, but it is still running
2017-09-28 12:18:19.544034 mon.compute1 mon.0 10.200.246.20:6789/0 378 : cluster [WRN] Health check update: Degraded data redundancy: 7/96 objects degraded (7.292%), 116 pgs unclean, 74 pgs degraded (PG_DEGRADED)
2017-09-28 12:18:22.642222 mon.compute1 mon.0 10.200.246.20:6789/0 379 : cluster [WRN] Health check update: Degraded data redundancy: 6/96 objects degraded (6.250%), 34 pgs unclean, 34 pgs degraded (PG_DEGRADED)
2017-09-28 12:18:22.642310 mon.compute1 mon.0 10.200.246.20:6789/0 380 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 42 pgs peering)
2017-09-28 12:18:25.543959 mon.compute1 mon.0 10.200.246.20:6789/0 382 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/96 objects degraded (6.250%), 34 pgs unclean, 34 pgs degraded)
2017-09-28 12:18:25.544030 mon.compute1 mon.0 10.200.246.20:6789/0 383 : cluster [INF] Cluster is now healthy
2017-09-28 12:18:29.649801 mon.compute1 mon.0 10.200.246.20:6789/0 384 : cluster [INF] osd.6 marked down after no beacon for 300.186509 seconds
2017-09-28 12:18:29.651629 mon.compute1 mon.0 10.200.246.20:6789/0 385 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 12:18:32.111352 mon.compute1 mon.0 10.200.246.20:6789/0 394 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 12:18:32.111424 mon.compute1 mon.0 10.200.246.20:6789/0 395 : cluster [INF] Cluster is now healthy
2017-09-28 12:18:32.190248 mon.compute1 mon.0 10.200.246.20:6789/0 396 : cluster [INF] osd.6 10.200.246.63:6800/1410 boot
2017-09-28 12:18:31.318335 osd.6 osd.6 10.200.246.63:6800/1410 1 : cluster [WRN] Monitor daemon marked osd.6 down, but it is still running
2017-09-28 12:18:33.446904 mon.compute1 mon.0 10.200.246.20:6789/0 405 : cluster [WRN] Health check failed: Reduced data availability: 30 pgs inactive, 33 pgs peering (PG_AVAILABILITY)
2017-09-28 12:18:33.446977 mon.compute1 mon.0 10.200.246.20:6789/0 406 : cluster [WRN] Health check failed: Degraded data redundancy: 1/96 objects degraded (1.042%), 50 pgs unclean, 20 pgs degraded (PG_DEGRADED)
2017-09-28 12:18:35.424075 mon.compute1 mon.0 10.200.246.20:6789/0 411 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive, 17 pgs peering (PG_AVAILABILITY)
2017-09-28 12:18:35.424167 mon.compute1 mon.0 10.200.246.20:6789/0 412 : cluster [WRN] Health check update: Degraded data redundancy: 1/96 objects degraded (1.042%), 50 pgs unclean, 12 pgs degraded (PG_DEGRADED)
2017-09-28 12:18:36.650825 mon.compute1 mon.0 10.200.246.20:6789/0 413 : cluster [WRN] Health check update: Degraded data redundancy: 50 pgs unclean (PG_DEGRADED)
2017-09-28 12:18:38.651705 mon.compute1 mon.0 10.200.246.20:6789/0 414 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 17 pgs peering)
2017-09-28 12:18:38.651811 mon.compute1 mon.0 10.200.246.20:6789/0 415 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50 pgs unclean)
2017-09-28 12:18:38.651861 mon.compute1 mon.0 10.200.246.20:6789/0 416 : cluster [INF] Cluster is now healthy
2017-09-28 12:29:09.878040 mon.compute1 mon.0 10.200.246.20:6789/0 425 : cluster [INF] osd.2 marked down after no beacon for 301.028929 seconds
2017-09-28 12:29:09.880256 mon.compute1 mon.0 10.200.246.20:6789/0 426 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 12:29:11.530727 mon.compute1 mon.0 10.200.246.20:6789/0 439 : cluster [WRN] Health check failed: Reduced data availability: 21 pgs peering (PG_AVAILABILITY)
2017-09-28 12:29:11.530779 mon.compute1 mon.0 10.200.246.20:6789/0 440 : cluster [WRN] Health check failed: Degraded data redundancy: 2/96 objects degraded (2.083%), 48 pgs unclean, 27 pgs degraded (PG_DEGRADED)
2017-09-28 12:29:12.898134 mon.compute1 mon.0 10.200.246.20:6789/0 441 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 12:29:12.965602 mon.compute1 mon.0 10.200.246.20:6789/0 442 : cluster [INF] osd.2 10.200.246.61:6804/1802 boot
2017-09-28 12:29:13.953566 mon.compute1 mon.0 10.200.246.20:6789/0 451 : cluster [WRN] Health check update: Degraded data redundancy: 6/96 objects degraded (6.250%), 88 pgs unclean, 67 pgs degraded (PG_DEGRADED)
2017-09-28 12:29:15.953698 mon.compute1 mon.0 10.200.246.20:6789/0 456 : cluster [WRN] Health check update: Degraded data redundancy: 6/96 objects degraded (6.250%), 113 pgs unclean, 67 pgs degraded (PG_DEGRADED)
2017-09-28 12:29:17.153823 mon.compute1 mon.0 10.200.246.20:6789/0 457 : cluster [WRN] Health check update: Degraded data redundancy: 1/96 objects degraded (1.042%), 42 pgs unclean, 17 pgs degraded (PG_DEGRADED)
2017-09-28 12:29:17.153894 mon.compute1 mon.0 10.200.246.20:6789/0 458 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 21 pgs peering)
2017-09-28 12:29:19.250750 mon.compute1 mon.0 10.200.246.20:6789/0 459 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/96 objects degraded (1.042%), 42 pgs unclean, 17 pgs degraded)
2017-09-28 12:29:19.250836 mon.compute1 mon.0 10.200.246.20:6789/0 460 : cluster [INF] Cluster is now healthy
2017-09-28 12:29:11.753340 osd.2 osd.2 10.200.246.61:6804/1802 5 : cluster [WRN] Monitor daemon marked osd.2 down, but it is still running
2017-09-28 12:34:54.944046 mon.compute1 mon.0 10.200.246.20:6789/0 462 : cluster [INF] osd.4 marked down after no beacon for 300.178744 seconds
2017-09-28 12:34:54.944159 mon.compute1 mon.0 10.200.246.20:6789/0 463 : cluster [INF] osd.5 marked down after no beacon for 300.201615 seconds
2017-09-28 12:34:54.946235 mon.compute1 mon.0 10.200.246.20:6789/0 464 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)
2017-09-28 12:34:54.946315 mon.compute1 mon.0 10.200.246.20:6789/0 465 : cluster [WRN] Health check failed: 1 host (2 osds) down (OSD_HOST_DOWN)
2017-09-28 12:34:55.294229 mon.compute1 mon.0 10.200.246.20:6789/0 470 : cluster [WRN] overall HEALTH_WARN 2 osds down; 1 host (2 osds) down
2017-09-28 12:34:57.492571 mon.compute1 mon.0 10.200.246.20:6789/0 479 : cluster [WRN] Health check failed: Reduced data availability: 30 pgs inactive, 126 pgs peering (PG_AVAILABILITY)
2017-09-28 12:34:57.492643 mon.compute1 mon.0 10.200.246.20:6789/0 480 : cluster [WRN] Health check failed: Degraded data redundancy: 5/96 objects degraded (5.208%), 191 pgs unclean, 66 pgs degraded (PG_DEGRADED)
2017-09-28 12:34:57.678557 osd.5 osd.5 10.200.246.62:6804/1867 5 : cluster [WRN] Monitor daemon marked osd.5 down, but it is still running
2017-09-28 12:34:59.591243 mon.compute1 mon.0 10.200.246.20:6789/0 481 : cluster [INF] Health check cleared: OSD_DOWN (was: 2 osds down)
2017-09-28 12:34:59.591309 mon.compute1 mon.0 10.200.246.20:6789/0 482 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
2017-09-28 12:34:59.606822 mon.compute1 mon.0 10.200.246.20:6789/0 483 : cluster [WRN] Health check update: Degraded data redundancy: 16/96 objects degraded (16.667%), 314 pgs unclean, 189 pgs degraded (PG_DEGRADED)
2017-09-28 12:34:57.952015 osd.4 osd.4 10.200.246.62:6800/1478 5 : cluster [WRN] Monitor daemon marked osd.4 down, but it is still running
2017-09-28 12:34:59.658887 mon.compute1 mon.0 10.200.246.20:6789/0 484 : cluster [INF] osd.5 10.200.246.62:6804/1867 boot
2017-09-28 12:34:59.659052 mon.compute1 mon.0 10.200.246.20:6789/0 485 : cluster [INF] osd.4 10.200.246.62:6800/1478 boot
2017-09-28 12:35:02.081135 mon.compute1 mon.0 10.200.246.20:6789/0 498 : cluster [WRN] Health check update: Reduced data availability: 48 pgs inactive, 55 pgs peering (PG_AVAILABILITY)
2017-09-28 12:35:02.081235 mon.compute1 mon.0 10.200.246.20:6789/0 499 : cluster [WRN] Health check update: Degraded data redundancy: 13/96 objects degraded (13.542%), 295 pgs unclean, 103 pgs degraded (PG_DEGRADED)
2017-09-28 12:35:03.246886 mon.compute1 mon.0 10.200.246.20:6789/0 500 : cluster [WRN] Health check update: Degraded data redundancy: 7/96 objects degraded (7.292%), 231 pgs unclean, 38 pgs degraded (PG_DEGRADED)
2017-09-28 12:35:05.366839 mon.compute1 mon.0 10.200.246.20:6789/0 501 : cluster [WRN] Health check update: Degraded data redundancy: 193 pgs unclean (PG_DEGRADED)
2017-09-28 12:35:07.260292 mon.compute1 mon.0 10.200.246.20:6789/0 502 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 48 pgs inactive, 55 pgs peering)
2017-09-28 12:35:07.260382 mon.compute1 mon.0 10.200.246.20:6789/0 503 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 193 pgs unclean)
2017-09-28 12:35:07.260434 mon.compute1 mon.0 10.200.246.20:6789/0 504 : cluster [INF] Cluster is now healthy
2017-09-28 12:35:55.303388 mon.compute1 mon.0 10.200.246.20:6789/0 505 : cluster [INF] overall HEALTH_OK
2017-09-28 12:37:29.961257 mon.compute1 mon.0 10.200.246.20:6789/0 507 : cluster [INF] osd.0 marked down after no beacon for 300.786825 seconds
2017-09-28 12:37:29.961338 mon.compute1 mon.0 10.200.246.20:6789/0 508 : cluster [INF] osd.1 marked down after no beacon for 300.598574 seconds
2017-09-28 12:37:29.963214 mon.compute1 mon.0 10.200.246.20:6789/0 509 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)
2017-09-28 12:37:29.963291 mon.compute1 mon.0 10.200.246.20:6789/0 510 : cluster [WRN] Health check failed: 1 host (2 osds) down (OSD_HOST_DOWN)
2017-09-28 12:37:31.570930 mon.compute1 mon.0 10.200.246.20:6789/0 523 : cluster [WRN] Health check failed: Reduced data availability: 11 pgs inactive, 99 pgs peering (PG_AVAILABILITY)
2017-09-28 12:37:31.571022 mon.compute1 mon.0 10.200.246.20:6789/0 524 : cluster [WRN] Health check failed: Degraded data redundancy: 4/96 objects degraded (4.167%), 127 pgs unclean, 28 pgs degraded (PG_DEGRADED)
2017-09-28 12:37:33.509784 mon.compute1 mon.0 10.200.246.20:6789/0 525 : cluster [WRN] Health check update: 1 osds down (OSD_DOWN)
2017-09-28 12:37:33.509858 mon.compute1 mon.0 10.200.246.20:6789/0 526 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
2017-09-28 12:37:33.525266 mon.compute1 mon.0 10.200.246.20:6789/0 527 : cluster [WRN] Health check update: Degraded data redundancy: 7/96 objects degraded (7.292%), 162 pgs unclean, 63 pgs degraded (PG_DEGRADED)
2017-09-28 12:37:33.600034 mon.compute1 mon.0 10.200.246.20:6789/0 528 : cluster [INF] osd.1 10.200.246.20:6804/1975 boot
2017-09-28 12:37:34.589805 mon.compute1 mon.0 10.200.246.20:6789/0 533 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 12:37:34.657487 mon.compute1 mon.0 10.200.246.20:6789/0 534 : cluster [INF] osd.0 10.200.246.20:6800/1732 boot
2017-09-28 12:37:35.645238 mon.compute1 mon.0 10.200.246.20:6789/0 539 : cluster [WRN] Health check update: Degraded data redundancy: 7/96 objects degraded (7.292%), 189 pgs unclean, 82 pgs degraded (PG_DEGRADED)
2017-09-28 12:37:37.645301 mon.compute1 mon.0 10.200.246.20:6789/0 544 : cluster [WRN] Health check update: Degraded data redundancy: 2/96 objects degraded (2.083%), 99 pgs unclean, 37 pgs degraded (PG_DEGRADED)
2017-09-28 12:37:37.645394 mon.compute1 mon.0 10.200.246.20:6789/0 545 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 99 pgs peering)
2017-09-28 12:37:39.355865 mon.compute1 mon.0 10.200.246.20:6789/0 546 : cluster [WRN] Health check update: Degraded data redundancy: 54 pgs unclean (PG_DEGRADED)
2017-09-28 12:37:33.200656 osd.1 osd.1 10.200.246.20:6804/1975 5 : cluster [WRN] Monitor daemon marked osd.1 down, but it is still running
2017-09-28 12:37:33.213274 osd.0 osd.0 10.200.246.20:6800/1732 5 : cluster [WRN] Monitor daemon marked osd.0 down, but it is still running
2017-09-28 12:37:41.588137 mon.compute1 mon.0 10.200.246.20:6789/0 547 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 54 pgs unclean)
2017-09-28 12:37:41.588199 mon.compute1 mon.0 10.200.246.20:6789/0 548 : cluster [INF] Cluster is now healthy
2017-09-28 12:43:20.007399 mon.compute1 mon.0 10.200.246.20:6789/0 551 : cluster [INF] osd.3 marked down after no beacon for 300.941190 seconds
2017-09-28 12:43:20.009632 mon.compute1 mon.0 10.200.246.20:6789/0 552 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 12:43:21.443008 mon.compute1 mon.0 10.200.246.20:6789/0 561 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 12:43:21.443070 mon.compute1 mon.0 10.200.246.20:6789/0 562 : cluster [INF] Cluster is now healthy
2017-09-28 12:43:21.499171 mon.compute1 mon.0 10.200.246.20:6789/0 563 : cluster [INF] osd.3 10.200.246.61:6800/1478 boot
2017-09-28 12:43:22.498769 mon.compute1 mon.0 10.200.246.20:6789/0 568 : cluster [WRN] Health check failed: Reduced data availability: 8 pgs inactive, 33 pgs peering (PG_AVAILABILITY)
2017-09-28 12:43:22.498832 mon.compute1 mon.0 10.200.246.20:6789/0 569 : cluster [WRN] Health check failed: Degraded data redundancy: 6/96 objects degraded (6.250%), 91 pgs unclean, 58 pgs degraded (PG_DEGRADED)
2017-09-28 12:43:27.570749 mon.compute1 mon.0 10.200.246.20:6789/0 574 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 33 pgs peering)
2017-09-28 12:43:27.570824 mon.compute1 mon.0 10.200.246.20:6789/0 575 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/96 objects degraded (6.250%), 91 pgs unclean, 58 pgs degraded)
2017-09-28 12:43:27.570873 mon.compute1 mon.0 10.200.246.20:6789/0 576 : cluster [INF] Cluster is now healthy
2017-09-28 12:43:20.819929 osd.3 osd.3 10.200.246.61:6800/1478 7 : cluster [WRN] Monitor daemon marked osd.3 down, but it is still running
2017-09-28 12:43:35.019388 mon.compute1 mon.0 10.200.246.20:6789/0 577 : cluster [INF] osd.6 marked down after no beacon for 300.377604 seconds
2017-09-28 12:43:35.019647 mon.compute1 mon.0 10.200.246.20:6789/0 578 : cluster [INF] osd.7 marked down after no beacon for 300.078165 seconds
2017-09-28 12:43:35.021628 mon.compute1 mon.0 10.200.246.20:6789/0 579 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)
2017-09-28 12:43:35.021722 mon.compute1 mon.0 10.200.246.20:6789/0 580 : cluster [WRN] Health check failed: 1 host (2 osds) down (OSD_HOST_DOWN)
2017-09-28 12:43:37.577047 mon.compute1 mon.0 10.200.246.20:6789/0 589 : cluster [WRN] Health check failed: Reduced data availability: 52 pgs inactive, 69 pgs peering (PG_AVAILABILITY)
2017-09-28 12:43:37.577119 mon.compute1 mon.0 10.200.246.20:6789/0 590 : cluster [WRN] Health check failed: Degraded data redundancy: 3/96 objects degraded (3.125%), 101 pgs unclean, 39 pgs degraded (PG_DEGRADED)
2017-09-28 12:43:38.728757 mon.compute1 mon.0 10.200.246.20:6789/0 591 : cluster [INF] Health check cleared: OSD_DOWN (was: 2 osds down)
2017-09-28 12:43:38.728854 mon.compute1 mon.0 10.200.246.20:6789/0 592 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
2017-09-28 12:43:38.825072 mon.compute1 mon.0 10.200.246.20:6789/0 593 : cluster [INF] osd.6 10.200.246.63:6800/1410 boot
2017-09-28 12:43:38.825202 mon.compute1 mon.0 10.200.246.20:6789/0 594 : cluster [INF] osd.7 10.200.246.63:6804/1636 boot
2017-09-28 12:43:42.333003 mon.compute1 mon.0 10.200.246.20:6789/0 607 : cluster [WRN] Health check update: Degraded data redundancy: 1/96 objects degraded (1.042%), 101 pgs unclean, 18 pgs degraded (PG_DEGRADED)
2017-09-28 12:43:37.088982 osd.7 osd.7 10.200.246.63:6804/1636 1 : cluster [WRN] Monitor daemon marked osd.7 down, but it is still running
2017-09-28 12:43:37.147367 osd.6 osd.6 10.200.246.63:6800/1410 3 : cluster [WRN] Monitor daemon marked osd.6 down, but it is still running
2017-09-28 12:43:43.582194 mon.compute1 mon.0 10.200.246.20:6789/0 608 : cluster [WRN] Health check update: Reduced data availability: 30 pgs inactive, 42 pgs peering (PG_AVAILABILITY)
2017-09-28 12:43:43.582263 mon.compute1 mon.0 10.200.246.20:6789/0 609 : cluster [WRN] Health check update: Degraded data redundancy: 50 pgs unclean, 8 pgs degraded (PG_DEGRADED)
2017-09-28 12:43:45.624364 mon.compute1 mon.0 10.200.246.20:6789/0 610 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 30 pgs inactive, 42 pgs peering)
2017-09-28 12:43:45.624431 mon.compute1 mon.0 10.200.246.20:6789/0 611 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50 pgs unclean, 8 pgs degraded)
2017-09-28 12:43:45.624515 mon.compute1 mon.0 10.200.246.20:6789/0 612 : cluster [INF] Cluster is now healthy
2017-09-28 12:59:15.173655 mon.compute1 mon.0 10.200.246.20:6789/0 618 : cluster [INF] osd.2 marked down after no beacon for 300.122351 seconds
2017-09-28 12:59:15.175601 mon.compute1 mon.0 10.200.246.20:6789/0 619 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 12:59:16.682022 mon.compute1 mon.0 10.200.246.20:6789/0 628 : cluster [WRN] Health check failed: Degraded data redundancy: 1/96 objects degraded (1.042%), 12 pgs unclean, 12 pgs degraded (PG_DEGRADED)
2017-09-28 12:59:18.556348 mon.compute1 mon.0 10.200.246.20:6789/0 633 : cluster [WRN] Health check failed: Reduced data availability: 41 pgs peering (PG_AVAILABILITY)
2017-09-28 12:59:18.556415 mon.compute1 mon.0 10.200.246.20:6789/0 634 : cluster [WRN] Health check update: Degraded data redundancy: 4/96 objects degraded (4.167%), 71 pgs unclean, 30 pgs degraded (PG_DEGRADED)
2017-09-28 12:59:18.575006 mon.compute1 mon.0 10.200.246.20:6789/0 635 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 12:59:18.210768 osd.2 osd.2 10.200.246.61:6804/1802 7 : cluster [WRN] Monitor daemon marked osd.2 down, but it is still running
2017-09-28 12:59:18.740067 mon.compute1 mon.0 10.200.246.20:6789/0 636 : cluster [INF] osd.2 10.200.246.61:6804/1802 boot
2017-09-28 12:59:20.893543 mon.compute1 mon.0 10.200.246.20:6789/0 645 : cluster [WRN] Health check update: Degraded data redundancy: 5/96 objects degraded (5.208%), 118 pgs unclean, 52 pgs degraded (PG_DEGRADED)
2017-09-28 12:59:22.990669 mon.compute1 mon.0 10.200.246.20:6789/0 646 : cluster [WRN] Health check update: Degraded data redundancy: 1/96 objects degraded (1.042%), 42 pgs unclean, 17 pgs degraded (PG_DEGRADED)
2017-09-28 12:59:22.990768 mon.compute1 mon.0 10.200.246.20:6789/0 647 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 41 pgs peering)
2017-09-28 12:59:24.139210 mon.compute1 mon.0 10.200.246.20:6789/0 648 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/96 objects degraded (1.042%), 42 pgs unclean, 17 pgs degraded)
2017-09-28 12:59:24.139290 mon.compute1 mon.0 10.200.246.20:6789/0 649 : cluster [INF] Cluster is now healthy
2017-09-28 13:00:00.000128 mon.compute1 mon.0 10.200.246.20:6789/0 651 : cluster [INF] overall HEALTH_OK
2017-09-28 13:00:00.202184 mon.compute1 mon.0 10.200.246.20:6789/0 652 : cluster [INF] osd.4 marked down after no beacon for 300.286577 seconds
2017-09-28 13:00:00.202267 mon.compute1 mon.0 10.200.246.20:6789/0 653 : cluster [INF] osd.5 marked down after no beacon for 300.308353 seconds
2017-09-28 13:00:00.204322 mon.compute1 mon.0 10.200.246.20:6789/0 654 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)
2017-09-28 13:00:00.204391 mon.compute1 mon.0 10.200.246.20:6789/0 655 : cluster [WRN] Health check failed: 1 host (2 osds) down (OSD_HOST_DOWN)
2017-09-28 13:00:02.367807 mon.compute1 mon.0 10.200.246.20:6789/0 664 : cluster [WRN] Health check failed: Reduced data availability: 21 pgs inactive, 16 pgs peering (PG_AVAILABILITY)
2017-09-28 13:00:02.367869 mon.compute1 mon.0 10.200.246.20:6789/0 665 : cluster [WRN] Health check failed: Degraded data redundancy: 54 pgs unclean, 39 pgs degraded (PG_DEGRADED)
2017-09-28 13:00:04.539160 mon.compute1 mon.0 10.200.246.20:6789/0 666 : cluster [WRN] Health check update: Degraded data redundancy: 11/96 objects degraded (11.458%), 96 pgs unclean, 162 pgs degraded (PG_DEGRADED)
2017-09-28 13:00:04.552221 mon.compute1 mon.0 10.200.246.20:6789/0 667 : cluster [INF] Health check cleared: OSD_DOWN (was: 2 osds down)
2017-09-28 13:00:04.552289 mon.compute1 mon.0 10.200.246.20:6789/0 668 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
2017-09-28 13:00:04.699683 mon.compute1 mon.0 10.200.246.20:6789/0 669 : cluster [INF] osd.5 10.200.246.62:6804/1867 boot
2017-09-28 13:00:04.699862 mon.compute1 mon.0 10.200.246.20:6789/0 670 : cluster [INF] osd.4 10.200.246.62:6800/1478 boot
2017-09-28 13:00:08.207819 mon.compute1 mon.0 10.200.246.20:6789/0 683 : cluster [WRN] Health check update: Reduced data availability: 32 pgs inactive (PG_AVAILABILITY)
2017-09-28 13:00:08.207888 mon.compute1 mon.0 10.200.246.20:6789/0 684 : cluster [WRN] Health check update: Degraded data redundancy: 10/96 objects degraded (10.417%), 104 pgs unclean, 81 pgs degraded (PG_DEGRADED)
2017-09-28 13:00:03.566759 osd.5 osd.5 10.200.246.62:6804/1867 7 : cluster [WRN] Monitor daemon marked osd.5 down, but it is still running
2017-09-28 13:00:03.510626 osd.4 osd.4 10.200.246.62:6800/1478 7 : cluster [WRN] Monitor daemon marked osd.4 down, but it is still running
2017-09-28 13:00:10.442090 mon.compute1 mon.0 10.200.246.20:6789/0 685 : cluster [WRN] Health check update: Degraded data redundancy: 104 pgs unclean (PG_DEGRADED)
2017-09-28 13:00:12.653514 mon.compute1 mon.0 10.200.246.20:6789/0 686 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 32 pgs inactive)
2017-09-28 13:00:12.653602 mon.compute1 mon.0 10.200.246.20:6789/0 687 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 104 pgs unclean)
2017-09-28 13:00:12.653681 mon.compute1 mon.0 10.200.246.20:6789/0 688 : cluster [INF] Cluster is now healthy
2017-09-28 13:02:35.297827 mon.compute1 mon.0 10.200.246.20:6789/0 691 : cluster [INF] osd.0 marked down after no beacon for 300.954480 seconds
2017-09-28 13:02:35.297932 mon.compute1 mon.0 10.200.246.20:6789/0 692 : cluster [INF] osd.1 marked down after no beacon for 300.757511 seconds
2017-09-28 13:02:35.299789 mon.compute1 mon.0 10.200.246.20:6789/0 693 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)
2017-09-28 13:02:35.299877 mon.compute1 mon.0 10.200.246.20:6789/0 694 : cluster [WRN] Health check failed: 1 host (2 osds) down (OSD_HOST_DOWN)
2017-09-28 13:02:36.790626 mon.compute1 mon.0 10.200.246.20:6789/0 703 : cluster [INF] Health check cleared: OSD_DOWN (was: 2 osds down)
2017-09-28 13:02:36.790712 mon.compute1 mon.0 10.200.246.20:6789/0 704 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
2017-09-28 13:02:36.790799 mon.compute1 mon.0 10.200.246.20:6789/0 705 : cluster [INF] Cluster is now healthy
2017-09-28 13:02:36.806242 mon.compute1 mon.0 10.200.246.20:6789/0 706 : cluster [WRN] Health check failed: Reduced data availability: 37 pgs inactive, 149 pgs peering (PG_AVAILABILITY)
2017-09-28 13:02:36.806313 mon.compute1 mon.0 10.200.246.20:6789/0 707 : cluster [WRN] Health check failed: Degraded data redundancy: 149 pgs unclean (PG_DEGRADED)
2017-09-28 13:02:36.869832 mon.compute1 mon.0 10.200.246.20:6789/0 708 : cluster [INF] osd.0 10.200.246.20:6800/1732 boot
2017-09-28 13:02:36.870042 mon.compute1 mon.0 10.200.246.20:6789/0 709 : cluster [INF] osd.1 10.200.246.20:6804/1975 boot
2017-09-28 13:02:35.925726 osd.0 osd.0 10.200.246.20:6800/1732 7 : cluster [WRN] Monitor daemon marked osd.0 down, but it is still running
2017-09-28 13:02:35.942101 osd.1 osd.1 10.200.246.20:6804/1975 7 : cluster [WRN] Monitor daemon marked osd.1 down, but it is still running
2017-09-28 13:02:42.292070 mon.compute1 mon.0 10.200.246.20:6789/0 718 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 37 pgs inactive, 149 pgs peering)
2017-09-28 13:02:42.292138 mon.compute1 mon.0 10.200.246.20:6789/0 719 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 149 pgs unclean)
2017-09-28 13:02:42.292187 mon.compute1 mon.0 10.200.246.20:6789/0 720 : cluster [INF] Cluster is now healthy
2017-09-28 13:08:40.521936 mon.compute1 mon.0 10.200.246.20:6789/0 726 : cluster [INF] osd.6 marked down after no beacon for 300.702597 seconds
2017-09-28 13:08:40.522013 mon.compute1 mon.0 10.200.246.20:6789/0 727 : cluster [INF] osd.7 marked down after no beacon for 300.403778 seconds
2017-09-28 13:08:40.524189 mon.compute1 mon.0 10.200.246.20:6789/0 728 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)
2017-09-28 13:08:40.524266 mon.compute1 mon.0 10.200.246.20:6789/0 729 : cluster [WRN] Health check failed: 1 host (2 osds) down (OSD_HOST_DOWN)
2017-09-28 13:08:41.643572 mon.compute1 mon.0 10.200.246.20:6789/0 734 : cluster [INF] Health check cleared: OSD_DOWN (was: 2 osds down)
2017-09-28 13:08:41.643656 mon.compute1 mon.0 10.200.246.20:6789/0 735 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
2017-09-28 13:08:41.643704 mon.compute1 mon.0 10.200.246.20:6789/0 736 : cluster [INF] Cluster is now healthy
2017-09-28 13:08:41.711568 mon.compute1 mon.0 10.200.246.20:6789/0 737 : cluster [INF] osd.7 10.200.246.63:6804/1636 boot
2017-09-28 13:08:41.711753 mon.compute1 mon.0 10.200.246.20:6789/0 738 : cluster [INF] osd.6 10.200.246.63:6800/1410 boot
2017-09-28 13:08:40.733107 osd.7 osd.7 10.200.246.63:6804/1636 3 : cluster [WRN] Monitor daemon marked osd.7 down, but it is still running
2017-09-28 13:08:41.066419 osd.6 osd.6 10.200.246.63:6800/1410 5 : cluster [WRN] Monitor daemon marked osd.6 down, but it is still running
2017-09-28 13:08:43.024886 mon.compute1 mon.0 10.200.246.20:6789/0 747 : cluster [WRN] Health check failed: Reduced data availability: 82 pgs inactive, 312 pgs peering (PG_AVAILABILITY)
2017-09-28 13:08:43.024957 mon.compute1 mon.0 10.200.246.20:6789/0 748 : cluster [WRN] Health check failed: Degraded data redundancy: 312 pgs unclean (PG_DEGRADED)
2017-09-28 13:08:44.819187 mon.compute1 mon.0 10.200.246.20:6789/0 749 : cluster [WRN] Health check update: Reduced data availability: 211 pgs peering (PG_AVAILABILITY)
2017-09-28 13:08:44.819255 mon.compute1 mon.0 10.200.246.20:6789/0 750 : cluster [WRN] Health check update: Degraded data redundancy: 211 pgs unclean (PG_DEGRADED)
2017-09-28 13:08:46.516856 mon.compute1 mon.0 10.200.246.20:6789/0 751 : cluster [WRN] Health check update: Reduced data availability: 20 pgs peering (PG_AVAILABILITY)
2017-09-28 13:08:46.516923 mon.compute1 mon.0 10.200.246.20:6789/0 752 : cluster [WRN] Health check update: Degraded data redundancy: 20 pgs unclean (PG_DEGRADED)
2017-09-28 13:08:48.517599 mon.compute1 mon.0 10.200.246.20:6789/0 753 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 20 pgs peering)
2017-09-28 13:08:48.517677 mon.compute1 mon.0 10.200.246.20:6789/0 754 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20 pgs unclean)
2017-09-28 13:08:48.517728 mon.compute1 mon.0 10.200.246.20:6789/0 755 : cluster [INF] Cluster is now healthy
2017-09-28 13:13:25.697429 mon.compute1 mon.0 10.200.246.20:6789/0 757 : cluster [INF] osd.3 marked down after no beacon for 300.447373 seconds
2017-09-28 13:13:25.699460 mon.compute1 mon.0 10.200.246.20:6789/0 758 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 13:13:27.107769 mon.compute1 mon.0 10.200.246.20:6789/0 767 : cluster [WRN] Health check failed: Reduced data availability: 23 pgs peering (PG_AVAILABILITY)
2017-09-28 13:13:27.107848 mon.compute1 mon.0 10.200.246.20:6789/0 768 : cluster [WRN] Health check failed: Degraded data redundancy: 1/96 objects degraded (1.042%), 42 pgs unclean, 19 pgs degraded (PG_DEGRADED)
2017-09-28 13:13:28.720689 mon.compute1 mon.0 10.200.246.20:6789/0 773 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 13:13:28.736400 mon.compute1 mon.0 10.200.246.20:6789/0 774 : cluster [WRN] Health check update: Reduced data availability: 1 pg inactive, 23 pgs peering (PG_AVAILABILITY)
2017-09-28 13:13:28.736513 mon.compute1 mon.0 10.200.246.20:6789/0 775 : cluster [WRN] Health check update: Degraded data redundancy: 4/96 objects degraded (4.167%), 90 pgs unclean, 67 pgs degraded (PG_DEGRADED)
2017-09-28 13:13:27.514495 osd.3 osd.3 10.200.246.61:6800/1478 9 : cluster [WRN] Monitor daemon marked osd.3 down, but it is still running
2017-09-28 13:13:28.793966 mon.compute1 mon.0 10.200.246.20:6789/0 776 : cluster [INF] osd.3 10.200.246.61:6800/1478 boot
2017-09-28 13:13:31.142824 mon.compute1 mon.0 10.200.246.20:6789/0 785 : cluster [WRN] Health check update: Degraded data redundancy: 4/96 objects degraded (4.167%), 105 pgs unclean, 67 pgs degraded (PG_DEGRADED)
2017-09-28 13:13:32.694168 mon.compute1 mon.0 10.200.246.20:6789/0 786 : cluster [WRN] Health check update: Degraded data redundancy: 2/96 objects degraded (2.083%), 34 pgs unclean, 19 pgs degraded (PG_DEGRADED)
2017-09-28 13:13:32.694244 mon.compute1 mon.0 10.200.246.20:6789/0 787 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 23 pgs peering)
2017-09-28 13:13:34.695141 mon.compute1 mon.0 10.200.246.20:6789/0 788 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/96 objects degraded (2.083%), 34 pgs unclean, 19 pgs degraded)
2017-09-28 13:13:34.695207 mon.compute1 mon.0 10.200.246.20:6789/0 789 : cluster [INF] Cluster is now healthy
2017-09-28 13:24:21.099249 mon.compute1 mon.0 10.200.246.20:6789/0 797 : cluster [INF] osd.2 marked down after no beacon for 300.908807 seconds
2017-09-28 13:24:21.101728 mon.compute1 mon.0 10.200.246.20:6789/0 798 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 13:24:23.716543 mon.compute1 mon.0 10.200.246.20:6789/0 811 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 27 pgs peering (PG_AVAILABILITY)
2017-09-28 13:24:23.716606 mon.compute1 mon.0 10.200.246.20:6789/0 812 : cluster [WRN] Health check failed: Degraded data redundancy: 1/96 objects degraded (1.042%), 45 pgs unclean, 18 pgs degraded (PG_DEGRADED)
2017-09-28 13:24:24.838025 mon.compute1 mon.0 10.200.246.20:6789/0 814 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 13:24:24.934306 mon.compute1 mon.0 10.200.246.20:6789/0 815 : cluster [INF] osd.2 10.200.246.61:6804/1802 boot
2017-09-28 13:24:25.922216 mon.compute1 mon.0 10.200.246.20:6789/0 820 : cluster [WRN] Health check update: Degraded data redundancy: 4/96 objects degraded (4.167%), 100 pgs unclean, 73 pgs degraded (PG_DEGRADED)
2017-09-28 13:24:27.916599 mon.compute1 mon.0 10.200.246.20:6789/0 825 : cluster [WRN] Health check update: Degraded data redundancy: 3/96 objects degraded (3.125%), 42 pgs unclean, 42 pgs degraded (PG_DEGRADED)
2017-09-28 13:24:27.916698 mon.compute1 mon.0 10.200.246.20:6789/0 826 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 27 pgs peering)
2017-09-28 13:24:29.065115 mon.compute1 mon.0 10.200.246.20:6789/0 827 : cluster [WRN] Health check update: Degraded data redundancy: 2/96 objects degraded (2.083%), 25 pgs unclean, 25 pgs degraded (PG_DEGRADED)
2017-09-28 13:24:23.424496 osd.2 osd.2 10.200.246.61:6804/1802 9 : cluster [WRN] Monitor daemon marked osd.2 down, but it is still running
2017-09-28 13:24:31.270774 mon.compute1 mon.0 10.200.246.20:6789/0 828 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/96 objects degraded (2.083%), 25 pgs unclean, 25 pgs degraded)
2017-09-28 13:24:31.270861 mon.compute1 mon.0 10.200.246.20:6789/0 829 : cluster [INF] Cluster is now healthy
2017-09-28 13:28:46.256607 mon.compute1 mon.0 10.200.246.20:6789/0 832 : cluster [INF] osd.7 marked down after no beacon for 300.996321 seconds
2017-09-28 13:28:46.258709 mon.compute1 mon.0 10.200.246.20:6789/0 833 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 13:28:47.662189 mon.compute1 mon.0 10.200.246.20:6789/0 842 : cluster [WRN] Health check failed: Reduced data availability: 27 pgs inactive, 69 pgs peering (PG_AVAILABILITY)
2017-09-28 13:28:47.662253 mon.compute1 mon.0 10.200.246.20:6789/0 843 : cluster [WRN] Health check failed: Degraded data redundancy: 69 pgs unclean (PG_DEGRADED)
2017-09-28 13:28:50.823721 mon.compute1 mon.0 10.200.246.20:6789/0 844 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 13:28:50.891194 mon.compute1 mon.0 10.200.246.20:6789/0 845 : cluster [INF] osd.7 10.200.246.63:6804/1636 boot
2017-09-28 13:28:49.744674 osd.7 osd.7 10.200.246.63:6804/1636 5 : cluster [WRN] Monitor daemon marked osd.7 down, but it is still running
2017-09-28 13:28:53.279389 mon.compute1 mon.0 10.200.246.20:6789/0 858 : cluster [WRN] Health check update: Reduced data availability: 27 pgs inactive, 27 pgs peering (PG_AVAILABILITY)
2017-09-28 13:28:53.279505 mon.compute1 mon.0 10.200.246.20:6789/0 859 : cluster [WRN] Health check update: Degraded data redundancy: 98 pgs unclean (PG_DEGRADED)
2017-09-28 13:28:55.308956 mon.compute1 mon.0 10.200.246.20:6789/0 860 : cluster [WRN] overall HEALTH_WARN Reduced data availability: 27 pgs inactive, 27 pgs peering; Degraded data redundancy: 98 pgs unclean
2017-09-28 13:28:55.393735 mon.compute1 mon.0 10.200.246.20:6789/0 861 : cluster [WRN] Health check update: Degraded data redundancy: 71 pgs unclean (PG_DEGRADED)
2017-09-28 13:28:55.393829 mon.compute1 mon.0 10.200.246.20:6789/0 862 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 27 pgs inactive, 27 pgs peering)
2017-09-28 13:28:57.599394 mon.compute1 mon.0 10.200.246.20:6789/0 863 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71 pgs unclean)
2017-09-28 13:28:57.599505 mon.compute1 mon.0 10.200.246.20:6789/0 864 : cluster [INF] Cluster is now healthy
2017-09-28 13:29:55.309149 mon.compute1 mon.0 10.200.246.20:6789/0 865 : cluster [INF] overall HEALTH_OK
2017-09-28 13:30:06.307602 mon.compute1 mon.0 10.200.246.20:6789/0 866 : cluster [INF] osd.4 marked down after no beacon for 300.211903 seconds
2017-09-28 13:30:06.307707 mon.compute1 mon.0 10.200.246.20:6789/0 867 : cluster [INF] osd.5 marked down after no beacon for 300.230919 seconds
2017-09-28 13:30:06.309739 mon.compute1 mon.0 10.200.246.20:6789/0 868 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)
2017-09-28 13:30:06.309802 mon.compute1 mon.0 10.200.246.20:6789/0 869 : cluster [WRN] Health check failed: 1 host (2 osds) down (OSD_HOST_DOWN)
2017-09-28 13:30:06.929202 osd.5 osd.5 10.200.246.62:6804/1867 9 : cluster [WRN] Monitor daemon marked osd.5 down, but it is still running
2017-09-28 13:30:07.684514 mon.compute1 mon.0 10.200.246.20:6789/0 878 : cluster [WRN] Health check failed: Reduced data availability: 22 pgs inactive, 73 pgs peering (PG_AVAILABILITY)
2017-09-28 13:30:07.684573 mon.compute1 mon.0 10.200.246.20:6789/0 879 : cluster [WRN] Health check failed: Degraded data redundancy: 73 pgs unclean (PG_DEGRADED)
2017-09-28 13:30:08.857672 mon.compute1 mon.0 10.200.246.20:6789/0 880 : cluster [INF] Health check cleared: OSD_DOWN (was: 2 osds down)
2017-09-28 13:30:08.857735 mon.compute1 mon.0 10.200.246.20:6789/0 881 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
2017-09-28 13:30:08.948078 mon.compute1 mon.0 10.200.246.20:6789/0 882 : cluster [INF] osd.5 10.200.246.62:6804/1867 boot
2017-09-28 13:30:08.948207 mon.compute1 mon.0 10.200.246.20:6789/0 883 : cluster [INF] osd.4 10.200.246.62:6800/1478 boot
2017-09-28 13:30:06.896329 osd.4 osd.4 10.200.246.62:6800/1478 9 : cluster [WRN] Monitor daemon marked osd.4 down, but it is still running
2017-09-28 13:30:10.136109 mon.compute1 mon.0 10.200.246.20:6789/0 892 : cluster [WRN] Health check update: Degraded data redundancy: 4/96 objects degraded (4.167%), 169 pgs unclean, 58 pgs degraded (PG_DEGRADED)
2017-09-28 13:30:13.297768 mon.compute1 mon.0 10.200.246.20:6789/0 897 : cluster [WRN] Health check update: Degraded data redundancy: 3/96 objects degraded (3.125%), 81 pgs unclean, 43 pgs degraded (PG_DEGRADED)
2017-09-28 13:30:13.297842 mon.compute1 mon.0 10.200.246.20:6789/0 898 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 22 pgs inactive, 73 pgs peering)
2017-09-28 13:30:16.050286 mon.compute1 mon.0 10.200.246.20:6789/0 899 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/96 objects degraded (3.125%), 81 pgs unclean, 43 pgs degraded)
2017-09-28 13:30:16.050373 mon.compute1 mon.0 10.200.246.20:6789/0 900 : cluster [INF] Cluster is now healthy
2017-09-28 13:32:41.397415 mon.compute1 mon.0 10.200.246.20:6789/0 903 : cluster [INF] osd.0 marked down after no beacon for 300.845121 seconds
2017-09-28 13:32:41.397522 mon.compute1 mon.0 10.200.246.20:6789/0 904 : cluster [INF] osd.1 marked down after no beacon for 300.647721 seconds
2017-09-28 13:32:41.399577 mon.compute1 mon.0 10.200.246.20:6789/0 905 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)
2017-09-28 13:32:41.399652 mon.compute1 mon.0 10.200.246.20:6789/0 906 : cluster [WRN] Health check failed: 1 host (2 osds) down (OSD_HOST_DOWN)
2017-09-28 13:32:44.045028 mon.compute1 mon.0 10.200.246.20:6789/0 919 : cluster [WRN] Health check update: 1 osds down (OSD_DOWN)
2017-09-28 13:32:44.045095 mon.compute1 mon.0 10.200.246.20:6789/0 920 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
2017-09-28 13:32:44.192323 mon.compute1 mon.0 10.200.246.20:6789/0 921 : cluster [INF] osd.1 10.200.246.20:6804/1975 boot
2017-09-28 13:32:45.183262 mon.compute1 mon.0 10.200.246.20:6789/0 926 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 13:32:45.183333 mon.compute1 mon.0 10.200.246.20:6789/0 927 : cluster [INF] Cluster is now healthy
2017-09-28 13:32:45.284220 mon.compute1 mon.0 10.200.246.20:6789/0 928 : cluster [INF] osd.0 10.200.246.20:6800/1732 boot
2017-09-28 13:32:46.283308 mon.compute1 mon.0 10.200.246.20:6789/0 933 : cluster [WRN] Health check failed: Degraded data redundancy: 4/96 objects degraded (4.167%), 66 pgs unclean, 66 pgs degraded (PG_DEGRADED)
2017-09-28 13:32:48.294764 mon.compute1 mon.0 10.200.246.20:6789/0 938 : cluster [WRN] Health check update: Degraded data redundancy: 3/96 objects degraded (3.125%), 99 pgs unclean, 45 pgs degraded (PG_DEGRADED)
2017-09-28 13:32:49.409063 mon.compute1 mon.0 10.200.246.20:6789/0 939 : cluster [WRN] Health check update: Degraded data redundancy: 54 pgs unclean (PG_DEGRADED)
2017-09-28 13:32:43.410142 osd.0 osd.0 10.200.246.20:6800/1732 9 : cluster [WRN] Monitor daemon marked osd.0 down, but it is still running
2017-09-28 13:32:43.423407 osd.1 osd.1 10.200.246.20:6804/1975 9 : cluster [WRN] Monitor daemon marked osd.1 down, but it is still running
2017-09-28 13:32:53.393843 mon.compute1 mon.0 10.200.246.20:6789/0 940 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 54 pgs unclean)
2017-09-28 13:32:53.393936 mon.compute1 mon.0 10.200.246.20:6789/0 941 : cluster [INF] Cluster is now healthy
2017-09-28 13:38:46.618951 mon.compute1 mon.0 10.200.246.20:6789/0 945 : cluster [INF] osd.6 marked down after no beacon for 300.588308 seconds
2017-09-28 13:38:46.621104 mon.compute1 mon.0 10.200.246.20:6789/0 946 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 13:38:48.110109 mon.compute1 mon.0 10.200.246.20:6789/0 955 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 13:38:48.110175 mon.compute1 mon.0 10.200.246.20:6789/0 956 : cluster [INF] Cluster is now healthy
2017-09-28 13:38:48.120027 mon.compute1 mon.0 10.200.246.20:6789/0 957 : cluster [WRN] Health check failed: Reduced data availability: 8 pgs inactive, 19 pgs peering (PG_AVAILABILITY)
2017-09-28 13:38:48.120106 mon.compute1 mon.0 10.200.246.20:6789/0 958 : cluster [WRN] Health check failed: Degraded data redundancy: 19 pgs unclean (PG_DEGRADED)
2017-09-28 13:38:48.172114 mon.compute1 mon.0 10.200.246.20:6789/0 959 : cluster [INF] osd.6 10.200.246.63:6800/1410 boot
2017-09-28 13:38:50.463050 mon.compute1 mon.0 10.200.246.20:6789/0 972 : cluster [WRN] Health check update: Reduced data availability: 47 pgs peering (PG_AVAILABILITY)
2017-09-28 13:38:50.463136 mon.compute1 mon.0 10.200.246.20:6789/0 973 : cluster [WRN] Health check update: Degraded data redundancy: 61 pgs unclean (PG_DEGRADED)
2017-09-28 13:38:47.788173 osd.6 osd.6 10.200.246.63:6800/1410 7 : cluster [WRN] Monitor daemon marked osd.6 down, but it is still running
2017-09-28 13:38:53.617347 mon.compute1 mon.0 10.200.246.20:6789/0 974 : cluster [WRN] Health check update: Reduced data availability: 36 pgs peering (PG_AVAILABILITY)
2017-09-28 13:38:53.617419 mon.compute1 mon.0 10.200.246.20:6789/0 975 : cluster [WRN] Health check update: Degraded data redundancy: 50 pgs unclean (PG_DEGRADED)
2017-09-28 13:38:55.310229 mon.compute1 mon.0 10.200.246.20:6789/0 976 : cluster [WRN] overall HEALTH_WARN Reduced data availability: 36 pgs peering; Degraded data redundancy: 50 pgs unclean
2017-09-28 13:38:55.731651 mon.compute1 mon.0 10.200.246.20:6789/0 977 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 36 pgs peering)
2017-09-28 13:38:55.731712 mon.compute1 mon.0 10.200.246.20:6789/0 978 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50 pgs unclean)
2017-09-28 13:38:55.731760 mon.compute1 mon.0 10.200.246.20:6789/0 979 : cluster [INF] Cluster is now healthy
2017-09-28 13:39:55.310455 mon.compute1 mon.0 10.200.246.20:6789/0 981 : cluster [INF] overall HEALTH_OK
2017-09-28 13:43:31.797566 mon.compute1 mon.0 10.200.246.20:6789/0 985 : cluster [INF] osd.3 marked down after no beacon for 300.364452 seconds
2017-09-28 13:43:31.799605 mon.compute1 mon.0 10.200.246.20:6789/0 986 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 13:43:34.382041 mon.compute1 mon.0 10.200.246.20:6789/0 999 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 13:43:34.382102 mon.compute1 mon.0 10.200.246.20:6789/0 1000 : cluster [INF] Cluster is now healthy
2017-09-28 13:43:34.382245 mon.compute1 mon.0 10.200.246.20:6789/0 1001 : cluster [WRN] Health check failed: Degraded data redundancy: 2/96 objects degraded (2.083%), 32 pgs unclean, 32 pgs degraded (PG_DEGRADED)
2017-09-28 13:43:34.563937 mon.compute1 mon.0 10.200.246.20:6789/0 1002 : cluster [INF] osd.3 10.200.246.61:6800/1478 boot
2017-09-28 13:43:36.517730 mon.compute1 mon.0 10.200.246.20:6789/0 1011 : cluster [WRN] Health check update: Degraded data redundancy: 7/96 objects degraded (7.292%), 58 pgs unclean, 58 pgs degraded (PG_DEGRADED)
2017-09-28 13:43:37.783991 mon.compute1 mon.0 10.200.246.20:6789/0 1012 : cluster [WRN] Health check update: Degraded data redundancy: 6/96 objects degraded (6.250%), 34 pgs unclean, 34 pgs degraded (PG_DEGRADED)
2017-09-28 13:43:33.868266 osd.3 osd.3 10.200.246.61:6800/1478 11 : cluster [WRN] Monitor daemon marked osd.3 down, but it is still running
2017-09-28 13:43:39.784864 mon.compute1 mon.0 10.200.246.20:6789/0 1013 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/96 objects degraded (6.250%), 34 pgs unclean, 34 pgs degraded)
2017-09-28 13:43:39.784938 mon.compute1 mon.0 10.200.246.20:6789/0 1014 : cluster [INF] Cluster is now healthy
2017-09-28 13:45:11.853593 mon.compute1 mon.0 10.200.246.20:6789/0 1016 : cluster [INF] osd.4 marked down after no beacon for 300.667398 seconds
2017-09-28 13:45:11.853673 mon.compute1 mon.0 10.200.246.20:6789/0 1017 : cluster [INF] osd.5 marked down after no beacon for 300.666860 seconds
2017-09-28 13:45:11.855545 mon.compute1 mon.0 10.200.246.20:6789/0 1018 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)
2017-09-28 13:45:11.855619 mon.compute1 mon.0 10.200.246.20:6789/0 1019 : cluster [WRN] Health check failed: 1 host (2 osds) down (OSD_HOST_DOWN)
2017-09-28 13:45:13.426722 mon.compute1 mon.0 10.200.246.20:6789/0 1028 : cluster [INF] Health check cleared: OSD_DOWN (was: 2 osds down)
2017-09-28 13:45:13.426795 mon.compute1 mon.0 10.200.246.20:6789/0 1029 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
2017-09-28 13:45:13.426839 mon.compute1 mon.0 10.200.246.20:6789/0 1030 : cluster [INF] Cluster is now healthy
2017-09-28 13:45:13.494419 mon.compute1 mon.0 10.200.246.20:6789/0 1031 : cluster [INF] osd.5 10.200.246.62:6804/1867 boot
2017-09-28 13:45:13.494655 mon.compute1 mon.0 10.200.246.20:6789/0 1032 : cluster [INF] osd.4 10.200.246.62:6800/1478 boot
2017-09-28 13:45:14.665106 mon.compute1 mon.0 10.200.246.20:6789/0 1041 : cluster [WRN] Health check failed: Degraded data redundancy: 46 pgs unclean, 9 pgs degraded (PG_DEGRADED)
2017-09-28 13:45:17.845820 mon.compute1 mon.0 10.200.246.20:6789/0 1042 : cluster [WRN] Health check update: Degraded data redundancy: 43 pgs unclean, 6 pgs degraded (PG_DEGRADED)
2017-09-28 13:45:12.039382 osd.5 osd.5 10.200.246.62:6804/1867 11 : cluster [WRN] Monitor daemon marked osd.5 down, but it is still running
2017-09-28 13:45:20.053672 mon.compute1 mon.0 10.200.246.20:6789/0 1043 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43 pgs unclean, 6 pgs degraded)
2017-09-28 13:45:20.053764 mon.compute1 mon.0 10.200.246.20:6789/0 1044 : cluster [INF] Cluster is now healthy
2017-09-28 13:45:12.434282 osd.4 osd.4 10.200.246.62:6800/1478 11 : cluster [WRN] Monitor daemon marked osd.4 down, but it is still running
2017-09-28 13:54:27.190519 mon.compute1 mon.0 10.200.246.20:6789/0 1047 : cluster [INF] osd.2 marked down after no beacon for 300.815535 seconds
2017-09-28 13:54:27.192656 mon.compute1 mon.0 10.200.246.20:6789/0 1048 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 13:54:30.361769 mon.compute1 mon.0 10.200.246.20:6789/0 1057 : cluster [WRN] Health check failed: Degraded data redundancy: 3/96 objects degraded (3.125%), 55 pgs unclean, 55 pgs degraded (PG_DEGRADED)
2017-09-28 13:54:31.677028 mon.compute1 mon.0 10.200.246.20:6789/0 1058 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 13:54:31.750823 mon.compute1 mon.0 10.200.246.20:6789/0 1059 : cluster [INF] osd.2 10.200.246.61:6804/1802 boot
2017-09-28 13:54:33.018953 mon.compute1 mon.0 10.200.246.20:6789/0 1068 : cluster [WRN] Health check update: Degraded data redundancy: 7/96 objects degraded (7.292%), 141 pgs unclean, 132 pgs degraded (PG_DEGRADED)
2017-09-28 13:54:35.047540 mon.compute1 mon.0 10.200.246.20:6789/0 1073 : cluster [WRN] Health check update: Degraded data redundancy: 6/96 objects degraded (6.250%), 124 pgs unclean, 115 pgs degraded (PG_DEGRADED)
2017-09-28 13:54:36.213215 mon.compute1 mon.0 10.200.246.20:6789/0 1074 : cluster [WRN] Health check update: Degraded data redundancy: 4/96 objects degraded (4.167%), 99 pgs unclean, 90 pgs degraded (PG_DEGRADED)
2017-09-28 13:54:38.282028 mon.compute1 mon.0 10.200.246.20:6789/0 1075 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/96 objects degraded (4.167%), 99 pgs unclean, 90 pgs degraded)
2017-09-28 13:54:38.282125 mon.compute1 mon.0 10.200.246.20:6789/0 1076 : cluster [INF] Cluster is now healthy
2017-09-28 13:54:30.786520 osd.2 osd.2 10.200.246.61:6804/1802 11 : cluster [WRN] Monitor daemon marked osd.2 down, but it is still running
2017-09-28 13:58:52.358878 mon.compute1 mon.0 10.200.246.20:6789/0 1078 : cluster [INF] osd.7 marked down after no beacon for 300.886959 seconds
2017-09-28 13:58:52.361250 mon.compute1 mon.0 10.200.246.20:6789/0 1079 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 13:58:54.863359 mon.compute1 mon.0 10.200.246.20:6789/0 1088 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 13:58:54.863458 mon.compute1 mon.0 10.200.246.20:6789/0 1089 : cluster [INF] Cluster is now healthy
2017-09-28 13:58:54.925158 mon.compute1 mon.0 10.200.246.20:6789/0 1090 : cluster [INF] osd.7 10.200.246.63:6804/1636 boot
2017-09-28 13:58:58.439027 mon.compute1 mon.0 10.200.246.20:6789/0 1103 : cluster [WRN] Health check failed: Degraded data redundancy: 39 pgs unclean (PG_DEGRADED)
2017-09-28 13:59:02.352728 mon.compute1 mon.0 10.200.246.20:6789/0 1104 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39 pgs unclean)
2017-09-28 13:59:02.352793 mon.compute1 mon.0 10.200.246.20:6789/0 1105 : cluster [INF] Cluster is now healthy
2017-09-28 13:58:53.424294 osd.7 osd.7 10.200.246.63:6804/1636 7 : cluster [WRN] Monitor daemon marked osd.7 down, but it is still running
2017-09-28 14:00:00.000119 mon.compute1 mon.0 10.200.246.20:6789/0 1106 : cluster [INF] overall HEALTH_OK
2017-09-28 14:02:47.505238 mon.compute1 mon.0 10.200.246.20:6789/0 1110 : cluster [INF] osd.0 marked down after no beacon for 300.743559 seconds
2017-09-28 14:02:47.505319 mon.compute1 mon.0 10.200.246.20:6789/0 1111 : cluster [INF] osd.1 marked down after no beacon for 300.550071 seconds
2017-09-28 14:02:47.507516 mon.compute1 mon.0 10.200.246.20:6789/0 1112 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)
2017-09-28 14:02:47.507601 mon.compute1 mon.0 10.200.246.20:6789/0 1113 : cluster [WRN] Health check failed: 1 host (2 osds) down (OSD_HOST_DOWN)
2017-09-28 14:02:48.203386 osd.1 osd.1 10.200.246.20:6804/1975 11 : cluster [WRN] Monitor daemon marked osd.1 down, but it is still running
2017-09-28 14:02:48.216619 osd.0 osd.0 10.200.246.20:6800/1732 11 : cluster [WRN] Monitor daemon marked osd.0 down, but it is still running
2017-09-28 14:02:49.255164 mon.compute1 mon.0 10.200.246.20:6789/0 1122 : cluster [INF] Health check cleared: OSD_DOWN (was: 2 osds down)
2017-09-28 14:02:49.255226 mon.compute1 mon.0 10.200.246.20:6789/0 1123 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
2017-09-28 14:02:49.255277 mon.compute1 mon.0 10.200.246.20:6789/0 1124 : cluster [INF] Cluster is now healthy
2017-09-28 14:02:49.340090 mon.compute1 mon.0 10.200.246.20:6789/0 1125 : cluster [INF] osd.0 10.200.246.20:6800/1732 boot
2017-09-28 14:02:49.340428 mon.compute1 mon.0 10.200.246.20:6789/0 1126 : cluster [INF] osd.1 10.200.246.20:6804/1975 boot
2017-09-28 14:02:51.381019 mon.compute1 mon.0 10.200.246.20:6789/0 1135 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 22 pgs peering (PG_AVAILABILITY)
2017-09-28 14:02:51.381129 mon.compute1 mon.0 10.200.246.20:6789/0 1136 : cluster [WRN] Health check failed: Degraded data redundancy: 3/96 objects degraded (3.125%), 57 pgs unclean, 35 pgs degraded (PG_DEGRADED)
2017-09-28 14:02:52.545147 mon.compute1 mon.0 10.200.246.20:6789/0 1137 : cluster [WRN] Health check update: Reduced data availability: 22 pgs peering (PG_AVAILABILITY)
2017-09-28 14:02:52.545242 mon.compute1 mon.0 10.200.246.20:6789/0 1138 : cluster [WRN] Health check update: Degraded data redundancy: 2/96 objects degraded (2.083%), 45 pgs unclean, 23 pgs degraded (PG_DEGRADED)
2017-09-28 14:02:54.636564 mon.compute1 mon.0 10.200.246.20:6789/0 1139 : cluster [WRN] Health check update: Degraded data redundancy: 22 pgs unclean (PG_DEGRADED)
2017-09-28 14:02:55.312869 mon.compute1 mon.0 10.200.246.20:6789/0 1140 : cluster [WRN] overall HEALTH_WARN Reduced data availability: 22 pgs peering; Degraded data redundancy: 22 pgs unclean
2017-09-28 14:02:56.836560 mon.compute1 mon.0 10.200.246.20:6789/0 1141 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 22 pgs peering)
2017-09-28 14:02:56.836632 mon.compute1 mon.0 10.200.246.20:6789/0 1142 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22 pgs unclean)
2017-09-28 14:02:56.836690 mon.compute1 mon.0 10.200.246.20:6789/0 1143 : cluster [INF] Cluster is now healthy
2017-09-28 14:03:55.313074 mon.compute1 mon.0 10.200.246.20:6789/0 1145 : cluster [INF] overall HEALTH_OK
2017-09-28 14:05:17.595287 mon.compute1 mon.0 10.200.246.20:6789/0 1147 : cluster [INF] osd.5 marked down after no beacon for 300.280493 seconds
2017-09-28 14:05:17.597420 mon.compute1 mon.0 10.200.246.20:6789/0 1148 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:05:19.156819 mon.compute1 mon.0 10.200.246.20:6789/0 1157 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:05:19.156882 mon.compute1 mon.0 10.200.246.20:6789/0 1158 : cluster [INF] Cluster is now healthy
2017-09-28 14:05:19.310255 mon.compute1 mon.0 10.200.246.20:6789/0 1159 : cluster [INF] osd.5 10.200.246.62:6804/1867 boot
2017-09-28 14:05:21.349495 mon.compute1 mon.0 10.200.246.20:6789/0 1168 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)
2017-09-28 14:05:21.349590 mon.compute1 mon.0 10.200.246.20:6789/0 1169 : cluster [WRN] Health check failed: Degraded data redundancy: 2/96 objects degraded (2.083%), 51 pgs unclean, 30 pgs degraded (PG_DEGRADED)
2017-09-28 14:05:22.583867 mon.compute1 mon.0 10.200.246.20:6789/0 1170 : cluster [WRN] Health check update: Degraded data redundancy: 2/96 objects degraded (2.083%), 46 pgs unclean, 25 pgs degraded (PG_DEGRADED)
2017-09-28 14:05:22.583933 mon.compute1 mon.0 10.200.246.20:6789/0 1171 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive)
2017-09-28 14:05:24.584723 mon.compute1 mon.0 10.200.246.20:6789/0 1173 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/96 objects degraded (2.083%), 46 pgs unclean, 25 pgs degraded)
2017-09-28 14:05:24.584809 mon.compute1 mon.0 10.200.246.20:6789/0 1174 : cluster [INF] Cluster is now healthy
2017-09-28 14:05:18.432581 osd.5 osd.5 10.200.246.62:6804/1867 13 : cluster [WRN] Monitor daemon marked osd.5 down, but it is still running
2017-09-28 14:08:52.724087 mon.compute1 mon.0 10.200.246.20:6789/0 1178 : cluster [INF] osd.6 marked down after no beacon for 300.481276 seconds
2017-09-28 14:08:52.725954 mon.compute1 mon.0 10.200.246.20:6789/0 1179 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:08:55.313707 mon.compute1 mon.0 10.200.246.20:6789/0 1192 : cluster [WRN] overall HEALTH_WARN 1 osds down
2017-09-28 14:08:55.337374 mon.compute1 mon.0 10.200.246.20:6789/0 1193 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:08:55.337477 mon.compute1 mon.0 10.200.246.20:6789/0 1194 : cluster [INF] Cluster is now healthy
2017-09-28 14:08:55.461789 mon.compute1 mon.0 10.200.246.20:6789/0 1195 : cluster [INF] osd.6 10.200.246.63:6800/1410 boot
2017-09-28 14:08:54.402784 osd.6 osd.6 10.200.246.63:6800/1410 9 : cluster [WRN] Monitor daemon marked osd.6 down, but it is still running
2017-09-28 14:09:55.313908 mon.compute1 mon.0 10.200.246.20:6789/0 1204 : cluster [INF] overall HEALTH_OK
2017-09-28 14:13:37.887190 mon.compute1 mon.0 10.200.246.20:6789/0 1207 : cluster [INF] osd.3 marked down after no beacon for 300.272058 seconds
2017-09-28 14:13:37.889288 mon.compute1 mon.0 10.200.246.20:6789/0 1208 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:13:39.321323 osd.3 osd.3 10.200.246.61:6800/1478 13 : cluster [WRN] Monitor daemon marked osd.3 down, but it is still running
2017-09-28 14:13:40.688570 mon.compute1 mon.0 10.200.246.20:6789/0 1221 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:13:40.688644 mon.compute1 mon.0 10.200.246.20:6789/0 1222 : cluster [INF] Cluster is now healthy
2017-09-28 14:13:40.767615 mon.compute1 mon.0 10.200.246.20:6789/0 1223 : cluster [INF] osd.3 10.200.246.61:6800/1478 boot
2017-09-28 14:13:41.755613 mon.compute1 mon.0 10.200.246.20:6789/0 1232 : cluster [WRN] Health check failed: Reduced data availability: 17 pgs inactive, 26 pgs peering (PG_AVAILABILITY)
2017-09-28 14:13:41.755682 mon.compute1 mon.0 10.200.246.20:6789/0 1233 : cluster [WRN] Health check failed: Degraded data redundancy: 1/96 objects degraded (1.042%), 58 pgs unclean, 32 pgs degraded (PG_DEGRADED)
2017-09-28 14:13:43.709967 mon.compute1 mon.0 10.200.246.20:6789/0 1238 : cluster [WRN] Health check update: Reduced data availability: 11 pgs inactive, 15 pgs peering (PG_AVAILABILITY)
2017-09-28 14:13:43.710037 mon.compute1 mon.0 10.200.246.20:6789/0 1239 : cluster [WRN] Health check update: Degraded data redundancy: 1/96 objects degraded (1.042%), 81 pgs unclean, 19 pgs degraded (PG_DEGRADED)
2017-09-28 14:13:44.858551 mon.compute1 mon.0 10.200.246.20:6789/0 1240 : cluster [WRN] Health check update: Reduced data availability: 11 pgs inactive (PG_AVAILABILITY)
2017-09-28 14:13:44.858612 mon.compute1 mon.0 10.200.246.20:6789/0 1241 : cluster [WRN] Health check update: Degraded data redundancy: 47 pgs unclean (PG_DEGRADED)
2017-09-28 14:13:46.883345 mon.compute1 mon.0 10.200.246.20:6789/0 1242 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive)
2017-09-28 14:13:46.883474 mon.compute1 mon.0 10.200.246.20:6789/0 1243 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 47 pgs unclean)
2017-09-28 14:13:46.883532 mon.compute1 mon.0 10.200.246.20:6789/0 1244 : cluster [INF] Cluster is now healthy
2017-09-28 14:15:17.949046 mon.compute1 mon.0 10.200.246.20:6789/0 1245 : cluster [INF] osd.4 marked down after no beacon for 300.581105 seconds
2017-09-28 14:15:17.951155 mon.compute1 mon.0 10.200.246.20:6789/0 1246 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:15:19.354588 mon.compute1 mon.0 10.200.246.20:6789/0 1255 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs inactive, 28 pgs peering (PG_AVAILABILITY)
2017-09-28 14:15:19.354642 mon.compute1 mon.0 10.200.246.20:6789/0 1256 : cluster [WRN] Health check failed: Degraded data redundancy: 28 pgs unclean (PG_DEGRADED)
2017-09-28 14:15:20.272690 osd.4 osd.4 10.200.246.62:6800/1478 13 : cluster [WRN] Monitor daemon marked osd.4 down, but it is still running
2017-09-28 14:15:21.504886 mon.compute1 mon.0 10.200.246.20:6789/0 1257 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:15:21.520373 mon.compute1 mon.0 10.200.246.20:6789/0 1258 : cluster [WRN] Health check update: Degraded data redundancy: 5/96 objects degraded (5.208%), 59 pgs unclean, 31 pgs degraded (PG_DEGRADED)
2017-09-28 14:15:21.595097 mon.compute1 mon.0 10.200.246.20:6789/0 1259 : cluster [INF] osd.4 10.200.246.62:6800/1478 boot
2017-09-28 14:15:23.874690 mon.compute1 mon.0 10.200.246.20:6789/0 1272 : cluster [WRN] Health check update: Reduced data availability: 79 pgs peering (PG_AVAILABILITY)
2017-09-28 14:15:23.874785 mon.compute1 mon.0 10.200.246.20:6789/0 1273 : cluster [WRN] Health check update: Degraded data redundancy: 5/96 objects degraded (5.208%), 155 pgs unclean, 17 pgs degraded (PG_DEGRADED)
2017-09-28 14:15:25.040459 mon.compute1 mon.0 10.200.246.20:6789/0 1274 : cluster [WRN] Health check update: Reduced data availability: 61 pgs peering (PG_AVAILABILITY)
2017-09-28 14:15:25.040525 mon.compute1 mon.0 10.200.246.20:6789/0 1275 : cluster [WRN] Health check update: Degraded data redundancy: 120 pgs unclean (PG_DEGRADED)
2017-09-28 14:15:27.137542 mon.compute1 mon.0 10.200.246.20:6789/0 1276 : cluster [WRN] Health check update: Degraded data redundancy: 20 pgs unclean (PG_DEGRADED)
2017-09-28 14:15:27.137614 mon.compute1 mon.0 10.200.246.20:6789/0 1277 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 61 pgs peering)
2017-09-28 14:15:28.945698 mon.compute1 mon.0 10.200.246.20:6789/0 1278 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20 pgs unclean)
2017-09-28 14:15:28.945770 mon.compute1 mon.0 10.200.246.20:6789/0 1279 : cluster [INF] Cluster is now healthy
2017-09-28 14:24:33.285980 mon.compute1 mon.0 10.200.246.20:6789/0 1285 : cluster [INF] osd.2 marked down after no beacon for 300.724317 seconds
2017-09-28 14:24:33.287990 mon.compute1 mon.0 10.200.246.20:6789/0 1286 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:24:34.864766 mon.compute1 mon.0 10.200.246.20:6789/0 1295 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:24:34.864875 mon.compute1 mon.0 10.200.246.20:6789/0 1296 : cluster [INF] Cluster is now healthy
2017-09-28 14:24:34.937712 mon.compute1 mon.0 10.200.246.20:6789/0 1297 : cluster [INF] osd.2 10.200.246.61:6804/1802 boot
2017-09-28 14:24:36.159855 mon.compute1 mon.0 10.200.246.20:6789/0 1306 : cluster [WRN] Health check failed: Reduced data availability: 13 pgs inactive, 55 pgs peering (PG_AVAILABILITY)
2017-09-28 14:24:36.159953 mon.compute1 mon.0 10.200.246.20:6789/0 1307 : cluster [WRN] Health check failed: Degraded data redundancy: 55 pgs unclean (PG_DEGRADED)
2017-09-28 14:24:37.777018 mon.compute1 mon.0 10.200.246.20:6789/0 1308 : cluster [WRN] Health check update: Reduced data availability: 42 pgs peering (PG_AVAILABILITY)
2017-09-28 14:24:37.777085 mon.compute1 mon.0 10.200.246.20:6789/0 1309 : cluster [WRN] Health check update: Degraded data redundancy: 42 pgs unclean (PG_DEGRADED)
2017-09-28 14:24:39.276782 mon.compute1 mon.0 10.200.246.20:6789/0 1310 : cluster [WRN] Health check update: Reduced data availability: 25 pgs peering (PG_AVAILABILITY)
2017-09-28 14:24:39.276883 mon.compute1 mon.0 10.200.246.20:6789/0 1311 : cluster [WRN] Health check update: Degraded data redundancy: 25 pgs unclean (PG_DEGRADED)
2017-09-28 14:24:34.245915 osd.2 osd.2 10.200.246.61:6804/1802 13 : cluster [WRN] Monitor daemon marked osd.2 down, but it is still running
2017-09-28 14:24:41.508415 mon.compute1 mon.0 10.200.246.20:6789/0 1312 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 25 pgs peering)
2017-09-28 14:24:41.508504 mon.compute1 mon.0 10.200.246.20:6789/0 1313 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25 pgs unclean)
2017-09-28 14:24:41.508556 mon.compute1 mon.0 10.200.246.20:6789/0 1314 : cluster [INF] Cluster is now healthy
2017-09-28 14:28:58.448556 mon.compute1 mon.0 10.200.246.20:6789/0 1318 : cluster [INF] osd.7 marked down after no beacon for 300.763779 seconds
2017-09-28 14:28:58.450504 mon.compute1 mon.0 10.200.246.20:6789/0 1319 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:29:00.564650 mon.compute1 mon.0 10.200.246.20:6789/0 1328 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:29:00.564740 mon.compute1 mon.0 10.200.246.20:6789/0 1329 : cluster [INF] Cluster is now healthy
2017-09-28 14:29:00.648871 mon.compute1 mon.0 10.200.246.20:6789/0 1330 : cluster [INF] osd.7 10.200.246.63:6804/1636 boot
2017-09-28 14:28:58.993970 osd.7 osd.7 10.200.246.63:6804/1636 9 : cluster [WRN] Monitor daemon marked osd.7 down, but it is still running
2017-09-28 14:29:03.768417 mon.compute1 mon.0 10.200.246.20:6789/0 1339 : cluster [WRN] Health check failed: Degraded data redundancy: 12 pgs unclean (PG_DEGRADED)
2017-09-28 14:29:07.442640 mon.compute1 mon.0 10.200.246.20:6789/0 1340 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12 pgs unclean)
2017-09-28 14:29:07.442730 mon.compute1 mon.0 10.200.246.20:6789/0 1341 : cluster [INF] Cluster is now healthy
2017-09-28 14:30:23.510323 mon.compute1 mon.0 10.200.246.20:6789/0 1343 : cluster [INF] osd.5 marked down after no beacon for 301.046881 seconds
2017-09-28 14:30:23.512255 mon.compute1 mon.0 10.200.246.20:6789/0 1344 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:30:26.058986 mon.compute1 mon.0 10.200.246.20:6789/0 1353 : cluster [WRN] Health check failed: Reduced data availability: 17 pgs inactive, 64 pgs peering (PG_AVAILABILITY)
2017-09-28 14:30:26.059059 mon.compute1 mon.0 10.200.246.20:6789/0 1354 : cluster [WRN] Health check failed: Degraded data redundancy: 64 pgs unclean (PG_DEGRADED)
2017-09-28 14:30:27.209640 mon.compute1 mon.0 10.200.246.20:6789/0 1355 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:30:25.344785 osd.5 osd.5 10.200.246.62:6804/1867 15 : cluster [WRN] Monitor daemon marked osd.5 down, but it is still running
2017-09-28 14:30:27.299572 mon.compute1 mon.0 10.200.246.20:6789/0 1356 : cluster [INF] osd.5 10.200.246.62:6804/1867 boot
2017-09-28 14:30:28.447674 mon.compute1 mon.0 10.200.246.20:6789/0 1365 : cluster [WRN] Health check update: Degraded data redundancy: 5/96 objects degraded (5.208%), 147 pgs unclean, 83 pgs degraded (PG_DEGRADED)
2017-09-28 14:30:29.681904 mon.compute1 mon.0 10.200.246.20:6789/0 1370 : cluster [WRN] Health check update: Reduced data availability: 17 pgs inactive, 39 pgs peering (PG_AVAILABILITY)
2017-09-28 14:30:29.681966 mon.compute1 mon.0 10.200.246.20:6789/0 1371 : cluster [WRN] Health check update: Degraded data redundancy: 5/96 objects degraded (5.208%), 133 pgs unclean, 83 pgs degraded (PG_DEGRADED)
2017-09-28 14:30:31.636203 mon.compute1 mon.0 10.200.246.20:6789/0 1372 : cluster [WRN] Health check update: Reduced data availability: 17 pgs inactive, 18 pgs peering (PG_AVAILABILITY)
2017-09-28 14:30:31.636298 mon.compute1 mon.0 10.200.246.20:6789/0 1373 : cluster [WRN] Health check update: Degraded data redundancy: 5/96 objects degraded (5.208%), 112 pgs unclean, 83 pgs degraded (PG_DEGRADED)
2017-09-28 14:30:33.496900 mon.compute1 mon.0 10.200.246.20:6789/0 1374 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 17 pgs inactive, 18 pgs peering)
2017-09-28 14:30:33.496996 mon.compute1 mon.0 10.200.246.20:6789/0 1375 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/96 objects degraded (5.208%), 112 pgs unclean, 83 pgs degraded)
2017-09-28 14:30:33.497056 mon.compute1 mon.0 10.200.246.20:6789/0 1376 : cluster [INF] Cluster is now healthy
2017-09-28 14:32:53.601044 mon.compute1 mon.0 10.200.246.20:6789/0 1379 : cluster [INF] osd.0 marked down after no beacon for 300.626970 seconds
2017-09-28 14:32:53.601126 mon.compute1 mon.0 10.200.246.20:6789/0 1380 : cluster [INF] osd.1 marked down after no beacon for 300.438670 seconds
2017-09-28 14:32:53.603597 mon.compute1 mon.0 10.200.246.20:6789/0 1381 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)
2017-09-28 14:32:53.603677 mon.compute1 mon.0 10.200.246.20:6789/0 1382 : cluster [WRN] Health check failed: 1 host (2 osds) down (OSD_HOST_DOWN)
2017-09-28 14:32:55.163104 mon.compute1 mon.0 10.200.246.20:6789/0 1391 : cluster [INF] Health check cleared: OSD_DOWN (was: 2 osds down)
2017-09-28 14:32:55.163194 mon.compute1 mon.0 10.200.246.20:6789/0 1392 : cluster [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
2017-09-28 14:32:55.163249 mon.compute1 mon.0 10.200.246.20:6789/0 1393 : cluster [INF] Cluster is now healthy
2017-09-28 14:32:55.242033 mon.compute1 mon.0 10.200.246.20:6789/0 1394 : cluster [INF] osd.0 10.200.246.20:6800/1732 boot
2017-09-28 14:32:55.242235 mon.compute1 mon.0 10.200.246.20:6789/0 1395 : cluster [INF] osd.1 10.200.246.20:6804/1975 boot
2017-09-28 14:32:56.235588 mon.compute1 mon.0 10.200.246.20:6789/0 1400 : cluster [WRN] Health check failed: Reduced data availability: 17 pgs inactive, 66 pgs peering (PG_AVAILABILITY)
2017-09-28 14:32:56.235651 mon.compute1 mon.0 10.200.246.20:6789/0 1401 : cluster [WRN] Health check failed: Degraded data redundancy: 66 pgs unclean (PG_DEGRADED)
2017-09-28 14:32:58.081234 mon.compute1 mon.0 10.200.246.20:6789/0 1406 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive, 54 pgs peering (PG_AVAILABILITY)
2017-09-28 14:32:58.081312 mon.compute1 mon.0 10.200.246.20:6789/0 1407 : cluster [WRN] Health check update: Degraded data redundancy: 54 pgs unclean (PG_DEGRADED)
2017-09-28 14:32:59.587760 mon.compute1 mon.0 10.200.246.20:6789/0 1408 : cluster [WRN] Health check update: Reduced data availability: 22 pgs peering (PG_AVAILABILITY)
2017-09-28 14:32:59.587849 mon.compute1 mon.0 10.200.246.20:6789/0 1409 : cluster [WRN] Health check update: Degraded data redundancy: 22 pgs unclean (PG_DEGRADED)
2017-09-28 14:32:53.921048 osd.0 osd.0 10.200.246.20:6800/1732 13 : cluster [WRN] Monitor daemon marked osd.0 down, but it is still running
2017-09-28 14:32:53.949895 osd.1 osd.1 10.200.246.20:6804/1975 13 : cluster [WRN] Monitor daemon marked osd.1 down, but it is still running
2017-09-28 14:33:01.818327 mon.compute1 mon.0 10.200.246.20:6789/0 1410 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 22 pgs peering)
2017-09-28 14:33:01.818366 mon.compute1 mon.0 10.200.246.20:6789/0 1411 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22 pgs unclean)
2017-09-28 14:33:01.818403 mon.compute1 mon.0 10.200.246.20:6789/0 1412 : cluster [INF] Cluster is now healthy
2017-09-28 14:38:58.816785 mon.compute1 mon.0 10.200.246.20:6789/0 1415 : cluster [INF] osd.6 marked down after no beacon for 300.362440 seconds
2017-09-28 14:38:58.818776 mon.compute1 mon.0 10.200.246.20:6789/0 1416 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:39:00.464321 mon.compute1 mon.0 10.200.246.20:6789/0 1425 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:39:00.464421 mon.compute1 mon.0 10.200.246.20:6789/0 1426 : cluster [INF] Cluster is now healthy
2017-09-28 14:39:00.531501 mon.compute1 mon.0 10.200.246.20:6789/0 1427 : cluster [INF] osd.6 10.200.246.63:6800/1410 boot
2017-09-28 14:38:59.100435 osd.6 osd.6 10.200.246.63:6800/1410 11 : cluster [WRN] Monitor daemon marked osd.6 down, but it is still running
2017-09-28 14:43:43.998309 mon.compute1 mon.0 10.200.246.20:6789/0 1437 : cluster [INF] osd.3 marked down after no beacon for 300.202116 seconds
2017-09-28 14:43:44.000610 mon.compute1 mon.0 10.200.246.20:6789/0 1438 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:43:45.554303 mon.compute1 mon.0 10.200.246.20:6789/0 1447 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:43:45.554373 mon.compute1 mon.0 10.200.246.20:6789/0 1448 : cluster [INF] Cluster is now healthy
2017-09-28 14:43:45.627630 mon.compute1 mon.0 10.200.246.20:6789/0 1449 : cluster [INF] osd.3 10.200.246.61:6800/1478 boot
2017-09-28 14:43:46.609836 mon.compute1 mon.0 10.200.246.20:6789/0 1458 : cluster [WRN] Health check failed: Reduced data availability: 8 pgs inactive, 26 pgs peering (PG_AVAILABILITY)
2017-09-28 14:43:46.609902 mon.compute1 mon.0 10.200.246.20:6789/0 1459 : cluster [WRN] Health check failed: Degraded data redundancy: 26 pgs unclean (PG_DEGRADED)
2017-09-28 14:43:48.627072 mon.compute1 mon.0 10.200.246.20:6789/0 1464 : cluster [WRN] Health check update: Reduced data availability: 38 pgs peering (PG_AVAILABILITY)
2017-09-28 14:43:48.627139 mon.compute1 mon.0 10.200.246.20:6789/0 1465 : cluster [WRN] Health check update: Degraded data redundancy: 38 pgs unclean (PG_DEGRADED)
2017-09-28 14:43:45.067210 osd.3 osd.3 10.200.246.61:6800/1478 15 : cluster [WRN] Monitor daemon marked osd.3 down, but it is still running
2017-09-28 14:43:50.821390 mon.compute1 mon.0 10.200.246.20:6789/0 1466 : cluster [WRN] Health check update: Reduced data availability: 23 pgs peering (PG_AVAILABILITY)
2017-09-28 14:43:50.821571 mon.compute1 mon.0 10.200.246.20:6789/0 1467 : cluster [WRN] Health check update: Degraded data redundancy: 23 pgs unclean (PG_DEGRADED)
2017-09-28 14:43:53.991359 mon.compute1 mon.0 10.200.246.20:6789/0 1468 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 23 pgs peering)
2017-09-28 14:43:53.991425 mon.compute1 mon.0 10.200.246.20:6789/0 1469 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23 pgs unclean)
2017-09-28 14:43:53.991536 mon.compute1 mon.0 10.200.246.20:6789/0 1470 : cluster [INF] Cluster is now healthy
2017-09-28 14:45:24.086279 mon.compute1 mon.0 10.200.246.20:6789/0 1471 : cluster [INF] osd.4 marked down after no beacon for 300.536088 seconds
2017-09-28 14:45:24.088550 mon.compute1 mon.0 10.200.246.20:6789/0 1472 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:45:26.667026 mon.compute1 mon.0 10.200.246.20:6789/0 1485 : cluster [WRN] Health check failed: Reduced data availability: 13 pgs inactive, 31 pgs peering (PG_AVAILABILITY)
2017-09-28 14:45:26.667119 mon.compute1 mon.0 10.200.246.20:6789/0 1486 : cluster [WRN] Health check failed: Degraded data redundancy: 31 pgs unclean (PG_DEGRADED)
2017-09-28 14:45:28.907068 mon.compute1 mon.0 10.200.246.20:6789/0 1487 : cluster [WRN] Health check update: Degraded data redundancy: 8/96 objects degraded (8.333%), 128 pgs unclean, 98 pgs degraded (PG_DEGRADED)
2017-09-28 14:45:28.925887 mon.compute1 mon.0 10.200.246.20:6789/0 1488 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:45:29.073246 mon.compute1 mon.0 10.200.246.20:6789/0 1489 : cluster [INF] osd.4 10.200.246.62:6800/1478 boot
2017-09-28 14:45:27.817903 osd.4 osd.4 10.200.246.62:6800/1478 15 : cluster [WRN] Monitor daemon marked osd.4 down, but it is still running
2017-09-28 14:45:30.381549 mon.compute1 mon.0 10.200.246.20:6789/0 1498 : cluster [WRN] Health check update: Reduced data availability: 13 pgs inactive, 14 pgs peering (PG_AVAILABILITY)
2017-09-28 14:45:30.381630 mon.compute1 mon.0 10.200.246.20:6789/0 1499 : cluster [WRN] Health check update: Degraded data redundancy: 8/96 objects degraded (8.333%), 156 pgs unclean, 126 pgs degraded (PG_DEGRADED)
2017-09-28 14:45:34.054276 mon.compute1 mon.0 10.200.246.20:6789/0 1504 : cluster [WRN] Health check update: Degraded data redundancy: 35 pgs unclean, 18 pgs degraded (PG_DEGRADED)
2017-09-28 14:45:34.054349 mon.compute1 mon.0 10.200.246.20:6789/0 1505 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 14 pgs peering)
2017-09-28 14:45:36.055181 mon.compute1 mon.0 10.200.246.20:6789/0 1506 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35 pgs unclean, 18 pgs degraded)
2017-09-28 14:45:36.055268 mon.compute1 mon.0 10.200.246.20:6789/0 1507 : cluster [INF] Cluster is now healthy
2017-09-28 14:47:59.152723 mon.compute1 mon.0 10.200.246.20:6789/0 1509 : cluster [INF] osd.1 marked down after no beacon for 300.886271 seconds
2017-09-28 14:47:59.154945 mon.compute1 mon.0 10.200.246.20:6789/0 1510 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:48:00.638308 mon.compute1 mon.0 10.200.246.20:6789/0 1519 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 12 pgs peering (PG_AVAILABILITY)
2017-09-28 14:48:00.638379 mon.compute1 mon.0 10.200.246.20:6789/0 1520 : cluster [WRN] Health check failed: Degraded data redundancy: 12 pgs unclean (PG_DEGRADED)
2017-09-28 14:48:02.611313 mon.compute1 mon.0 10.200.246.20:6789/0 1525 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:48:02.621193 mon.compute1 mon.0 10.200.246.20:6789/0 1526 : cluster [WRN] Health check update: Degraded data redundancy: 7/96 objects degraded (7.292%), 85 pgs unclean, 73 pgs degraded (PG_DEGRADED)
2017-09-28 14:48:02.673049 mon.compute1 mon.0 10.200.246.20:6789/0 1527 : cluster [INF] osd.1 10.200.246.20:6804/1975 boot
2017-09-28 14:48:06.148432 mon.compute1 mon.0 10.200.246.20:6789/0 1536 : cluster [WRN] Health check update: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY)
2017-09-28 14:48:06.148522 mon.compute1 mon.0 10.200.246.20:6789/0 1537 : cluster [WRN] Health check update: Degraded data redundancy: 7/96 objects degraded (7.292%), 79 pgs unclean, 73 pgs degraded (PG_DEGRADED)
2017-09-28 14:48:08.150154 mon.compute1 mon.0 10.200.246.20:6789/0 1538 : cluster [WRN] Health check update: Degraded data redundancy: 2 pgs unclean (PG_DEGRADED)
2017-09-28 14:48:08.150216 mon.compute1 mon.0 10.200.246.20:6789/0 1539 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering)
2017-09-28 14:48:01.259717 osd.1 osd.1 10.200.246.20:6804/1975 15 : cluster [WRN] Monitor daemon marked osd.1 down, but it is still running
2017-09-28 14:48:11.084090 mon.compute1 mon.0 10.200.246.20:6789/0 1540 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs unclean)
2017-09-28 14:48:11.084158 mon.compute1 mon.0 10.200.246.20:6789/0 1541 : cluster [INF] Cluster is now healthy
2017-09-28 14:52:59.340358 mon.compute1 mon.0 10.200.246.20:6789/0 1544 : cluster [INF] osd.0 marked down after no beacon for 300.224962 seconds
2017-09-28 14:52:59.342223 mon.compute1 mon.0 10.200.246.20:6789/0 1545 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:53:01.100433 mon.compute1 mon.0 10.200.246.20:6789/0 1554 : cluster [WRN] Health check failed: Reduced data availability: 5 pgs inactive, 19 pgs peering (PG_AVAILABILITY)
2017-09-28 14:53:01.100543 mon.compute1 mon.0 10.200.246.20:6789/0 1555 : cluster [WRN] Health check failed: Degraded data redundancy: 19 pgs unclean (PG_DEGRADED)
2017-09-28 14:53:02.831879 mon.compute1 mon.0 10.200.246.20:6789/0 1560 : cluster [WRN] Health check update: Degraded data redundancy: 7/96 objects degraded (7.292%), 95 pgs unclean, 77 pgs degraded (PG_DEGRADED)
2017-09-28 14:53:02.844945 mon.compute1 mon.0 10.200.246.20:6789/0 1561 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:53:02.963916 mon.compute1 mon.0 10.200.246.20:6789/0 1562 : cluster [INF] osd.0 10.200.246.20:6800/1732 boot
2017-09-28 14:53:05.157621 mon.compute1 mon.0 10.200.246.20:6789/0 1571 : cluster [WRN] Health check update: Degraded data redundancy: 7/96 objects degraded (7.292%), 110 pgs unclean, 77 pgs degraded (PG_DEGRADED)
2017-09-28 14:53:06.289097 mon.compute1 mon.0 10.200.246.20:6789/0 1572 : cluster [WRN] Health check update: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY)
2017-09-28 14:53:06.289173 mon.compute1 mon.0 10.200.246.20:6789/0 1573 : cluster [WRN] Health check update: Degraded data redundancy: 7/96 objects degraded (7.292%), 96 pgs unclean, 77 pgs degraded (PG_DEGRADED)
2017-09-28 14:53:08.333526 mon.compute1 mon.0 10.200.246.20:6789/0 1574 : cluster [WRN] Health check update: Degraded data redundancy: 15 pgs unclean (PG_DEGRADED)
2017-09-28 14:53:08.333592 mon.compute1 mon.0 10.200.246.20:6789/0 1575 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering)
2017-09-28 14:53:01.899641 osd.0 osd.0 10.200.246.20:6800/1732 15 : cluster [WRN] Monitor daemon marked osd.0 down, but it is still running
2017-09-28 14:53:10.866229 mon.compute1 mon.0 10.200.246.20:6789/0 1576 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15 pgs unclean)
2017-09-28 14:53:10.866318 mon.compute1 mon.0 10.200.246.20:6789/0 1577 : cluster [INF] Cluster is now healthy
2017-09-28 14:54:04.380384 mon.compute1 mon.0 10.200.246.20:6789/0 1579 : cluster [INF] osd.6 marked down after no beacon for 300.819489 seconds
2017-09-28 14:54:04.382641 mon.compute1 mon.0 10.200.246.20:6789/0 1580 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:54:05.822527 mon.compute1 mon.0 10.200.246.20:6789/0 1585 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:54:05.822622 mon.compute1 mon.0 10.200.246.20:6789/0 1586 : cluster [INF] Cluster is now healthy
2017-09-28 14:54:05.975682 mon.compute1 mon.0 10.200.246.20:6789/0 1587 : cluster [INF] osd.6 10.200.246.63:6800/1410 boot
2017-09-28 14:54:09.083395 mon.compute1 mon.0 10.200.246.20:6789/0 1596 : cluster [WRN] Health check failed: Degraded data redundancy: 93 pgs unclean (PG_DEGRADED)
2017-09-28 14:54:12.372374 mon.compute1 mon.0 10.200.246.20:6789/0 1597 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 93 pgs unclean)
2017-09-28 14:54:12.372477 mon.compute1 mon.0 10.200.246.20:6789/0 1598 : cluster [INF] Cluster is now healthy
2017-09-28 14:54:05.071436 osd.6 osd.6 10.200.246.63:6800/1410 13 : cluster [WRN] Monitor daemon marked osd.6 down, but it is still running
2017-09-28 14:54:39.397852 mon.compute1 mon.0 10.200.246.20:6789/0 1601 : cluster [INF] osd.2 marked down after no beacon for 300.647558 seconds
2017-09-28 14:54:39.399874 mon.compute1 mon.0 10.200.246.20:6789/0 1602 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:54:40.700622 mon.compute1 mon.0 10.200.246.20:6789/0 1611 : cluster [WRN] Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY)
2017-09-28 14:54:40.700715 mon.compute1 mon.0 10.200.246.20:6789/0 1612 : cluster [WRN] Health check failed: Degraded data redundancy: 8 pgs unclean (PG_DEGRADED)
2017-09-28 14:54:41.873960 mon.compute1 mon.0 10.200.246.20:6789/0 1617 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:54:41.958227 mon.compute1 mon.0 10.200.246.20:6789/0 1618 : cluster [INF] osd.2 10.200.246.61:6804/1802 boot
2017-09-28 14:54:43.157865 mon.compute1 mon.0 10.200.246.20:6789/0 1627 : cluster [WRN] Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY)
2017-09-28 14:54:43.157973 mon.compute1 mon.0 10.200.246.20:6789/0 1628 : cluster [WRN] Health check update: Degraded data redundancy: 1 pg unclean (PG_DEGRADED)
2017-09-28 14:54:48.392928 mon.compute1 mon.0 10.200.246.20:6789/0 1633 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering)
2017-09-28 14:54:48.392996 mon.compute1 mon.0 10.200.246.20:6789/0 1634 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg unclean)
2017-09-28 14:54:48.393040 mon.compute1 mon.0 10.200.246.20:6789/0 1635 : cluster [INF] Cluster is now healthy
2017-09-28 14:54:40.018250 osd.2 osd.2 10.200.246.61:6804/1802 15 : cluster [WRN] Monitor daemon marked osd.2 down, but it is still running
2017-09-28 14:59:04.562195 mon.compute1 mon.0 10.200.246.20:6789/0 1640 : cluster [INF] osd.7 marked down after no beacon for 300.664099 seconds
2017-09-28 14:59:04.564283 mon.compute1 mon.0 10.200.246.20:6789/0 1641 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2017-09-28 14:59:07.095761 mon.compute1 mon.0 10.200.246.20:6789/0 1650 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2017-09-28 14:59:07.095832 mon.compute1 mon.0 10.200.246.20:6789/0 1651 : cluster [INF] Cluster is now healthy
2017-09-28 14:59:07.168587 mon.compute1 mon.0 10.200.246.20:6789/0 1652 : cluster [INF] osd.7 10.200.246.63:6804/1636 boot
2017-09-28 14:59:09.225222 mon.compute1 mon.0 10.200.246.20:6789/0 1661 : cluster [WRN] Health check failed: Degraded data redundancy: 8/96 objects degraded (8.333%), 135 pgs unclean, 108 pgs degraded (PG_DEGRADED)
2017-09-28 14:59:10.613852 mon.compute1 mon.0 10.200.246.20:6789/0 1662 : cluster [WRN] Health check update: Degraded data redundancy: 5/96 objects degraded (5.208%), 98 pgs unclean, 71 pgs degraded (PG_DEGRADED)
2017-09-28 14:59:12.553913 mon.compute1 mon.0 10.200.246.20:6789/0 1663 : cluster [WRN] Health check update: Degraded data redundancy: 9 pgs unclean (PG_DEGRADED)
2017-09-28 14:59:06.358014 osd.7 osd.7 10.200.246.63:6804/1636 11 : cluster [WRN] Monitor daemon marked osd.7 down, but it is still running
2017-09-28 14:59:14.762436 mon.compute1 mon.0 10.200.246.20:6789/0 1664 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9 pgs unclean)
2017-09-28 14:59:14.762522 mon.compute1 mon.0 10.200.246.20:6789/0 1665 : cluster [INF] Cluster is now healthy
2017-09-28 15:00:00.000118 mon.compute1 mon.0 10.200.246.20:6789/0 1666 : cluster [INF] overall HEALTH_OK