ceph_w_2.log - Ceph - Ceph

Bug #23447 » ceph_w_2.log

Ceph -w when the test runs - Suvendu Mitra, 03/23/2018 09:05 AM

         cluster:
           id:     1c90ccdc-b322-4eb4-80af-7d8bca09206d
           health: HEALTH_OK
         services:
           mon: 3 daemons, quorum controller-1,controller-2,controller-3
           mgr: controller-1(active), standbys: controller-3, controller-2
           osd: 6 osds: 6 up, 6 in
         data:
           pools:   4 pools, 736 pgs
           objects: 25744 objects, 100 GB
           usage:   210 GB used, 2025 GB / 2235 GB avail
           pgs:     736 active+clean
         io:
           client:   38027 B/s rd, 40797 kB/s wr, 10 op/s rd, 17 op/s wr
 -03-23 10:40:56.495311 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:41:38.991254 mon.controller-2 [INF] mon.controller-2 calling new monitor election
 -03-23 10:41:38.992419 mon.controller-1 [INF] mon.controller-1 calling new monitor election
 -03-23 10:41:40.994395 mon.controller-1 [INF] mon.controller-1@0 won leader election with quorum 0,1
 -03-23 10:41:40.995146 mon.controller-1 [INF] overall HEALTH_OK
 -03-23 10:41:40.997025 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:41:40.998981 mon.controller-1 [INF] monmap e1: 3 mons at {controller-1=192.168.1.23:6789/0,controller-2=192.168.1.24:6789/0,controller-3=192.168.1.25:6789/0}
 -03-23 10:41:40.999597 mon.controller-1 [INF] osd.2 failed (root=default,host=controller-3) (2 reporters from different host after 4.914030 >= grace 3.000000)
 -03-23 10:41:40.999701 mon.controller-1 [INF] osd.5 failed (root=default,host=controller-3) (2 reporters from different host after 4.914071 >= grace 3.000000)
 -03-23 10:41:42.001417 mon.controller-1 [WRN] Health check failed: 2 osds down (OSD_DOWN)
 -03-23 10:41:42.001461 mon.controller-1 [WRN] Health check failed: 1 host (2 osds) down (OSD_HOST_DOWN)
 -03-23 10:41:44.227609 mon.controller-1 [WRN] Health check failed: 1/3 mons down, quorum controller-1,controller-2 (MON_DOWN)
 -03-23 10:41:45.230032 mon.controller-1 [WRN] Health check failed: Reduced data availability: 13 pgs inactive, 45 pgs peering (PG_AVAILABILITY)
 -03-23 10:41:45.230096 mon.controller-1 [WRN] Health check failed: Degraded data redundancy: 3697/51488 objects degraded (7.180%), 97 pgs unclean, 113 pgs degraded (PG_DEGRADED)
 -03-23 10:41:54.228766 mon.controller-1 [WRN] Health check update: Reduced data availability: 23 pgs inactive, 89 pgs peering (PG_AVAILABILITY)
 -03-23 10:41:54.228809 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 8179/51488 objects degraded (15.885%), 201 pgs unclean, 244 pgs degraded (PG_DEGRADED)
 -03-23 10:41:55.282550 mon.controller-1 [WRN] Health check failed: 1 slow requests are blocked > 32 sec (REQUEST_SLOW)
 -03-23 10:42:04.230561 mon.controller-1 [WRN] Health check update: 4 slow requests are blocked > 32 sec (REQUEST_SLOW)
 -03-23 10:42:05.199963 mon.controller-1 [WRN] Health check update: Reduced data availability: 10 pgs inactive, 44 pgs peering (PG_AVAILABILITY)
 -03-23 10:42:05.200000 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 12309/51488 objects degraded (23.907%), 201 pgs unclean, 359 pgs degraded (PG_DEGRADED)
 -03-23 10:42:07.207324 mon.controller-1 [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 44 pgs peering)
 -03-23 10:42:07.207366 mon.controller-1 [INF] Health check cleared: REQUEST_SLOW (was: 3 slow requests are blocked > 32 sec)
 -03-23 10:42:12.718109 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 17058/51488 objects degraded (33.130%), 204 pgs unclean, 483 pgs degraded (PG_DEGRADED)
 -03-23 10:42:18.723368 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 17058/51488 objects degraded (33.130%), 251 pgs unclean, 483 pgs degraded (PG_DEGRADED)
 -03-23 10:42:24.233715 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 17058/51488 objects degraded (33.130%), 272 pgs unclean, 483 pgs degraded (PG_DEGRADED)
 -03-23 10:42:29.234845 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 17058/51488 objects degraded (33.130%), 342 pgs unclean, 483 pgs degraded (PG_DEGRADED)
 -03-23 10:42:34.235181 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 17058/51488 objects degraded (33.130%), 404 pgs unclean, 483 pgs degraded (PG_DEGRADED)
 -03-23 10:42:39.235555 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 17058/51488 objects degraded (33.130%), 483 pgs unclean, 483 pgs degraded (PG_DEGRADED)
 -03-23 10:42:40.995337 mon.controller-1 [WRN] overall HEALTH_WARN 2 osds down; 1 host (2 osds) down; Degraded data redundancy: 17058/51488 objects degraded (33.130%), 483 pgs unclean, 483 pgs degraded; 1/3 mons down, quorum controller-1,controller-2
 -03-23 10:42:40.995799 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:42:44.761701 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 17058/51488 objects degraded (33.130%), 483 pgs unclean, 483 pgs degraded, 244 pgs undersized (PG_DEGRADED)
 -03-23 10:43:02.763052 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 17058/51488 objects degraded (33.130%), 483 pgs unclean, 483 pgs degraded, 359 pgs undersized (PG_DEGRADED)
 -03-23 10:43:09.238054 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 17058/51488 objects degraded (33.130%), 483 pgs unclean, 483 pgs degraded, 483 pgs undersized (PG_DEGRADED)
 -03-23 10:43:40.995496 mon.controller-1 [WRN] overall HEALTH_WARN 2 osds down; 1 host (2 osds) down; Degraded data redundancy: 17058/51488 objects degraded (33.130%), 483 pgs unclean, 483 pgs degraded, 483 pgs undersized; 1/3 mons down, quorum controller-1,controller-2
 -03-23 10:43:40.995820 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:44:37.168036 mon.controller-3 [INF] mon.controller-3 calling new monitor election
 -03-23 10:44:37.183142 mon.controller-3 [WRN] message from mon.0 was stamped 0.237671s in the future, clocks not synchronized
 -03-23 10:44:37.407897 mon.controller-1 [INF] mon.controller-1 calling new monitor election
 -03-23 10:44:37.408005 mon.controller-2 [INF] mon.controller-2 calling new monitor election
 -03-23 10:44:37.412459 mon.controller-1 [INF] mon.controller-1@0 won leader election with quorum 0,1,2
 -03-23 10:44:37.413079 mon.controller-1 [WRN] overall HEALTH_WARN 2 osds down; 1 host (2 osds) down; Degraded data redundancy: 17058/51488 objects degraded (33.130%), 483 pgs unclean, 483 pgs degraded, 483 pgs undersized; 1/3 mons down, quorum controller-1,controller-2
 -03-23 10:44:37.416249 mon.controller-1 [WRN] mon.2 192.168.1.25:6789/0 clock skew 0.234967s > max 0.1s
 -03-23 10:44:37.421153 mon.controller-1 [INF] monmap e1: 3 mons at {controller-1=192.168.1.23:6789/0,controller-2=192.168.1.24:6789/0,controller-3=192.168.1.25:6789/0}
 -03-23 10:44:37.424336 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:44:39.256422 mon.controller-1 [WRN] Health check failed: clock skew detected on mon.controller-3 (MON_CLOCK_SKEW)
 -03-23 10:44:39.256459 mon.controller-1 [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum controller-1,controller-2)
 -03-23 10:44:42.214686 mon.controller-3 [WRN] message from mon.0 was stamped 0.237538s in the future, clocks not synchronized
 -03-23 10:44:45.868227 mon.controller-1 [WRN] Health check update: 1 osds down (OSD_DOWN)
 -03-23 10:44:45.868261 mon.controller-1 [INF] Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
 -03-23 10:44:45.872047 mon.controller-1 [INF] osd.5 192.168.1.25:6800/3877 boot
 -03-23 10:44:46.874778 mon.controller-1 [WRN] Health check failed: Reduced data availability: 6 pgs peering (PG_AVAILABILITY)
 -03-23 10:44:46.874816 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 15847/51488 objects degraded (30.778%), 483 pgs unclean, 453 pgs degraded, 453 pgs undersized (PG_DEGRADED)
 -03-23 10:44:52.874571 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 10723/51488 objects degraded (20.826%), 418 pgs unclean, 418 pgs degraded, 237 pgs undersized (PG_DEGRADED)
 -03-23 10:44:52.874619 mon.controller-1 [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs peering)
 -03-23 10:44:59.258202 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 10678/51488 objects degraded (20.739%), 414 pgs unclean, 414 pgs degraded, 237 pgs undersized (PG_DEGRADED)
 -03-23 10:45:02.180857 mon.controller-1 [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
 -03-23 10:45:02.184723 mon.controller-1 [INF] osd.2 192.168.1.25:6802/6996 boot
 -03-23 10:45:04.259031 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 10612/51488 objects degraded (20.611%), 410 pgs unclean, 410 pgs degraded, 237 pgs undersized (PG_DEGRADED)
 -03-23 10:45:07.425096 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:45:07.425162 mon.controller-1 [WRN] mon.2 192.168.1.25:6789/0 clock skew 0.237574s > max 0.1s
 -03-23 10:45:07.451240 mon.controller-3 [WRN] message from mon.0 was stamped 0.237453s in the future, clocks not synchronized
 -03-23 10:45:09.259533 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 5122/51488 objects degraded (9.948%), 336 pgs unclean, 336 pgs degraded (PG_DEGRADED)
 -03-23 10:45:14.260469 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 5081/51488 objects degraded (9.868%), 333 pgs unclean, 333 pgs degraded (PG_DEGRADED)
 -03-23 10:45:19.260782 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 5007/51488 objects degraded (9.725%), 328 pgs unclean, 328 pgs degraded (PG_DEGRADED)
 -03-23 10:45:24.261144 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 4925/51488 objects degraded (9.565%), 324 pgs unclean, 324 pgs degraded (PG_DEGRADED)
 -03-23 10:45:29.261865 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 4865/51488 objects degraded (9.449%), 320 pgs unclean, 320 pgs degraded (PG_DEGRADED)
 -03-23 10:45:34.262225 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 4786/51488 objects degraded (9.295%), 314 pgs unclean, 314 pgs degraded (PG_DEGRADED)
 -03-23 10:45:37.413158 mon.controller-1 [WRN] overall HEALTH_WARN Degraded data redundancy: 4709/51488 objects degraded (9.146%), 309 pgs unclean, 309 pgs degraded; clock skew detected on mon.controller-3
 -03-23 10:45:39.262607 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 4676/51488 objects degraded (9.082%), 306 pgs unclean, 306 pgs degraded (PG_DEGRADED)
 -03-23 10:45:44.263060 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 4602/51488 objects degraded (8.938%), 301 pgs unclean, 301 pgs degraded (PG_DEGRADED)
 -03-23 10:45:49.263530 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 4490/51488 objects degraded (8.720%), 294 pgs unclean, 294 pgs degraded (PG_DEGRADED)
 -03-23 10:45:54.263852 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 4419/51488 objects degraded (8.583%), 287 pgs unclean, 287 pgs degraded (PG_DEGRADED)
 -03-23 10:45:59.264162 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 4359/51488 objects degraded (8.466%), 282 pgs unclean, 282 pgs degraded (PG_DEGRADED)
 -03-23 10:46:04.264537 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 4233/51488 objects degraded (8.221%), 277 pgs unclean, 277 pgs degraded (PG_DEGRADED)
 -03-23 10:46:07.425910 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:46:07.425969 mon.controller-1 [WRN] mon.2 192.168.1.25:6789/0 clock skew 0.237544s > max 0.1s
 -03-23 10:46:09.264914 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 4156/51488 objects degraded (8.072%), 271 pgs unclean, 271 pgs degraded (PG_DEGRADED)
 -03-23 10:46:14.265856 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 4071/51488 objects degraded (7.907%), 266 pgs unclean, 266 pgs degraded (PG_DEGRADED)
 -03-23 10:46:19.266230 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 3974/51488 objects degraded (7.718%), 261 pgs unclean, 261 pgs degraded (PG_DEGRADED)
 -03-23 10:46:24.267137 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 3891/51488 objects degraded (7.557%), 255 pgs unclean, 255 pgs degraded (PG_DEGRADED)
 -03-23 10:46:29.267473 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 3798/51488 objects degraded (7.376%), 248 pgs unclean, 248 pgs degraded (PG_DEGRADED)
 -03-23 10:46:34.267809 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 3701/51488 objects degraded (7.188%), 242 pgs unclean, 242 pgs degraded (PG_DEGRADED)
 -03-23 10:46:37.413327 mon.controller-1 [WRN] overall HEALTH_WARN Degraded data redundancy: 3632/51488 objects degraded (7.054%), 237 pgs unclean, 237 pgs degraded; clock skew detected on mon.controller-3
 -03-23 10:46:39.268176 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 3601/51488 objects degraded (6.994%), 235 pgs unclean, 235 pgs degraded (PG_DEGRADED)
 -03-23 10:46:44.268452 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 3509/51488 objects degraded (6.815%), 230 pgs unclean, 230 pgs degraded (PG_DEGRADED)
 -03-23 10:46:49.268756 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 3445/51488 objects degraded (6.691%), 225 pgs unclean, 225 pgs degraded (PG_DEGRADED)
 -03-23 10:46:54.269099 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 3343/51488 objects degraded (6.493%), 220 pgs unclean, 220 pgs degraded (PG_DEGRADED)
 -03-23 10:46:59.269572 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 3305/51488 objects degraded (6.419%), 215 pgs unclean, 215 pgs degraded (PG_DEGRADED)
 -03-23 10:47:04.270929 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 3153/51488 objects degraded (6.124%), 206 pgs unclean, 206 pgs degraded (PG_DEGRADED)
 -03-23 10:47:07.426705 mon.controller-1 [WRN] mon.2 192.168.1.25:6789/0 clock skew 0.237601s > max 0.1s
 -03-23 10:47:07.426760 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:47:09.273406 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 3102/51488 objects degraded (6.025%), 203 pgs unclean, 203 pgs degraded (PG_DEGRADED)
 -03-23 10:47:12.577442 mon.controller-3 [WRN] message from mon.0 was stamped 0.237538s in the future, clocks not synchronized
 -03-23 10:47:14.274397 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 2986/51488 objects degraded (5.799%), 195 pgs unclean, 195 pgs degraded (PG_DEGRADED)
 -03-23 10:47:19.274751 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 2891/51488 objects degraded (5.615%), 191 pgs unclean, 191 pgs degraded (PG_DEGRADED)
 -03-23 10:47:24.275060 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 2799/51488 objects degraded (5.436%), 184 pgs unclean, 184 pgs degraded (PG_DEGRADED)
 -03-23 10:47:29.275370 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 2737/51488 objects degraded (5.316%), 180 pgs unclean, 180 pgs degraded (PG_DEGRADED)
 -03-23 10:47:34.276147 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 2621/51488 objects degraded (5.091%), 173 pgs unclean, 173 pgs degraded (PG_DEGRADED)
 -03-23 10:47:37.413525 mon.controller-1 [WRN] overall HEALTH_WARN Degraded data redundancy: 2553/51488 objects degraded (4.958%), 168 pgs unclean, 168 pgs degraded; clock skew detected on mon.controller-3
 -03-23 10:47:39.276452 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 2517/51488 objects degraded (4.889%), 166 pgs unclean, 166 pgs degraded (PG_DEGRADED)
 -03-23 10:47:44.276784 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 2435/51488 objects degraded (4.729%), 161 pgs unclean, 161 pgs degraded (PG_DEGRADED)
 -03-23 10:47:49.277190 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 2349/51488 objects degraded (4.562%), 156 pgs unclean, 156 pgs degraded (PG_DEGRADED)
 -03-23 10:47:54.277678 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 2270/51488 objects degraded (4.409%), 151 pgs unclean, 151 pgs degraded (PG_DEGRADED)
 -03-23 10:47:59.278827 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 2207/51488 objects degraded (4.286%), 147 pgs unclean, 147 pgs degraded (PG_DEGRADED)
 -03-23 10:48:04.279168 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 2089/51488 objects degraded (4.057%), 138 pgs unclean, 138 pgs degraded (PG_DEGRADED)
 -03-23 10:48:07.427463 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:48:07.427540 mon.controller-1 [WRN] mon.2 192.168.1.25:6789/0 clock skew 0.23751s > max 0.1s
 -03-23 10:48:09.279566 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 1955/51488 objects degraded (3.797%), 128 pgs unclean, 128 pgs degraded (PG_DEGRADED)
 -03-23 10:48:14.280680 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 1833/51488 objects degraded (3.560%), 122 pgs unclean, 122 pgs degraded (PG_DEGRADED)
 -03-23 10:48:19.281615 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 1720/51488 objects degraded (3.341%), 115 pgs unclean, 115 pgs degraded (PG_DEGRADED)
 -03-23 10:48:24.281920 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 1607/51488 objects degraded (3.121%), 107 pgs unclean, 107 pgs degraded (PG_DEGRADED)
 -03-23 10:48:29.282246 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 1470/51488 objects degraded (2.855%), 97 pgs unclean, 97 pgs degraded (PG_DEGRADED)
 -03-23 10:48:34.282583 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 1329/51488 objects degraded (2.581%), 90 pgs unclean, 90 pgs degraded (PG_DEGRADED)
 -03-23 10:48:37.413754 mon.controller-1 [WRN] overall HEALTH_WARN Degraded data redundancy: 1232/51488 objects degraded (2.393%), 80 pgs unclean, 80 pgs degraded; clock skew detected on mon.controller-3
 -03-23 10:48:39.282992 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 1232/51488 objects degraded (2.393%), 80 pgs unclean, 80 pgs degraded (PG_DEGRADED)
 -03-23 10:48:44.283327 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 1157/51488 objects degraded (2.247%), 74 pgs unclean, 74 pgs degraded (PG_DEGRADED)
 -03-23 10:48:49.283706 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 1086/51488 objects degraded (2.109%), 70 pgs unclean, 70 pgs degraded (PG_DEGRADED)
 -03-23 10:48:54.284136 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 1063/51488 objects degraded (2.065%), 67 pgs unclean, 67 pgs degraded (PG_DEGRADED)
 -03-23 10:48:59.285735 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 1020/51488 objects degraded (1.981%), 64 pgs unclean, 64 pgs degraded (PG_DEGRADED)
 -03-23 10:49:04.286109 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 993/51488 objects degraded (1.929%), 63 pgs unclean, 63 pgs degraded (PG_DEGRADED)
 -03-23 10:49:07.428262 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:49:07.428320 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:49:09.286410 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 949/51488 objects degraded (1.843%), 60 pgs unclean, 60 pgs degraded (PG_DEGRADED)
 -03-23 10:49:09.286585 mon.controller-1 [INF] Health check cleared: MON_CLOCK_SKEW (was: clock skew detected on mon.controller-3)
 -03-23 10:49:14.287342 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 921/51488 objects degraded (1.789%), 59 pgs unclean, 59 pgs degraded (PG_DEGRADED)
 -03-23 10:49:19.287788 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 878/51488 objects degraded (1.705%), 56 pgs unclean, 56 pgs degraded (PG_DEGRADED)
 -03-23 10:49:24.288439 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 850/51488 objects degraded (1.651%), 55 pgs unclean, 55 pgs degraded (PG_DEGRADED)
 -03-23 10:49:29.288781 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 817/51488 objects degraded (1.587%), 51 pgs unclean, 51 pgs degraded (PG_DEGRADED)
 -03-23 10:49:34.289111 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 773/51488 objects degraded (1.501%), 48 pgs unclean, 48 pgs degraded (PG_DEGRADED)
 -03-23 10:49:37.413916 mon.controller-1 [WRN] overall HEALTH_WARN Degraded data redundancy: 733/51488 objects degraded (1.424%), 46 pgs unclean, 46 pgs degraded
 -03-23 10:49:37.428918 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:49:37.428960 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:49:39.289497 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 733/51488 objects degraded (1.424%), 46 pgs unclean, 46 pgs degraded (PG_DEGRADED)
 -03-23 10:49:44.289901 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 701/51488 objects degraded (1.361%), 43 pgs unclean, 43 pgs degraded (PG_DEGRADED)
 -03-23 10:49:49.290288 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 674/51488 objects degraded (1.309%), 40 pgs unclean, 40 pgs degraded (PG_DEGRADED)
 -03-23 10:49:54.290556 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 632/51488 objects degraded (1.227%), 38 pgs unclean, 38 pgs degraded (PG_DEGRADED)
 -03-23 10:49:59.291542 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 592/51488 objects degraded (1.150%), 37 pgs unclean, 37 pgs degraded (PG_DEGRADED)
 -03-23 10:50:04.292143 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 568/51488 objects degraded (1.103%), 34 pgs unclean, 34 pgs degraded (PG_DEGRADED)
 -03-23 10:50:09.292767 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 527/51488 objects degraded (1.024%), 32 pgs unclean, 32 pgs degraded (PG_DEGRADED)
 -03-23 10:50:14.293467 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 492/51488 objects degraded (0.956%), 31 pgs unclean, 31 pgs degraded (PG_DEGRADED)
 -03-23 10:50:19.293898 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 457/51488 objects degraded (0.888%), 28 pgs unclean, 28 pgs degraded (PG_DEGRADED)
 -03-23 10:50:24.294931 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 418/51488 objects degraded (0.812%), 26 pgs unclean, 26 pgs degraded (PG_DEGRADED)
 -03-23 10:50:29.295180 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 388/51488 objects degraded (0.754%), 25 pgs unclean, 25 pgs degraded (PG_DEGRADED)
 -03-23 10:50:34.295701 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 344/51488 objects degraded (0.668%), 22 pgs unclean, 22 pgs degraded (PG_DEGRADED)
 -03-23 10:50:37.414099 mon.controller-1 [WRN] overall HEALTH_WARN Degraded data redundancy: 312/51488 objects degraded (0.606%), 19 pgs unclean, 19 pgs degraded
 -03-23 10:50:37.429206 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:50:37.429256 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:50:39.296009 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 312/51488 objects degraded (0.606%), 19 pgs unclean, 19 pgs degraded (PG_DEGRADED)
 -03-23 10:50:44.296330 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 279/51488 objects degraded (0.542%), 17 pgs unclean, 17 pgs degraded (PG_DEGRADED)
 -03-23 10:50:49.296670 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 244/51488 objects degraded (0.474%), 15 pgs unclean, 15 pgs degraded (PG_DEGRADED)
 -03-23 10:50:54.296998 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 209/51488 objects degraded (0.406%), 13 pgs unclean, 13 pgs degraded (PG_DEGRADED)
 -03-23 10:50:59.297353 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 173/51488 objects degraded (0.336%), 11 pgs unclean, 11 pgs degraded (PG_DEGRADED)
 -03-23 10:51:04.297694 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 135/51488 objects degraded (0.262%), 9 pgs unclean, 9 pgs degraded (PG_DEGRADED)
 -03-23 10:51:09.298171 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 97/51488 objects degraded (0.188%), 7 pgs unclean, 7 pgs degraded (PG_DEGRADED)
 -03-23 10:51:14.298498 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 61/51488 objects degraded (0.118%), 4 pgs unclean, 4 pgs degraded (PG_DEGRADED)
 -03-23 10:51:19.298770 mon.controller-1 [WRN] Health check update: Degraded data redundancy: 23/51488 objects degraded (0.045%), 2 pgs unclean, 2 pgs degraded (PG_DEGRADED)
 -03-23 10:51:21.242514 mon.controller-1 [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/51488 objects degraded (0.045%), 2 pgs unclean, 2 pgs degraded)
 -03-23 10:51:21.242571 mon.controller-1 [INF] Cluster is now healthy
 -03-23 10:51:37.414281 mon.controller-1 [INF] overall HEALTH_OK
 -03-23 10:51:37.429365 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:51:37.429452 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:52:37.429580 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:52:37.429631 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:53:37.429712 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:53:37.429769 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:54:37.429859 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:54:37.429914 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:55:37.430094 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:55:37.430150 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:56:37.430307 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:56:37.430361 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:57:37.430467 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:57:37.430523 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:58:37.430753 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:58:37.430809 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 10:59:37.430620 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0
 -03-23 10:59:37.430696 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 11:00:00.000154 mon.controller-1 [INF] overall HEALTH_OK
 -03-23 11:00:37.430974 mon.controller-1 [INF] mon.1 192.168.1.24:6789/0
 -03-23 11:00:37.431040 mon.controller-1 [INF] mon.2 192.168.1.25:6789/0

« Previous
1
2
3
4
Next »

(1-1/4)

Project

General

Profile

Ceph

Bug #23447 » ceph_w_2.log