health HEALTH_WARN noscrub,nodeep-scrub flag(s) set monmap e1: 3 mons at {0=10.6.0.1:6789/0,2=10.6.0.3:6789/0,3=10.6.0.4:6789/0}, election epoch 1596, quorum 0,1,2 0,2,3 osdmap e5878: 7 osds: 7 up, 7 in pgmap v579242: 1152 pgs: 1152 active+clean; 2336 GB data, 7223 GB used, 18812 GB / 26035 GB avail; 661KB/s rd, 826op/s mdsmap e1: 0/0/1 up 2013-05-23 22:19:19.937892 mon.0 [INF] pgmap v579242: 1152 pgs: 1152 active+clean; 2336 GB data, 7223 GB used, 18812 GB / 26035 GB avail; 661KB/s rd, 826op/s 2013-05-23 22:19:22.095391 mon.0 [INF] pgmap v579243: 1152 pgs: 1152 active+clean; 2336 GB data, 7223 GB used, 18812 GB / 26035 GB avail; 132KB/s rd, 163op/s 2013-05-23 22:19:23.138855 mon.0 [INF] pgmap v579244: 1152 pgs: 1152 active+clean; 2336 GB data, 7223 GB used, 18812 GB / 26035 GB avail; 119KB/s rd, 6316B/s wr, 152op/s 2013-05-23 22:19:24.482235 mon.0 [INF] pgmap v579245: 1152 pgs: 1152 active+clean; 2336 GB data, 7223 GB used, 18812 GB / 26035 GB avail; 113KB/s rd, 13685B/s wr, 148op/s 2013-05-23 22:19:27.899250 mon.0 [INF] pgmap v579246: 1152 pgs: 1152 active+clean; 2336 GB data, 7223 GB used, 18812 GB / 26035 GB avail; 6029B/s rd, 2583B/s wr, 2op/s 2013-05-23 22:19:31.410655 mon.0 [INF] osdmap e5879: 7 osds: 7 up, 6 in 2013-05-23 22:19:31.547648 mon.0 [INF] pgmap v579247: 1152 pgs: 1152 active+clean; 2336 GB data, 5248 GB used, 17063 GB / 22311 GB avail; 4094B/s rd, 1754B/s wr, 1op/s 2013-05-23 22:19:32.959446 mon.0 [INF] osdmap e5880: 7 osds: 7 up, 5 in 2013-05-23 22:19:33.641334 mon.0 [INF] pgmap v579248: 1152 pgs: 1152 active+clean; 2336 GB data, 4763 GB used, 13835 GB / 18598 GB avail; 2445B/s wr, 0op/s 2013-05-23 22:19:34.012512 mon.0 [INF] osdmap e5881: 7 osds: 7 up, 5 in 2013-05-23 22:19:34.159739 mon.0 [INF] pgmap v579249: 1152 pgs: 1152 active+clean; 2336 GB data, 4763 GB used, 13835 GB / 18598 GB avail 2013-05-23 22:19:35.315692 mon.0 [INF] osdmap e5882: 7 osds: 7 up, 5 in 2013-05-23 22:19:35.484900 mon.0 [INF] pgmap v579250: 1152 pgs: 1152 active+clean; 2336 GB data, 4763 GB used, 13835 GB / 18598 GB avail 2013-05-23 22:19:36.269074 mon.0 [INF] osdmap e5883: 7 osds: 7 up, 5 in 2013-05-23 22:19:35.182493 osd.7 [INF] 3.255 restarting backfill on osd.4 from (0'0,0'0] MAX to 3796'3678 2013-05-23 22:19:35.588159 osd.7 [INF] 3.1bd restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'23505 2013-05-23 22:19:36.276604 osd.7 [INF] 3.3df restarting backfill on osd.6 from (0'0,0'0] MAX to 5869'11190 2013-05-23 22:19:36.721204 osd.7 [INF] 3.12e restarting backfill on osd.6 from (0'0,0'0] MAX to 2851'3831 2013-05-23 22:19:36.907369 osd.7 [INF] 3.b0 restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'8047 2013-05-23 22:19:36.929791 osd.7 [INF] 3.3ad restarting backfill on osd.4 from (0'0,0'0] MAX to 1367'6713 2013-05-23 22:19:36.941155 osd.7 [INF] 3.10e restarting backfill on osd.10 from (0'0,0'0] MAX to 2912'3162 2013-05-23 22:19:36.311235 osd.6 [INF] 3.312 restarting backfill on osd.10 from (0'0,0'0] MAX to 3645'7184 2013-05-23 22:19:36.328244 osd.6 [INF] 3.2a3 restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'5072 2013-05-23 22:19:36.358715 osd.6 [INF] 3.2e2 restarting backfill on osd.7 from (0'0,0'0] MAX to 5159'4629 2013-05-23 22:19:36.425913 osd.6 [INF] 3.2e3 restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'3605 2013-05-23 22:19:36.435360 osd.6 [INF] 3.272 restarting backfill on osd.10 from (0'0,0'0] MAX to 5209'4259 2013-05-23 22:19:36.618317 mon.0 [INF] pgmap v579251: 1152 pgs: 1152 active+clean; 2336 GB data, 4763 GB used, 13835 GB / 18598 GB avail 2013-05-23 22:19:36.686325 osd.6 [INF] 3.57 restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'4753 2013-05-23 22:19:36.726827 osd.6 [INF] 3.19f restarting backfill on osd.7 from (0'0,0'0] MAX to 2878'5532 2013-05-23 22:19:36.727045 osd.6 [INF] 4.56 restarting backfill on osd.4 from (0'0,0'0] MAX to 108'8788 2013-05-23 22:19:36.770665 osd.6 [INF] 3.174 restarting backfill on osd.4 from (0'0,0'0] MAX to 4926'11463 2013-05-23 22:19:36.821916 osd.6 [INF] 4.4d restarting backfill on osd.4 from (0'0,0'0] MAX to 108'4593 2013-05-23 22:19:36.975450 osd.6 [INF] 3.331 restarting backfill on osd.7 from (0'0,0'0] MAX to 4969'47482 2013-05-23 22:19:37.021157 osd.6 [INF] 3.2c restarting backfill on osd.7 from (0'0,0'0] MAX to 455'6084 2013-05-23 22:19:37.055100 osd.6 [INF] 4.2b restarting backfill on osd.7 from (0'0,0'0] MAX to 108'5582 2013-05-23 22:19:37.120980 osd.6 [INF] 3.205 restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'4082 2013-05-23 22:19:37.141564 osd.6 [INF] 3.171 restarting backfill on osd.10 from (0'0,0'0] MAX to 2912'7762 2013-05-23 22:19:37.160096 osd.6 [INF] 3.172 restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'10723 2013-05-23 22:19:37.350379 osd.6 [INF] 3.2f8 restarting backfill on osd.11 from (0'0,0'0] MAX to 4969'47312 2013-05-23 22:19:37.837410 osd.6 [INF] 3.211 restarting backfill on osd.11 from (0'0,0'0] MAX to 3094'5765 2013-05-23 22:19:37.869969 osd.6 [INF] 3.181 restarting backfill on osd.11 from (0'0,0'0] MAX to 2400'4201 2013-05-23 22:19:37.977047 osd.6 [INF] 3.344 restarting backfill on osd.11 from (0'0,0'0] MAX to 3796'15824 2013-05-23 22:19:38.100347 mon.0 [INF] osdmap e5884: 7 osds: 7 up, 5 in 2013-05-23 22:19:38.245974 mon.0 [INF] pgmap v579252: 1152 pgs: 1152 active+clean; 2336 GB data, 4763 GB used, 13835 GB / 18598 GB avail 2013-05-23 22:19:38.358850 osd.0 [INF] 3.24c restarting backfill on osd.7 from (0'0,0'0] MAX to 2851'4014 2013-05-23 22:19:38.462925 osd.0 [INF] 3.234 restarting backfill on osd.7 from (0'0,0'0] MAX to 3815'4319 2013-05-23 22:19:39.310609 mon.0 [INF] osdmap e5885: 7 osds: 7 up, 5 in 2013-05-23 22:19:40.081057 osd.0 [INF] 3.3b9 restarting backfill on osd.7 from (0'0,0'0] MAX to 3094'3897 2013-05-23 22:19:40.111109 osd.0 [INF] 3.1a8 restarting backfill on osd.6 from (0'0,0'0] MAX to 2805'4099 2013-05-23 22:19:40.381914 osd.0 [INF] 4.1a restarting backfill on osd.6 from (0'0,0'0] MAX to 108'3849 2013-05-23 22:19:40.407022 osd.0 [INF] 4.13 restarting backfill on osd.7 from (0'0,0'0] MAX to 108'3165 2013-05-23 22:19:40.429797 osd.0 [INF] 3.14 restarting backfill on osd.7 from (0'0,0'0] MAX to 2851'3190 2013-05-23 22:19:40.431696 osd.0 [INF] 3.308 restarting backfill on osd.6 from (0'0,0'0] MAX to 2344'7291 2013-05-23 22:19:40.462036 osd.0 [INF] 3.30a restarting backfill on osd.10 from (0'0,0'0] MAX to 2851'7310 2013-05-23 22:19:40.602225 osd.0 [INF] 3.101 restarting backfill on osd.4 from (0'0,0'0] MAX to 2851'4965 2013-05-23 22:19:40.604696 osd.0 [INF] 3.36b restarting backfill on osd.7 from (0'0,0'0] MAX to 2055'3628 2013-05-23 22:19:40.621567 osd.0 [INF] 3.e3 restarting backfill on osd.10 from (0'0,0'0] MAX to 2383'35906 2013-05-23 22:19:40.623950 osd.0 [INF] 3.2d4 restarting backfill on osd.7 from (0'0,0'0] MAX to 2800'3666 2013-05-23 22:19:40.641656 osd.0 [INF] 3.3e0 restarting backfill on osd.7 from (0'0,0'0] MAX to 2912'3958 2013-05-23 22:19:40.648145 osd.0 [INF] 3.c2 restarting backfill on osd.4 from (0'0,0'0] MAX to 2893'5139 2013-05-23 22:19:40.664293 osd.0 [INF] 3.254 restarting backfill on osd.10 from (0'0,0'0] MAX to 2912'3532 2013-05-23 22:19:40.677229 osd.0 [INF] 3.1fd restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'5869 2013-05-23 22:19:36.886631 osd.10 [INF] 4.67 restarting backfill on osd.11 from (0'0,0'0] MAX to 3094'4083 2013-05-23 22:19:36.916044 osd.10 [INF] 3.102 restarting backfill on osd.7 from (0'0,0'0] MAX to 5541'3906 2013-05-23 22:19:36.958412 osd.10 [INF] 3.242 restarting backfill on osd.7 from (0'0,0'0] MAX to 2851'4606 2013-05-23 22:19:37.258448 osd.10 [INF] 3.c5 restarting backfill on osd.7 from (0'0,0'0] MAX to 5878'22379 2013-05-23 22:19:37.267340 osd.10 [INF] 3.1c4 restarting backfill on osd.7 from (0'0,0'0] MAX to 4969'13086 2013-05-23 22:19:37.352062 osd.8 [INF] 3.2ee restarting backfill on osd.4 from (0'0,0'0] MAX to 5689'8963 2013-05-23 22:19:37.752207 osd.10 [INF] 3.15e restarting backfill on osd.11 from (0'0,0'0] MAX to 2912'4042 2013-05-23 22:19:37.776460 osd.10 [INF] 3.189 restarting backfill on osd.11 from (0'0,0'0] MAX to 5869'4313 2013-05-23 22:19:37.803082 osd.10 [INF] 3.68 restarting backfill on osd.11 from (0'0,0'0] MAX to 4969'39208 2013-05-23 22:19:37.913724 osd.8 [INF] 3.ef restarting backfill on osd.6 from (0'0,0'0] MAX to 5858'4652 2013-05-23 22:19:37.999730 osd.10 [INF] 3.70 restarting backfill on osd.6 from (0'0,0'0] MAX to 2344'4392 2013-05-23 22:19:38.111489 osd.8 [INF] 3.32 restarting backfill on osd.6 from (0'0,0'0] MAX to 3094'3428 2013-05-23 22:19:38.232750 osd.10 [INF] 4.6f restarting backfill on osd.6 from (0'0,0'0] MAX to 108'3528 2013-05-23 22:19:38.253637 osd.8 [INF] 3.31f restarting backfill on osd.6 from (0'0,0'0] MAX to 2767'3597 2013-05-23 22:19:39.534315 osd.10 [INF] 3.360 restarting backfill on osd.7 from (0'0,0'0] MAX to 5876'47800 2013-05-23 22:19:39.502882 mon.0 [INF] pgmap v579253: 1152 pgs: 1152 active+clean; 2336 GB data, 4763 GB used, 13835 GB / 18598 GB avail 2013-05-23 22:19:43.807209 mon.0 [INF] pgmap v579254: 1152 pgs: 717 active+clean, 8 active+remapped+wait_backfill, 187 active+remapped, 5 active+recovery_wait+remapped, 235 remapped+peering; 2197 GB data, 5248 GB used, 17062 GB / 22311 GB avail; 15EB/s rd, 15EB/s wr, 15Eop/s; 58377/1763200 degraded (3.311%); recovering 15E o/s, 15EB/s, 15E key/s 2013-05-23 22:19:37.431266 osd.4 [INF] 3.2f1 restarting backfill on osd.10 from (0'0,0'0] MAX to 5011'40054 2013-05-23 22:19:37.723761 osd.4 [INF] 3.21d restarting backfill on osd.7 from (0'0,0'0] MAX to 4230'7206 2013-05-23 22:19:37.944398 osd.4 [INF] 3.78 restarting backfill on osd.7 from (0'0,0'0] MAX to 2805'3187 2013-05-23 22:19:37.981293 osd.4 [INF] 3.266 restarting backfill on osd.10 from (0'0,0'0] MAX to 3090'3408 2013-05-23 22:19:38.001854 osd.4 [INF] 3.25d restarting backfill on osd.11 from (0'0,0'0] MAX to 5715'3051 2013-05-23 22:19:38.127624 osd.4 [INF] 4.77 restarting backfill on osd.7 from (0'0,0'0] MAX to 108'3187 2013-05-23 22:19:38.149493 osd.4 [INF] 3.8b restarting backfill on osd.6 from (0'0,0'0] MAX to 3094'8518 2013-05-23 22:19:38.528615 osd.4 [INF] 3.21c restarting backfill on osd.11 from (0'0,0'0] MAX to 2383'3010 2013-05-23 22:19:39.573083 osd.4 [INF] 3.3be restarting backfill on osd.7 from (0'0,0'0] MAX to 2851'3192 2013-05-23 22:19:39.574197 osd.4 [INF] 3.180 restarting backfill on osd.6 from (0'0,0'0] MAX to 3094'8659 2013-05-23 22:19:39.714978 osd.4 [INF] 3.1d9 restarting backfill on osd.11 from (0'0,0'0] MAX to 2757'3012 2013-05-23 22:19:40.102251 osd.4 [INF] 3.2c4 restarting backfill on osd.7 from (0'0,0'0] MAX to 2912'11091 2013-05-23 22:19:40.102863 osd.4 [INF] 3.302 restarting backfill on osd.11 from (0'0,0'0] MAX to 2912'4161 2013-05-23 22:19:40.153526 osd.4 [INF] 3.2bf restarting backfill on osd.7 from (0'0,0'0] MAX to 5871'5892 2013-05-23 22:19:40.270688 osd.4 [INF] 3.325 restarting backfill on osd.7 from (0'0,0'0] MAX to 5697'11699 2013-05-23 22:19:40.297487 osd.4 [INF] 4.48 restarting backfill on osd.11 from (0'0,0'0] MAX to 108'10008 2013-05-23 22:19:40.423286 osd.4 [INF] 3.348 restarting backfill on osd.11 from (0'0,0'0] MAX to 2851'9313 2013-05-23 22:19:42.573365 osd.4 [INF] 3.d5 restarting backfill on osd.11 from (0'0,0'0] MAX to 2851'3059 2013-05-23 22:19:42.573508 osd.4 [INF] 3.47 restarting backfill on osd.7 from (0'0,0'0] MAX to 4997'45528 2013-05-23 22:19:42.600736 osd.4 [INF] 3.aa restarting backfill on osd.11 from (0'0,0'0] MAX to 2910'4453 2013-05-23 22:19:42.647281 osd.4 [INF] 3.3a9 restarting backfill on osd.11 from (0'0,0'0] MAX to 455'8404 2013-05-23 22:19:42.733178 osd.4 [INF] 3.126 restarting backfill on osd.11 from (0'0,0'0] MAX to 3094'81327 2013-05-23 22:19:42.771734 osd.4 [INF] 3.232 restarting backfill on osd.6 from (0'0,0'0] MAX to 2805'58087 2013-05-23 22:19:42.796003 osd.4 [INF] 3.224 restarting backfill on osd.11 from (0'0,0'0] MAX to 3093'3460 2013-05-23 22:19:42.798502 osd.4 [INF] 4.14 restarting backfill on osd.11 from (0'0,0'0] MAX to 108'3369 2013-05-23 22:19:42.851178 osd.4 [INF] 3.b1 restarting backfill on osd.10 from (0'0,0'0] MAX to 2851'8049 2013-05-23 22:19:42.862300 osd.4 [INF] 4.f restarting backfill on osd.7 from (0'0,0'0] MAX to 108'6303 2013-05-23 22:19:42.875469 osd.4 [INF] 3.3a0 restarting backfill on osd.10 from (0'0,0'0] MAX to 2851'4150 2013-05-23 22:19:42.958578 osd.4 [INF] 3.1d7 restarting backfill on osd.6 from (0'0,0'0] MAX to 4997'42785 2013-05-23 22:19:42.996993 osd.4 [INF] 3.358 restarting backfill on osd.11 from (0'0,0'0] MAX to 4997'36380 2013-05-23 22:19:43.015255 osd.4 [INF] 3.34d restarting backfill on osd.6 from (0'0,0'0] MAX to 2912'3248 2013-05-23 22:19:43.093592 osd.4 [INF] 3.17f restarting backfill on osd.7 from (0'0,0'0] MAX to 3094'5277 2013-05-23 22:19:43.208671 osd.4 [INF] 3.16a restarting backfill on osd.6 from (0'0,0'0] MAX to 413'5349 2013-05-23 22:19:43.261521 osd.4 [INF] 3.2d6 restarting backfill on osd.6 from (0'0,0'0] MAX to 3094'4271 2013-05-23 22:19:43.588886 osd.4 [INF] 3.140 restarting backfill on osd.7 from (0'0,0'0] MAX to 2344'3354 2013-05-23 22:19:44.016553 osd.4 [INF] 3.202 restarting backfill on osd.6 from (0'0,0'0] MAX to 2912'3027 2013-05-23 22:19:44.047003 osd.4 [INF] 3.2a5 restarting backfill on osd.7 from (0'0,0'0] MAX to 2912'5538 2013-05-23 22:19:44.154734 osd.4 [INF] 3.23b restarting backfill on osd.7 from (0'0,0'0] MAX to 4650'6100 2013-05-23 22:19:37.349657 osd.7 [INF] 3.2ad restarting backfill on osd.11 from (0'0,0'0] MAX to 4522'9643 2013-05-23 22:19:37.813426 osd.7 [INF] 3.d0 restarting backfill on osd.11 from (0'0,0'0] MAX to 5467'20505 2013-05-23 22:19:38.880194 osd.7 [INF] 3.2f7 restarting backfill on osd.10 from (0'0,0'0] MAX to 2851'5954 2013-05-23 22:19:39.823532 osd.7 [INF] 3.3d0 restarting backfill on osd.4 from (0'0,0'0] MAX to 3982'7139 2013-05-23 22:19:39.966194 osd.7 [INF] 3.3bf restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'5575 2013-05-23 22:19:40.076324 osd.7 [INF] 3.11a restarting backfill on osd.10 from (0'0,0'0] MAX to 2110'3757 2013-05-23 22:19:40.287450 osd.7 [INF] 3.a7 restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'8402 2013-05-23 22:19:40.352075 osd.7 [INF] 3.1f6 restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'4615 2013-05-23 22:19:40.390664 osd.7 [INF] 3.1ee restarting backfill on osd.4 from (0'0,0'0] MAX to 2379'3041 2013-05-23 22:19:40.421616 osd.7 [INF] 4.1e restarting backfill on osd.10 from (0'0,0'0] MAX to 108'6549 2013-05-23 22:19:40.635241 osd.7 [INF] 3.2cb restarting backfill on osd.4 from (0'0,0'0] MAX to 3796'5613 2013-05-23 22:19:40.776581 osd.7 [INF] 3.156 restarting backfill on osd.10 from (0'0,0'0] MAX to 2912'3467 2013-05-23 22:19:41.563315 osd.7 [INF] 3.91 restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'3537 2013-05-23 22:19:41.881410 osd.7 [INF] 3.66 restarting backfill on osd.10 from (0'0,0'0] MAX to 2851'3910 2013-05-23 22:19:43.107655 osd.7 [INF] 3.55 restarting backfill on osd.10 from (0'0,0'0] MAX to 3796'6925 2013-05-23 22:19:43.657701 osd.7 [INF] 3.347 restarting backfill on osd.4 from (0'0,0'0] MAX to 5849'15219 2013-05-23 22:19:44.957040 mon.0 [INF] pgmap v579255: 1152 pgs: 122 active+clean, 48 active+remapped+wait_backfill, 497 active+remapped, 2 active+remapped+backfilling, 33 active+recovery_wait+remapped, 450 remapped+peering; 2041 GB data, 5248 GB used, 17062 GB / 22311 GB avail; 15EB/s rd, 15EB/s wr, 15Eop/s; 177091/1707661 degraded (10.370%); recovering 15E o/s, 15EB/s, 15E key/s 2013-05-23 22:19:46.329897 mon.0 [INF] pgmap v579256: 1152 pgs: 50 active+remapped+wait_backfill, 586 active+remapped, 2 active+remapped+backfilling, 33 active+recovery_wait+remapped, 481 remapped+peering; 2041 GB data, 7223 GB used, 18811 GB / 26035 GB avail; 15EB/s rd, 15EB/s wr, 15Eop/s; 223592/1771795 degraded (12.620%); recovering 15E o/s, 15EB/s, 15E key/s 2013-05-23 22:19:41.123524 osd.6 [INF] 3.31b restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'4751 2013-05-23 22:19:41.185605 osd.6 [INF] 3.310 restarting backfill on osd.10 from (0'0,0'0] MAX to 3796'4840 2013-05-23 22:19:42.933769 osd.6 [INF] 3.112 restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'3414 2013-05-23 22:19:43.155898 osd.6 [INF] 3.2da restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'7576 2013-05-23 22:19:43.392478 osd.6 [INF] 3.233 restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'5501 2013-05-23 22:19:43.854366 osd.6 [INF] 3.212 restarting backfill on osd.10 from (0'0,0'0] MAX to 4863'5166 2013-05-23 22:19:43.901286 osd.6 [INF] 3.20b restarting backfill on osd.4 from (0'0,0'0] MAX to 2851'85888 2013-05-23 22:19:43.983884 osd.6 [INF] 3.24d restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'3353 2013-05-23 22:19:44.242255 osd.6 [INF] 3.103 restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'4752 2013-05-23 22:19:44.306642 osd.6 [INF] 3.1ef restarting backfill on osd.10 from (0'0,0'0] MAX to 5878'36089 2013-05-23 22:19:44.364446 osd.6 [INF] 3.1e0 restarting backfill on osd.10 from (0'0,0'0] MAX to 5854'3349 2013-05-23 22:19:44.390522 osd.6 [INF] 3.1db restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'28099 2013-05-23 22:19:44.410331 osd.6 [INF] 3.2b3 restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'3611 2013-05-23 22:19:44.433576 osd.6 [INF] 3.1ba restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'6390 2013-05-23 22:19:44.462601 osd.6 [INF] 3.1b0 restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'4021 2013-05-23 22:19:44.465734 osd.6 [INF] 3.c3 restarting backfill on osd.4 from (0'0,0'0] MAX to 4997'42413 2013-05-23 22:19:44.473808 osd.6 [INF] 3.29b restarting backfill on osd.4 from (0'0,0'0] MAX to 5381'11776 2013-05-23 22:19:44.530119 osd.6 [INF] 3.b8 restarting backfill on osd.4 from (0'0,0'0] MAX to 2344'3613 2013-05-23 22:19:44.549131 osd.6 [INF] 3.278 restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'11044 2013-05-23 22:19:44.892159 osd.6 [INF] 3.14d restarting backfill on osd.10 from (0'0,0'0] MAX to 4997'41453 2013-05-23 22:19:44.909577 osd.6 [INF] 3.bb restarting backfill on osd.4 from (0'0,0'0] MAX to 4969'40148 2013-05-23 22:19:44.999427 osd.6 [INF] 3.13c restarting backfill on osd.10 from (0'0,0'0] MAX to 4211'3501 2013-05-23 22:19:45.045473 osd.6 [INF] 3.9e restarting backfill on osd.10 from (0'0,0'0] MAX to 3583'4629 2013-05-23 22:19:46.636496 osd.6 [INF] 4.6c restarting backfill on osd.10 from (0'0,0'0] MAX to 108'4184 2013-05-23 22:19:48.961716 mon.0 [INF] pgmap v579257: 1152 pgs: 51 active+remapped+wait_backfill, 597 active+remapped, 3 active+remapped+backfilling, 33 active+recovery_wait+remapped, 468 remapped+peering; 2045 GB data, 7223 GB used, 18811 GB / 26035 GB avail; 1031MB/s rd, 1155MB/s wr, 329Kop/s; 230001/1775239 degraded (12.956%); recovering 922 o/s, 3615MB/s, 45 key/s 2013-05-23 22:19:39.114554 osd.11 [INF] 4.3 restarting backfill on osd.4 from (0'0,0'0] MAX to 108'7412 2013-05-23 22:19:39.122650 osd.11 [INF] 4.19 restarting backfill on osd.10 from (0'0,0'0] MAX to 108'10044 2013-05-23 22:19:39.162016 osd.11 [INF] 4.1d restarting backfill on osd.10 from (0'0,0'0] MAX to 108'4770 2013-05-23 22:19:39.172359 osd.11 [INF] 3.23a restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'3876 2013-05-23 22:19:39.203984 osd.11 [INF] 3.1c7 restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'27541 2013-05-23 22:19:39.224729 osd.11 [INF] 3.1ed restarting backfill on osd.10 from (0'0,0'0] MAX to 4224'3863 2013-05-23 22:19:39.875200 osd.11 [INF] 4.9 restarting backfill on osd.10 from (0'0,0'0] MAX to 108'5744 2013-05-23 22:19:39.876259 osd.11 [INF] 3.206 restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'4025 2013-05-23 22:19:39.947484 osd.11 [INF] 3.a restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'3317 2013-05-23 22:19:40.178258 osd.11 [INF] 3.1bb restarting backfill on osd.4 from (0'0,0'0] MAX to 2851'5372 2013-05-23 22:19:40.676751 osd.11 [INF] 3.19d restarting backfill on osd.10 from (0'0,0'0] MAX to 5869'10935 2013-05-23 22:19:40.865672 osd.11 [INF] 3.198 restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'10144 2013-05-23 22:19:40.890033 osd.11 [INF] 3.19a restarting backfill on osd.10 from (0'0,0'0] MAX to 2805'3727 2013-05-23 22:19:40.892570 osd.11 [INF] 3.8 restarting backfill on osd.4 from (0'0,0'0] MAX to 2797'3641 2013-05-23 22:19:40.978554 osd.11 [INF] 3.192 restarting backfill on osd.4 from (0'0,0'0] MAX to 3545'3677 2013-05-23 22:19:41.030634 osd.11 [INF] 3.18c restarting backfill on osd.4 from (0'0,0'0] MAX to 2805'3090 2013-05-23 22:19:41.196482 osd.11 [INF] 3.179 restarting backfill on osd.4 from (0'0,0'0] MAX to 5872'22699 2013-05-23 22:19:41.270408 osd.11 [INF] 3.157 restarting backfill on osd.4 from (0'0,0'0] MAX to 2851'3627 2013-05-23 22:19:43.603016 osd.11 [INF] 3.ea restarting backfill on osd.4 from (0'0,0'0] MAX to 3796'7373 2013-05-23 22:19:43.614986 osd.11 [INF] 3.3a2 restarting backfill on osd.4 from (0'0,0'0] MAX to 3796'5058 2013-05-23 22:19:43.640376 osd.11 [INF] 3.2bc restarting backfill on osd.10 from (0'0,0'0] MAX to 3796'3368 2013-05-23 22:19:43.676975 osd.11 [INF] 3.23d restarting backfill on osd.10 from (0'0,0'0] MAX to 2912'3374 2013-05-23 22:19:43.679544 osd.11 [INF] 3.a9 restarting backfill on osd.4 from (0'0,0'0] MAX to 5337'8846 2013-05-23 22:19:43.946233 osd.11 [INF] 3.3cf restarting backfill on osd.10 from (0'0,0'0] MAX to 3796'12134 2013-05-23 22:19:44.055712 osd.11 [INF] 3.3e6 restarting backfill on osd.4 from (0'0,0'0] MAX to 2848'4080 2013-05-23 22:19:44.056912 osd.11 [INF] 3.369 restarting backfill on osd.10 from (0'0,0'0] MAX to 4969'43134 2013-05-23 22:19:44.121028 osd.11 [INF] 3.3b2 restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'3380 2013-05-23 22:19:44.240330 osd.11 [INF] 3.3c7 restarting backfill on osd.4 from (0'0,0'0] MAX to 5263'8574 2013-05-23 22:19:44.278623 osd.11 [INF] 3.ff restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'3145 2013-05-23 22:19:44.408679 osd.11 [INF] 3.3dd restarting backfill on osd.4 from (0'0,0'0] MAX to 2851'4031 2013-05-23 22:19:44.442499 osd.11 [INF] 3.3f2 restarting backfill on osd.4 from (0'0,0'0] MAX to 5428'3404 2013-05-23 22:19:44.534910 osd.11 [INF] 3.b4 restarting backfill on osd.10 from (0'0,0'0] MAX to 4362'8818 2013-05-23 22:19:44.859680 osd.11 [INF] 3.87 restarting backfill on osd.4 from (0'0,0'0] MAX to 3579'4582 2013-05-23 22:19:45.610354 osd.11 [INF] 3.7a restarting backfill on osd.4 from (0'0,0'0] MAX to 4362'3083 2013-05-23 22:19:46.306291 osd.11 [INF] 3.275 restarting backfill on osd.10 from (0'0,0'0] MAX to 4185'5213 2013-05-23 22:19:47.418790 osd.11 [INF] 3.359 restarting backfill on osd.4 from (0'0,0'0] MAX to 5010'14586 2013-05-23 22:19:47.631500 osd.11 [INF] 3.71 restarting backfill on osd.10 from (0'0,0'0] MAX to 4126'4102 2013-05-23 22:19:47.848902 osd.11 [INF] 3.86 restarting backfill on osd.10 from (0'0,0'0] MAX to 5861'7895 2013-05-23 22:19:47.995831 osd.11 [INF] 3.351 restarting backfill on osd.10 from (0'0,0'0] MAX to 3645'6927 2013-05-23 22:19:48.169766 osd.11 [INF] 4.70 restarting backfill on osd.10 from (0'0,0'0] MAX to 108'4820 2013-05-23 22:19:40.776894 osd.0 [INF] 4.39 restarting backfill on osd.11 from (0'0,0'0] MAX to 108'4799 2013-05-23 22:19:40.786715 osd.0 [INF] 3.120 restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'3188 2013-05-23 22:19:40.811901 osd.0 [INF] 3.3a restarting backfill on osd.11 from (0'0,0'0] MAX to 3094'3685 2013-05-23 22:19:40.834608 osd.0 [INF] 3.3b0 restarting backfill on osd.6 from (0'0,0'0] MAX to 3094'7361 2013-05-23 22:19:40.836155 osd.0 [INF] 3.92 restarting backfill on osd.11 from (0'0,0'0] MAX to 5159'3208 2013-05-23 22:19:40.867921 osd.0 [INF] 3.2f restarting backfill on osd.7 from (0'0,0'0] MAX to 2803'3402 2013-05-23 22:19:40.877295 osd.0 [INF] 4.71 restarting backfill on osd.10 from (0'0,0'0] MAX to 108'4807 2013-05-23 22:19:40.907814 osd.0 [INF] 3.82 restarting backfill on osd.10 from (0'0,0'0] MAX to 3796'8031 2013-05-23 22:19:40.918532 osd.0 [INF] 3.42 restarting backfill on osd.7 from (0'0,0'0] MAX to 3094'3701 2013-05-23 22:19:40.950099 osd.0 [INF] 3.ad restarting backfill on osd.10 from (0'0,0'0] MAX to 2803'3659 2013-05-23 22:19:41.032433 osd.0 [INF] 3.27 restarting backfill on osd.4 from (0'0,0'0] MAX to 3796'3827 2013-05-23 22:19:41.036406 osd.0 [INF] 3.13e restarting backfill on osd.10 from (0'0,0'0] MAX to 2912'5257 2013-05-23 22:19:41.040051 osd.0 [INF] 3.286 restarting backfill on osd.11 from (0'0,0'0] MAX to 4969'46977 2013-05-23 22:19:41.052038 osd.0 [INF] 3.381 restarting backfill on osd.11 from (0'0,0'0] MAX to 5729'8070 2013-05-23 22:19:41.060205 osd.0 [INF] 3.21a restarting backfill on osd.7 from (0'0,0'0] MAX to 5878'6077 2013-05-23 22:19:41.083717 osd.0 [INF] 4.78 restarting backfill on osd.11 from (0'0,0'0] MAX to 108'4292 2013-05-23 22:19:41.093099 osd.0 [INF] 3.1c2 restarting backfill on osd.6 from (0'0,0'0] MAX to 3094'4502 2013-05-23 22:19:41.133591 osd.0 [INF] 4.26 restarting backfill on osd.4 from (0'0,0'0] MAX to 108'3362 2013-05-23 22:19:41.142137 osd.0 [INF] 3.79 restarting backfill on osd.11 from (0'0,0'0] MAX to 2912'3246 2013-05-23 22:19:41.150720 osd.0 [INF] 3.12f restarting backfill on osd.10 from (0'0,0'0] MAX to 455'3615 2013-05-23 22:19:41.175414 osd.0 [INF] 3.1d restarting backfill on osd.10 from (0'0,0'0] MAX to 2912'3263 2013-05-23 22:19:41.184555 osd.0 [INF] 3.28 restarting backfill on osd.11 from (0'0,0'0] MAX to 3094'3692 2013-05-23 22:19:41.187113 osd.0 [INF] 3.125 restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'4521 2013-05-23 22:19:41.212901 osd.0 [INF] 3.17 restarting backfill on osd.10 from (0'0,0'0] MAX to 4058'3972 2013-05-23 22:19:41.215400 osd.0 [INF] 4.27 restarting backfill on osd.11 from (0'0,0'0] MAX to 108'4876 2013-05-23 22:19:41.235731 osd.0 [INF] 3.105 restarting backfill on osd.6 from (0'0,0'0] MAX to 5873'7813 2013-05-23 22:19:41.251011 osd.0 [INF] 3.35a restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'3226 2013-05-23 22:19:41.295350 osd.0 [INF] 3.e8 restarting backfill on osd.6 from (0'0,0'0] MAX to 2759'3119 2013-05-23 22:19:41.328197 osd.0 [INF] 4.17 restarting backfill on osd.4 from (0'0,0'0] MAX to 108'3678 2013-05-23 22:19:41.338318 osd.0 [INF] 3.353 restarting backfill on osd.7 from (0'0,0'0] MAX to 413'3595 2013-05-23 22:19:41.342243 osd.0 [INF] 3.e6 restarting backfill on osd.10 from (0'0,0'0] MAX to 2851'6944 2013-05-23 22:19:41.366554 osd.0 [INF] 3.17d restarting backfill on osd.7 from (0'0,0'0] MAX to 2912'5116 2013-05-23 22:19:41.422935 osd.0 [INF] 3.37c restarting backfill on osd.10 from (0'0,0'0] MAX to 3796'3871 2013-05-23 22:19:41.455942 osd.10 [INF] 3.a6 restarting backfill on osd.6 from (0'0,0'0] MAX to 5467'13380 2013-05-23 22:19:41.470876 osd.10 [INF] 3.3f4 restarting backfill on osd.6 from (0'0,0'0] MAX to 5678'4562 2013-05-23 22:19:41.491517 osd.0 [INF] 3.d9 restarting backfill on osd.11 from (0'0,0'0] MAX to 3094'3598 2013-05-23 22:19:41.502952 osd.10 [INF] 3.39e restarting backfill on osd.11 from (0'0,0'0] MAX to 5844'5219 2013-05-23 22:19:41.522103 osd.10 [INF] 3.39f restarting backfill on osd.7 from (0'0,0'0] MAX to 5672'9074 2013-05-23 22:19:41.526149 osd.10 [INF] 3.3e9 restarting backfill on osd.11 from (0'0,0'0] MAX to 3796'5161 2013-05-23 22:19:41.530755 osd.0 [INF] 3.167 restarting backfill on osd.11 from (0'0,0'0] MAX to 3091'3845 2013-05-23 22:19:41.554222 osd.0 [INF] 3.398 restarting backfill on osd.6 from (0'0,0'0] MAX to 3094'56694 2013-05-23 22:19:41.558310 osd.0 [INF] 3.b5 restarting backfill on osd.6 from (0'0,0'0] MAX to 5011'41329 2013-05-23 22:19:41.574491 osd.10 [INF] 3.3e1 restarting backfill on osd.11 from (0'0,0'0] MAX to 3796'3046 2013-05-23 22:19:41.605306 osd.0 [INF] 3.3f5 restarting backfill on osd.6 from (0'0,0'0] MAX to 3094'3168 2013-05-23 22:19:41.617683 osd.0 [INF] 3.9c restarting backfill on osd.7 from (0'0,0'0] MAX to 2851'3254 2013-05-23 22:19:41.633436 osd.0 [INF] 3.3d1 restarting backfill on osd.6 from (0'0,0'0] MAX to 2912'4557 2013-05-23 22:19:41.642105 osd.0 [INF] 3.373 restarting backfill on osd.7 from (0'0,0'0] MAX to 3796'7509 2013-05-23 22:19:41.671765 osd.10 [INF] 3.121 restarting backfill on osd.7 from (0'0,0'0] MAX to 2912'4396 2013-05-23 22:19:41.705699 osd.10 [INF] 3.21 restarting backfill on osd.7 from (0'0,0'0] MAX to 5755'6872 2013-05-23 22:19:41.715090 osd.0 [INF] 3.228 restarting backfill on osd.6 from (0'0,0'0] MAX to 5115'6427 2013-05-23 22:19:41.729463 osd.0 [INF] 3.165 restarting backfill on osd.4 from (0'0,0'0] MAX to 3601'7301 2013-05-23 22:19:41.754815 osd.0 [INF] 3.239 restarting backfill on osd.7 from (0'0,0'0] MAX to 2805'37022 2013-05-23 22:19:41.763074 osd.10 [INF] 3.f6 restarting backfill on osd.6 from (0'0,0'0] MAX to 3796'4115 2013-05-23 22:19:41.790560 osd.10 [INF] 3.c9 restarting backfill on osd.6 from (0'0,0'0] MAX to 5861'5317 2013-05-23 22:19:41.826157 osd.10 [INF] 3.c1 restarting backfill on osd.6 from (0'0,0'0] MAX to 2912'10707 2013-05-23 22:19:41.840433 osd.0 [INF] 3.2e6 restarting backfill on osd.11 from (0'0,0'0] MAX to 2379'3271 2013-05-23 22:19:42.041605 osd.0 [INF] 3.1dc restarting backfill on osd.10 from (0'0,0'0] MAX to 2912'3260 2013-05-23 22:19:42.054123 osd.0 [INF] 3.3e restarting backfill on osd.4 from (0'0,0'0] MAX to 4809'23453 2013-05-23 22:19:42.088487 osd.0 [INF] 3.2d3 restarting backfill on osd.7 from (0'0,0'0] MAX to 4362'5931 2013-05-23 22:19:42.092029 osd.10 [INF] 3.1af restarting backfill on osd.7 from (0'0,0'0] MAX to 5873'11057 2013-05-23 22:19:42.132630 osd.0 [INF] 3.1ad restarting backfill on osd.4 from (0'0,0'0] MAX to 5672'11255 2013-05-23 22:19:42.137771 osd.0 [INF] 4.3d restarting backfill on osd.4 from (0'0,0'0] MAX to 108'4340 2013-05-23 22:19:42.146159 osd.0 [INF] 3.2aa restarting backfill on osd.7 from (0'0,0'0] MAX to 5869'6045 2013-05-23 22:19:42.165179 osd.0 [INF] 3.1d1 restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'11839 2013-05-23 22:19:42.177291 osd.10 [INF] 3.191 restarting backfill on osd.6 from (0'0,0'0] MAX to 2912'3154 2013-05-23 22:19:42.214385 osd.0 [INF] 3.288 restarting backfill on osd.6 from (0'0,0'0] MAX to 4362'3376 2013-05-23 22:19:42.255893 osd.0 [INF] 3.af restarting backfill on osd.4 from (0'0,0'0] MAX to 3796'4489 2013-05-23 22:19:42.277479 osd.10 [INF] 3.305 restarting backfill on osd.6 from (0'0,0'0] MAX to 4354'4082 2013-05-23 22:19:42.289025 osd.0 [INF] 3.332 restarting backfill on osd.7 from (0'0,0'0] MAX to 2912'22041 2013-05-23 22:19:42.337089 osd.0 [INF] 3.d8 restarting backfill on osd.10 from (0'0,0'0] MAX to 5873'22036 2013-05-23 22:19:42.426303 osd.10 [INF] 3.2f3 restarting backfill on osd.7 from (0'0,0'0] MAX to 3094'6516 2013-05-23 22:19:42.519142 osd.10 [INF] 3.16 restarting backfill on osd.11 from (0'0,0'0] MAX to 3796'82230 2013-05-23 22:19:42.639162 osd.0 [INF] 3.31c restarting backfill on osd.11 from (0'0,0'0] MAX to 3094'11425 2013-05-23 22:19:42.656366 osd.0 [INF] 3.46 restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'100045 2013-05-23 22:19:42.660355 osd.10 [INF] 3.97 restarting backfill on osd.11 from (0'0,0'0] MAX to 3094'17230 2013-05-23 22:19:42.695018 osd.0 [INF] 3.e7 restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'6017 2013-05-23 22:19:42.742689 osd.0 [INF] 3.2d0 restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'4345 2013-05-23 22:19:42.763865 osd.0 [INF] 3.38 restarting backfill on osd.11 from (0'0,0'0] MAX to 5877'11919 2013-05-23 22:19:42.791822 osd.0 [INF] 3.ed restarting backfill on osd.10 from (0'0,0'0] MAX to 3094'15197 2013-05-23 22:19:42.801980 osd.10 [INF] 4.7a restarting backfill on osd.11 from (0'0,0'0] MAX to 108'6929 2013-05-23 22:19:42.810202 osd.8 [INF] 3.dd restarting backfill on osd.4 from (0'0,0'0] MAX to 5869'13851 2013-05-23 22:19:42.833006 osd.8 [INF] 3.db restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'3225 2013-05-23 22:19:42.871582 osd.10 [INF] 3.7b restarting backfill on osd.11 from (0'0,0'0] MAX to 2912'5066 2013-05-23 22:19:42.906532 osd.0 [INF] 3.2ed restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'4174 2013-05-23 22:19:42.928855 osd.0 [INF] 3.3fc restarting backfill on osd.11 from (0'0,0'0] MAX to 839'4148 2013-05-23 22:19:42.932208 osd.0 [INF] 3.1e3 restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'9077 2013-05-23 22:19:42.962886 osd.10 [INF] 3.20e restarting backfill on osd.7 from (0'0,0'0] MAX to 2851'3291 2013-05-23 22:19:42.972540 osd.0 [INF] 3.319 restarting backfill on osd.4 from (0'0,0'0] MAX to 2912'4926 2013-05-23 22:19:42.984323 osd.10 [INF] 3.204 restarting backfill on osd.7 from (0'0,0'0] MAX to 2912'6188 2013-05-23 22:19:42.995441 osd.0 [INF] 3.1fc restarting backfill on osd.4 from (0'0,0'0] MAX to 5648'24563 2013-05-23 22:19:43.009230 osd.0 [INF] 3.19 restarting backfill on osd.6 from (0'0,0'0] MAX to 2344'4294 2013-05-23 22:19:43.059751 osd.10 [INF] 3.69 restarting backfill on osd.6 from (0'0,0'0] MAX to 4750'7888 2013-05-23 22:19:43.071340 osd.10 [INF] 3.1cb restarting backfill on osd.7 from (0'0,0'0] MAX to 4969'48846 2013-05-23 22:19:43.253085 osd.10 [INF] 4.66 restarting backfill on osd.6 from (0'0,0'0] MAX to 108'9436 2013-05-23 22:19:43.389134 osd.10 [INF] 4.5b restarting backfill on osd.6 from (0'0,0'0] MAX to 108'4801 2013-05-23 22:19:44.352718 osd.0 [INF] 3.3f8 restarting backfill on osd.11 from (0'0,0'0] MAX to 4354'12738 2013-05-23 22:19:44.416537 osd.0 [INF] 4.18 restarting backfill on osd.6 from (0'0,0'0] MAX to 108'7307 2013-05-23 22:19:44.492935 osd.0 [INF] 4.32 restarting backfill on osd.7 from (0'0,0'0] MAX to 108'4864 2013-05-23 22:19:44.791004 osd.10 [INF] 3.37a restarting backfill on osd.6 from (0'0,0'0] MAX to 3094'4242 2013-05-23 22:19:44.835531 osd.0 [INF] 4.57 restarting backfill on osd.6 from (0'0,0'0] MAX to 108'4978 2013-05-23 22:19:44.919977 osd.0 [INF] 4.51 restarting backfill on osd.7 from (0'0,0'0] MAX to 108'6101 2013-05-23 22:19:45.045845 osd.0 [INF] 3.389 restarting backfill on osd.11 from (0'0,0'0] MAX to 5849'9717 2013-05-23 22:19:45.473304 osd.4 [INF] 3.10 restarting backfill on osd.7 from (0'0,0'0] MAX to 3094'3009 2013-05-23 22:19:45.691729 osd.4 [INF] 3.3f7 restarting backfill on osd.7 from (0'0,0'0] MAX to 2777'3446 2013-05-23 22:19:45.717376 osd.4 [INF] 3.3f0 restarting backfill on osd.7 from (0'0,0'0] MAX to 4700'3953 2013-05-23 22:19:45.927160 osd.4 [INF] 3.3c9 restarting backfill on osd.11 from (0'0,0'0] MAX to 2851'3184 2013-05-23 22:19:46.264833 osd.4 [INF] 3.383 restarting backfill on osd.7 from (0'0,0'0] MAX to 5011'40065 2013-05-23 22:19:46.603380 osd.4 [INF] 3.3d9 restarting backfill on osd.7 from (0'0,0'0] MAX to 5513'59533 2013-05-23 22:19:47.432688 osd.10 [INF] 3.2c0 restarting backfill on osd.11 from (0'0,0'0] MAX to 4997'41265 2013-05-23 22:19:47.527050 osd.10 [INF] 3.365 restarting backfill on osd.6 from (0'0,0'0] MAX to 2795'4146 2013-05-23 22:19:47.605751 osd.10 [INF] 3.363 restarting backfill on osd.6 from (0'0,0'0] MAX to 3796'8534 2013-05-23 22:19:47.641516 osd.0 [INF] 3.185 restarting backfill on osd.11 from (0'0,0'0] MAX to 4354'3102 2013-05-23 22:19:47.708665 osd.0 [INF] 3.2dd restarting backfill on osd.4 from (0'0,0'0] MAX to 4354'3535 2013-05-23 22:19:47.727215 osd.0 [INF] 3.f0 restarting backfill on osd.6 from (0'0,0'0] MAX to 4354'5490 2013-05-23 22:19:47.735060 osd.0 [INF] 3.315 restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'4914 2013-05-23 22:19:47.754229 osd.0 [INF] 3.f1 restarting backfill on osd.11 from (0'0,0'0] MAX to 2912'4414 2013-05-23 22:19:47.761694 osd.0 [INF] 3.334 restarting backfill on osd.4 from (0'0,0'0] MAX to 3796'4605 2013-05-23 22:19:47.784072 osd.0 [INF] 3.1b5 restarting backfill on osd.6 from (0'0,0'0] MAX to 2797'3918 2013-05-23 22:19:47.792528 osd.0 [INF] 3.284 restarting backfill on osd.10 from (0'0,0'0] MAX to 3625'6048 2013-05-23 22:19:47.811551 osd.0 [INF] 3.20d restarting backfill on osd.7 from (0'0,0'0] MAX to 4969'42745 2013-05-23 22:19:47.830667 osd.0 [INF] 3.2e8 restarting backfill on osd.6 from (0'0,0'0] MAX to 4362'4948 2013-05-23 22:19:47.851492 osd.0 [INF] 3.99 restarting backfill on osd.7 from (0'0,0'0] MAX to 2805'3266 2013-05-23 22:19:47.853737 osd.0 [INF] 3.5d restarting backfill on osd.11 from (0'0,0'0] MAX to 2767'3292 2013-05-23 22:19:47.870264 osd.0 [INF] 3.27c restarting backfill on osd.4 from (0'0,0'0] MAX to 3796'4563 2013-05-23 22:19:47.881296 osd.0 [INF] 4.73 restarting backfill on osd.11 from (0'0,0'0] MAX to 108'6714 2013-05-23 22:19:47.887520 osd.0 [INF] 3.110 restarting backfill on osd.7 from (0'0,0'0] MAX to 2912'3998 2013-05-23 22:19:47.891452 osd.0 [INF] 3.29c restarting backfill on osd.4 from (0'0,0'0] MAX to 3094'8929 2013-05-23 22:19:47.914915 osd.10 [INF] 3.295 restarting backfill on osd.7 from (0'0,0'0] MAX to 2912'3451 2013-05-23 22:19:47.943862 osd.0 [INF] 3.35d restarting backfill on osd.6 from (0'0,0'0] MAX to 2912'6042 2013-05-23 22:19:47.971668 osd.0 [INF] 3.e9 restarting backfill on osd.11 from (0'0,0'0] MAX to 3100'4466 2013-05-23 22:19:48.012808 osd.0 [INF] 3.29f restarting backfill on osd.4 from (0'0,0'0] MAX to 5709'4827 2013-05-23 22:19:48.044762 osd.0 [INF] 3.362 restarting backfill on osd.11 from (0'0,0'0] MAX to 2851'3537 2013-05-23 22:19:48.050642 osd.4 [INF] 3.cd restarting backfill on osd.11 from (0'0,0'0] MAX to 4432'20736 2013-05-23 22:19:48.077218 osd.0 [INF] 3.1ae restarting backfill on osd.6 from (0'0,0'0] MAX to 3094'3109 2013-05-23 22:19:48.087018 osd.0 [INF] 3.2df restarting backfill on osd.11 from (0'0,0'0] MAX to 2851'3131 2013-05-23 22:19:48.111515 osd.10 [INF] 3.32f restarting backfill on osd.11 from (0'0,0'0] MAX to 5873'6008 2013-05-23 22:19:48.114865 osd.10 [INF] 3.291 restarting backfill on osd.7 from (0'0,0'0] MAX to 3094'7165 2013-05-23 22:19:48.169766 osd.11 [INF] 4.70 restarting backfill on osd.10 from (0'0,0'0] MAX to 108'4820 2013-05-23 22:19:48.189861 osd.10 [INF] 3.329 restarting backfill on osd.11 from (0'0,0'0] MAX to 4354'3681 2013-05-23 22:19:48.193240 osd.0 [INF] 3.1ff restarting backfill on osd.11 from (0'0,0'0] MAX to 2797'6475 2013-05-23 22:19:48.197272 osd.0 [INF] 3.2b1 restarting backfill on osd.7 from (0'0,0'0] MAX to 2912'4400 2013-05-23 22:19:48.218691 osd.10 [INF] 3.28d restarting backfill on osd.7 from (0'0,0'0] MAX to 3796'6580 2013-05-23 22:19:48.282935 osd.10 [INF] 3.327 restarting backfill on osd.11 from (0'0,0'0] MAX to 2912'4293 2013-05-23 22:19:48.298469 osd.0 [INF] 3.236 restarting backfill on osd.6 from (0'0,0'0] MAX to 3094'4341 2013-05-23 22:19:48.299570 osd.4 [INF] 3.b7 restarting backfill on osd.6 from (0'0,0'0] MAX to 4476'6485 2013-05-23 22:19:48.301902 osd.0 [INF] 3.2cd restarting backfill on osd.6 from (0'0,0'0] MAX to 2912'4216 2013-05-23 22:19:48.465441 osd.4 [INF] 3.ab restarting backfill on osd.11 from (0'0,0'0] MAX to 3096'4952 2013-05-23 22:19:48.714662 osd.0 [INF] 3.248 restarting backfill on osd.4 from (0'0,0'0] MAX to 2851'3787 2013-05-23 22:19:48.879584 osd.0 [INF] 3.273 restarting backfill on osd.6 from (0'0,0'0] MAX to 2912'6769 2013-05-23 22:19:48.886742 osd.4 [INF] 4.7d restarting backfill on osd.11 from (0'0,0'0] MAX to 108'7505 2013-05-23 22:19:49.256650 osd.4 [INF] 4.76 restarting backfill on osd.6 from (0'0,0'0] MAX to 108'4802 2013-05-23 22:19:49.313238 osd.4 [INF] 3.391 restarting backfill on osd.7 from (0'0,0'0] MAX to 3094'3280 2013-05-23 22:19:49.786392 osd.4 [INF] 3.38b restarting backfill on osd.7 from (0'0,0'0] MAX to 2912'5795 2013-05-23 22:19:49.870849 osd.10 [INF] 3.276 restarting backfill on osd.7 from (0'0,0'0] MAX to 5001'42017 2013-05-23 22:19:49.883646 osd.4 [INF] 3.37e restarting backfill on osd.7 from (0'0,0'0] MAX to 3094'3186 2013-05-23 22:19:50.042622 osd.10 [INF] 3.269 restarting backfill on osd.11 from (0'0,0'0] MAX to 2912'3852 2013-05-23 22:19:50.610780 osd.11 [INF] 4.30 restarting backfill on osd.10 from (0'0,0'0] MAX to 108'7100 2013-05-23 22:19:50.820390 osd.4 [INF] 3.346 restarting backfill on osd.7 from (0'0,0'0] MAX to 2912'7602 2013-05-23 22:19:50.933961 osd.4 [INF] 4.61 restarting backfill on osd.7 from (0'0,0'0] MAX to 108'3770 2013-05-23 22:19:50.987771 osd.10 [INF] 3.40 restarting backfill on osd.6 from (0'0,0'0] MAX to 5873'6120 2013-05-23 22:19:51.111874 osd.10 [INF] 3.238 restarting backfill on osd.11 from (0'0,0'0] MAX to 5159'23542 2013-05-23 22:19:51.171032 osd.6 [INF] 3.379 restarting backfill on osd.4 from (0'0,0'0] MAX to 3796'4918 2013-05-23 22:19:51.252716 osd.6 [INF] 3.35f restarting backfill on osd.4 from (0'0,0'0] MAX to 4354'3826 2013-05-23 22:19:51.340786 osd.6 [INF] 3.343 restarting backfill on osd.10 from (0'0,0'0] MAX to 2918'3079 2013-05-23 22:19:51.405381 osd.6 [INF] 3.32c restarting backfill on osd.10 from (0'0,0'0] MAX to 2761'19806 2013-05-23 22:19:51.424254 osd.6 [INF] 3.32d restarting backfill on osd.10 from (0'0,0'0] MAX to 4985'45103 2013-05-23 22:19:52.766927 osd.4 [INF] 3.32b restarting backfill on osd.6 from (0'0,0'0] MAX to 2069'3421 2013-05-23 22:19:52.868939 osd.4 [INF] 3.350 restarting backfill on osd.6 from (0'0,0'0] MAX to 455'5562 2013-05-23 22:19:59.440121 mon.0 [INF] pgmap v579258: 1152 pgs: 53 active+remapped+wait_backfill, 770 active+remapped, 3 active+remapped+backfilling, 38 active+recovery_wait+remapped, 288 remapped+peering; 2050 GB data, 7223 GB used, 18811 GB / 26035 GB avail; 2678MB/s rd, 2424MB/s wr, 318Kop/s; 310147/1778944 degraded (17.434%); recovering 1944 o/s, 7682MB/s, 43 key/s 2013-05-23 22:19:50.898302 osd.0 [INF] 3.2cc restarting backfill on osd.10 from (0'0,0'0] MAX to 2356'4608 2013-05-23 22:19:51.712664 osd.10 [INF] 4.38 restarting backfill on osd.6 from (0'0,0'0] MAX to 108'6329 2013-05-23 22:19:51.809704 osd.10 [INF] 4.3f restarting backfill on osd.6 from (0'0,0'0] MAX to 108'4566 2013-05-23 22:20:00.504876 mon.0 [INF] pgmap v579259: 1152 pgs: 72 active+remapped+wait_backfill, 928 active+remapped, 4 active+remapped+backfilling, 57 active+recovery_wait+remapped, 91 remapped+peering; 2147 GB data, 7223 GB used, 18811 GB / 26035 GB avail; 8792MB/s rd, 10156MB/s wr, 398Kop/s; 428186/1853983 degraded (23.095%); recovering 4210 o/s, 16608MB/s, 49 key/s 2013-05-23 22:20:01.659453 mon.0 [INF] pgmap v579260: 1152 pgs: 73 active+remapped+wait_backfill, 927 active+remapped, 4 active+remapped+backfilling, 58 active+recovery_wait+remapped, 90 remapped+peering; 2147 GB data, 7223 GB used, 18811 GB / 26035 GB avail; 9011MB/s rd, 10267MB/s wr, 418Kop/s; 428423/1854220 degraded (23.105%); recovering 4022 o/s, 15853MB/s, 55 key/s 2013-05-23 22:19:55.501530 osd.0 [INF] 3.240 restarting backfill on osd.11 from (0'0,0'0] MAX to 2912'5334 2013-05-23 22:20:02.834028 mon.0 [INF] pgmap v579261: 1152 pgs: 75 active+remapped+wait_backfill, 931 active+remapped, 4 active+remapped+backfilling, 62 active+recovery_wait+remapped, 80 remapped+peering; 2167 GB data, 7223 GB used, 18811 GB / 26035 GB avail; 9495MB/s rd, 10288MB/s wr, 1093Kop/s; 438156/1869865 degraded (23.432%); recovering 2499 o/s, 9833MB/s, 136 key/s 2013-05-23 22:20:04.790648 mon.0 [INF] pgmap v579262: 1152 pgs: 82 active+remapped+wait_backfill, 917 active+remapped, 4 active+remapped+backfilling, 69 active+recovery_wait+remapped, 80 remapped+peering; 2167 GB data, 7223 GB used, 18811 GB / 26035 GB avail; 5713MB/s rd, 7511MB/s wr, 832Kop/s; 438156/1869865 degraded (23.432%); recovering 1859 o/s, 7314MB/s, 90 key/s 2013-05-23 22:20:03.406699 osd.7 [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.447075 secs 2013-05-23 22:20:03.406710 osd.7 [WRN] slow request 30.447075 seconds old, received at 2013-05-23 22:19:32.959585: osd_op(client.40022.0:36986 rbd_header.11302ae8944a [watch add cookie 1 ver 0] 3.c19a3ea1 e5880) v4 currently waiting for missing object 2013-05-23 22:20:06.849984 mon.0 [INF] pgmap v579263: 1152 pgs: 85 active+remapped+wait_backfill, 913 active+remapped, 4 active+remapped+backfilling, 71 active+recovery_wait+remapped, 79 remapped+peering; 2167 GB data, 7223 GB used, 18811 GB / 26035 GB avail; 901MB/s rd, 289MB/s wr, 4086op/s; 438423/1870132 degraded (23.443%); recovering 23 o/s, 96913KB/s 2013-05-23 22:20:08.266846 mon.0 [INF] pgmap v579264: 1152 pgs: 100 active+remapped+wait_backfill, 896 active+remapped, 5 active+remapped+backfilling, 78 active+recovery_wait+remapped, 73 remapped+peering; 2181 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 3322MB/s rd, 4583MB/s wr, 182Kop/s; 446434/1880374 degraded (23.742%); recovering 1326 o/s, 5235MB/s, 23 key/s 2013-05-23 22:20:09.435223 mon.0 [INF] pgmap v579265: 1152 pgs: 122 active+remapped+wait_backfill, 880 active+remapped, 5 active+remapped+backfilling, 89 active+recovery_wait+remapped, 56 remapped+peering; 2217 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 13595MB/s rd, 21479MB/s wr, 1310Kop/s; 459469/1908514 degraded (24.075%); recovering 6778 o/s, 26784MB/s, 91 key/s 2013-05-23 22:20:09.663189 osd.0 [INF] 3.3d3 restarting backfill on osd.4 from (0'0,0'0] MAX to 5011'39776 2013-05-23 22:20:10.549284 mon.0 [INF] pgmap v579266: 1152 pgs: 122 active+remapped+wait_backfill, 877 active+remapped, 5 active+remapped+backfilling, 92 active+recovery_wait+remapped, 56 remapped+peering; 2217 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 10646MB/s rd, 15840MB/s wr, 1074Kop/s; 459469/1908514 degraded (24.075%); recovering 5048 o/s, 19959MB/s, 59 key/s 2013-05-23 22:20:11.699556 mon.0 [INF] pgmap v579267: 1152 pgs: 131 active+remapped+wait_backfill, 865 active+remapped, 5 active+remapped+backfilling, 103 active+recovery_wait+remapped, 48 remapped+peering; 2234 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 4643MB/s rd, 8628MB/s wr, 234Kop/s; 470911/1922179 degraded (24.499%); recovering 3752 o/s, 14757MB/s, 42 key/s 2013-05-23 22:20:12.829468 mon.0 [INF] pgmap v579268: 1152 pgs: 138 active+remapped+wait_backfill, 856 active+remapped, 5 active+remapped+backfilling, 105 active+recovery_wait+remapped, 48 remapped+peering; 2234 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 4309MB/s rd, 8009MB/s wr, 217Kop/s; 470911/1922179 degraded (24.499%); recovering 3482 o/s, 13698MB/s, 39 key/s 2013-05-23 22:20:14.859068 mon.0 [INF] pgmap v579269: 1152 pgs: 158 active+remapped+wait_backfill, 822 active+remapped, 5 active+remapped+backfilling, 123 active+recovery_wait+remapped, 44 remapped+peering; 2241 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 778MB/s rd, 3035MB/s wr, 8232op/s; 473267/1927462 degraded (24.554%); recovering 1639 o/s, 6513MB/s 2013-05-23 22:20:15.581123 mon.0 [INF] pgmap v579270: 1152 pgs: 158 active+remapped+wait_backfill, 818 active+remapped, 5 active+remapped+backfilling, 127 active+recovery_wait+remapped, 44 remapped+peering; 2241 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 742MB/s rd, 2893MB/s wr, 7847op/s; 473267/1927462 degraded (24.554%); recovering 1562 o/s, 6208MB/s 2013-05-23 22:20:08.411250 osd.7 [WRN] 2 slow requests, 1 included below; oldest blocked for > 35.451629 secs 2013-05-23 22:20:08.411257 osd.7 [WRN] slow request 30.616485 seconds old, received at 2013-05-23 22:19:37.794729: osd_op(client.40022.0:36987 rbd_header.11302ae8944a [watch add cookie 1 ver 0] 3.c19a3ea1 e5883) v4 currently waiting for missing object 2013-05-23 22:20:16.603939 mon.0 [INF] pgmap v579271: 1152 pgs: 171 active+remapped+wait_backfill, 801 active+remapped, 5 active+remapped+backfilling, 139 active+recovery_wait+remapped, 36 remapped+peering; 2259 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 3566MB/s rd, 7359MB/s wr, 398Kop/s; 483651/1941349 degraded (24.913%); recovering 4485 o/s, 17696MB/s, 28 key/s 2013-05-23 22:20:17.820899 mon.0 [INF] pgmap v579272: 1152 pgs: 171 active+remapped+wait_backfill, 805 active+remapped, 5 active+remapped+backfilling, 139 active+recovery_wait+remapped, 32 remapped+peering; 2268 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 5421MB/s rd, 12071MB/s wr, 717Kop/s; 487150/1948384 degraded (25.003%); recovering 5630 o/s, 22168MB/s, 45 key/s 2013-05-23 22:20:18.910007 mon.0 [INF] pgmap v579273: 1152 pgs: 195 active+remapped+wait_backfill, 771 active+remapped, 5 active+remapped+backfilling, 168 active+recovery_wait+remapped, 13 remapped+peering; 2309 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 13246MB/s rd, 23768MB/s wr, 1432Kop/s; 510662/1979995 degraded (25.791%); recovering 12056 o/s, 47615MB/s, 114 key/s 2013-05-23 22:20:24.681711 mon.0 [INF] pgmap v579274: 1152 pgs: 195 active+remapped+wait_backfill, 759 active+remapped, 5 active+remapped+backfilling, 180 active+recovery_wait+remapped, 13 remapped+peering; 2309 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 9116MB/s rd, 15293MB/s wr, 889Kop/s; 510662/1979995 degraded (25.791%); recovering 8833 o/s, 34925MB/s, 78 key/s 2013-05-23 22:20:24.903123 mon.0 [INF] pgmap v579275: 1152 pgs: 232 active+remapped+wait_backfill, 706 active+remapped, 6 active+remapped+backfilling, 205 active+recovery_wait+remapped, 3 remapped+peering; 2331 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 1834MB/s rd, 4116MB/s wr, 85225op/s; 525794/1997449 degraded (26.323%); recovering 950 o/s, 3727MB/s 2013-05-23 22:20:26.113547 mon.0 [INF] pgmap v579276: 1152 pgs: 241 active+remapped+wait_backfill, 687 active+remapped, 6 active+remapped+backfilling, 215 active+recovery_wait+remapped, 3 remapped+peering; 2331 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 2040MB/s rd, 4580MB/s wr, 94816op/s; 525794/1997449 degraded (26.323%); recovering 1057 o/s, 4147MB/s 2013-05-23 22:20:27.460338 mon.0 [INF] pgmap v579277: 1152 pgs: 251 active+remapped+wait_backfill, 671 active+remapped, 6 active+remapped+backfilling, 221 active+recovery_wait+remapped, 3 remapped+peering; 2331 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 12426B/s rd, 4op/s; 525794/1997449 degraded (26.323%) 2013-05-23 22:20:28.644313 mon.0 [INF] pgmap v579278: 1152 pgs: 259 active+remapped+wait_backfill, 635 active+remapped, 6 active+remapped+backfilling, 249 active+recovery_wait+remapped, 3 remapped+peering; 2331 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 525756/1997449 degraded (26.321%); recovering 14 o/s, 57761KB/s, 1 key/s 2013-05-23 22:20:22.459370 osd.0 [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.428462 secs 2013-05-23 22:20:22.459385 osd.0 [WRN] slow request 30.428462 seconds old, received at 2013-05-23 22:19:52.030855: osd_op(client.40022.0:36991 rbd_data.11302ae8944a.000000000000002e [write 3194880~16384] 3.db9f7a1a e5885) v4 currently no flag points reached 2013-05-23 22:20:29.986001 mon.0 [INF] pgmap v579279: 1152 pgs: 259 active+remapped+wait_backfill, 638 active+remapped, 6 active+remapped+backfilling, 249 active+recovery_wait+remapped; 2336 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 2197MB/s rd, 2073MB/s wr, 49212op/s; 526976/2001103 degraded (26.334%); recovering 638 o/s, 2523MB/s, 1 key/s 2013-05-23 22:20:30.303503 osd.0 [WRN] 2 slow requests, 1 included below; oldest blocked for > 38.272615 secs 2013-05-23 22:20:30.303511 osd.0 [WRN] slow request 32.603376 seconds old, received at 2013-05-23 22:19:57.700094: osd_sub_op(osd.7.0:7210 3.2a1 c19a3ea1/rbd_header.11302ae8944a/head//3 [pull] v 2851'2119 snapset=0=[]:[] snapc=0=[]) v7 currently no flag points reached 2013-05-23 22:20:31.303458 mon.0 [INF] pgmap v579280: 1152 pgs: 320 active+remapped+wait_backfill, 577 active+remapped, 6 active+remapped+backfilling, 249 active+recovery_wait+remapped; 2336 GB data, 7224 GB used, 18811 GB / 26035 GB avail; 2389MB/s rd, 2254MB/s wr, 53516op/s; 526976/2001103 degraded (26.334%); recovering 678 o/s, 2686MB/s 2013-05-23 22:20:32.636809 mon.0 [INF] pgmap v579281: 1152 pgs: 320 active+remapped+wait_backfill, 576 active+remapped, 6 active+remapped+backfilling, 250 active+recovery_wait+remapped; 2336 GB data, 7224 GB used, 18810 GB / 26035 GB avail; 10850B/s rd, 4650B/s wr, 4op/s; 526976/2001103 degraded (26.334%) 2013-05-23 22:20:33.987548 mon.0 [INF] pgmap v579282: 1152 pgs: 320 active+remapped+wait_backfill, 564 active+remapped, 6 active+remapped+backfilling, 262 active+recovery_wait+remapped; 2336 GB data, 7224 GB used, 18810 GB / 26035 GB avail; 3149B/s wr, 0op/s; 526930/2001103 degraded (26.332%); recovering 17 o/s, 72432KB/s 2013-05-23 22:20:35.162747 mon.0 [INF] pgmap v579283: 1152 pgs: 320 active+remapped+wait_backfill, 564 active+remapped, 6 active+remapped+backfilling, 262 active+recovery_wait+remapped; 2336 GB data, 7224 GB used, 18810 GB / 26035 GB avail; 4832B/s wr, 0op/s; 526929/2001103 degraded (26.332%); recovering 18 o/s, 74552KB/s 2013-05-23 22:20:33.419766 osd.7 [WRN] 2 slow requests, 1 included below; oldest blocked for > 60.460145 secs 2013-05-23 22:20:33.419774 osd.7 [WRN] slow request 60.460145 seconds old, received at 2013-05-23 22:19:32.959585: osd_op(client.40022.0:36986 rbd_header.11302ae8944a [watch add cookie 1 ver 0] 3.c19a3ea1 e5880) v4 currently waiting for missing object 2013-05-23 22:20:38.514219 mon.0 [INF] pgmap v579284: 1152 pgs: 339 active+remapped+wait_backfill, 528 active+remapped, 6 active+remapped+backfilling, 279 active+recovery_wait+remapped; 2336 GB data, 7224 GB used, 18810 GB / 26035 GB avail; 11082B/s rd, 4749B/s wr, 5op/s; 526929/2001103 degraded (26.332%); recovering 1 o/s, 3608KB/s 2013-05-23 22:20:39.863578 mon.0 [INF] pgmap v579285: 1152 pgs: 339 active+remapped+wait_backfill, 525 active+remapped, 6 active+remapped+backfilling, 282 active+recovery_wait+remapped; 2336 GB data, 7224 GB used, 18810 GB / 26035 GB avail; 7344B/s rd, 1836B/s wr, 5op/s; 526928/2001103 degraded (26.332%); recovering 0 o/s, 1836KB/s, 1 key/s 2013-05-23 22:20:41.039193 mon.0 [INF] pgmap v579286: 1152 pgs: 343 active+remapped+wait_backfill, 488 active+remapped, 6 active+remapped+backfilling, 315 active+recovery_wait+remapped; 2336 GB data, 7224 GB used, 18810 GB / 26035 GB avail; 1125B/s rd, 12602B/s wr, 4op/s; 526856/2001103 degraded (26.328%); recovering 16 o/s, 64285KB/s, 1 key/s 2013-05-23 22:20:42.281002 mon.0 [INF] pgmap v579287: 1152 pgs: 343 active+remapped+wait_backfill, 487 active+remapped, 6 active+remapped+backfilling, 316 active+recovery_wait+remapped; 2336 GB data, 7224 GB used, 18810 GB / 26035 GB avail; 22510B/s rd, 31593B/s wr, 30op/s; 526856/2001103 degraded (26.328%); recovering 28 o/s, 110MB/s 2013-05-23 22:20:36.099096 osd.4 [WRN] 3 slow requests, 3 included below; oldest blocked for > 30.445684 secs 2013-05-23 22:20:36.099105 osd.4 [WRN] slow request 30.445684 seconds old, received at 2013-05-23 22:20:05.653356: osd_op(client.39941.0:2702698 windows2008.rbd [stat] 3.3f34261b e5885) v4 currently waiting for missing object 2013-05-23 22:20:36.099115 osd.4 [WRN] slow request 30.436366 seconds old, received at 2013-05-23 22:20:05.662674: osd_op(client.40005.0:2698817 windows2008.rbd [stat] 3.3f34261b e5885) v4 currently waiting for missing object 2013-05-23 22:20:36.099122 osd.4 [WRN] slow request 30.426557 seconds old, received at 2013-05-23 22:20:05.672483: osd_op(client.43802.0:59339 windows2008.rbd [stat] 3.3f34261b e5885) v4 currently waiting for missing object 2013-05-23 22:20:43.327921 mon.0 [INF] pgmap v579288: 1152 pgs: 342 active+remapped+wait_backfill, 487 active+remapped, 7 active+remapped+backfilling, 316 active+recovery_wait+remapped; 2336 GB data, 7225 GB used, 18810 GB / 26035 GB avail; 586KB/s rd, 19420B/s wr, 187op/s; 526814/2001103 degraded (26.326%); recovering 18 o/s, 60028KB/s, 8 key/s 2013-05-23 22:20:44.470020 mon.0 [INF] pgmap v579289: 1152 pgs: 342 active+remapped+wait_backfill, 487 active+remapped, 7 active+remapped+backfilling, 316 active+recovery_wait+remapped; 2336 GB data, 7225 GB used, 18810 GB / 26035 GB avail; 1546KB/s rd, 23131B/s wr, 287op/s; 526806/2001103 degraded (26.326%); recovering 21 o/s, 60499KB/s, 19 key/s 2013-05-23 22:20:45.677978 mon.0 [INF] pgmap v579290: 1152 pgs: 342 active+remapped+wait_backfill, 487 active+remapped, 7 active+remapped+backfilling, 316 active+recovery_wait+remapped; 2336 GB data, 7225 GB used, 18810 GB / 26035 GB avail; 4417KB/s rd, 16108B/s wr, 521op/s; 526796/2001103 degraded (26.325%); recovering 7 o/s, 1789KB/s, 17 key/s 2013-05-23 22:20:57.565406 mon.0 [INF] mon.0 calling new monitor election 2013-05-23 22:21:02.722718 mon.0 [INF] mon.0@0 won leader election with quorum 0,1 2013-05-23 22:21:03.470766 mon.0 [INF] pgmap v579290: 1152 pgs: 342 active+remapped+wait_backfill, 487 active+remapped, 7 active+remapped+backfilling, 316 active+recovery_wait+remapped; 2336 GB data, 7225 GB used, 18810 GB / 26035 GB avail; 4417KB/s rd, 16108B/s wr, 521op/s; 526796/2001103 degraded (26.325%); recovering 7 o/s, 1789KB/s, 17 key/s 2013-05-23 22:21:03.470886 mon.0 [INF] mdsmap e1: 0/0/1 up 2013-05-23 22:21:03.527904 mon.0 [INF] osdmap e5885: 7 osds: 7 up, 5 in 2013-05-23 22:21:03.528273 mon.0 [INF] monmap e1: 3 mons at {0=10.6.0.1:6789/0,2=10.6.0.3:6789/0,3=10.6.0.4:6789/0} 2013-05-23 22:21:05.079219 mon.0 [INF] mon.0 calling new monitor election 2013-05-23 22:20:59.776594 mon.2 [INF] mon.3 calling new monitor election 2013-05-23 22:21:05.578990 mon.1 [INF] mon.2 calling new monitor election 2013-05-23 22:21:06.198124 mon.0 [INF] mon.0@0 won leader election with quorum 0,1,2 2013-05-23 22:21:07.320191 mon.0 [INF] pgmap v579290: 1152 pgs: 342 active+remapped+wait_backfill, 487 active+remapped, 7 active+remapped+backfilling, 316 active+recovery_wait+remapped; 2336 GB data, 7225 GB used, 18810 GB / 26035 GB avail; 4417KB/s rd, 16108B/s wr, 521op/s; 526796/2001103 degraded (26.325%); recovering 7 o/s, 1789KB/s, 17 key/s 2013-05-23 22:21:07.320328 mon.0 [INF] mdsmap e1: 0/0/1 up 2013-05-23 22:21:07.760295 mon.0 [INF] osdmap e5886: 7 osds: 7 up, 5 in 2013-05-23 22:21:07.760515 mon.0 [INF] monmap e1: 3 mons at {0=10.6.0.1:6789/0,2=10.6.0.3:6789/0,3=10.6.0.4:6789/0} 2013-05-23 22:21:08.500946 mon.0 [INF] pgmap v579291: 1152 pgs: 342 active+remapped+wait_backfill, 487 active+remapped, 7 active+remapped+backfilling, 316 active+recovery_wait+remapped; 2336 GB data, 7225 GB used, 18810 GB / 26035 GB avail; 339KB/s rd, 177B/s wr, 38op/s; 526796/2001103 degraded (26.325%); recovering 0 o/s, 177KB/s, 0 key/s 2013-05-23 22:21:09.015702 mon.0 [INF] osdmap e5887: 7 osds: 7 up, 5 in 2013-05-23 22:21:09.145648 mon.0 [INF] pgmap v579292: 1152 pgs: 334 active+remapped+wait_backfill, 487 active+remapped, 15 active+remapped+backfilling, 316 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 3237KB/s rd, 11512B/s wr, 475op/s; 526201/2001103 degraded (26.296%); recovering 26 o/s, 100146KB/s, 3 key/s 2013-05-23 22:21:09.346464 mon.0 [INF] osdmap e5888: 7 osds: 7 up, 5 in 2013-05-23 22:21:09.367363 mon.0 [INF] pgmap v579293: 1152 pgs: 334 active+remapped+wait_backfill, 487 active+remapped, 15 active+remapped+backfilling, 316 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 42564KB/s rd, 147KB/s wr, 6245op/s; 526201/2001103 degraded (26.296%); recovering 343 o/s, 1285MB/s, 39 key/s 2013-05-23 22:21:10.387060 mon.0 [INF] osdmap e5889: 7 osds: 7 up, 5 in 2013-05-23 22:21:10.426204 mon.0 [INF] pgmap v579294: 1152 pgs: 334 active+remapped+wait_backfill, 487 active+remapped, 15 active+remapped+backfilling, 316 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 526201/2001103 degraded (26.296%) 2013-05-23 22:21:11.791879 mon.0 [INF] osdmap e5890: 7 osds: 7 up, 5 in 2013-05-23 22:21:12.068473 mon.0 [INF] pgmap v579295: 1152 pgs: 334 active+remapped+wait_backfill, 487 active+remapped, 15 active+remapped+backfilling, 316 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 526201/2001103 degraded (26.296%) 2013-05-23 22:21:14.588722 mon.0 [INF] pgmap v579296: 1152 pgs: 333 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 16 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 1543KB/s rd, 3898B/s wr, 116op/s; 526197/2001103 degraded (26.295%); recovering 0 o/s, 3898KB/s 2013-05-23 22:21:15.704310 mon.0 [INF] pgmap v579297: 1152 pgs: 330 active+remapped+wait_backfill, 4 active+recovering+remapped, 490 active+remapped, 16 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 4370KB/s rd, 20201B/s wr, 180op/s; 526160/2001103 degraded (26.293%); recovering 10 o/s, 43593KB/s 2013-05-23 22:21:16.855678 mon.0 [INF] pgmap v579298: 1152 pgs: 327 active+remapped+wait_backfill, 4 active+recovering+remapped, 490 active+remapped, 15 active+remapped+backfilling, 312 active+recovery_wait+remapped, 4 remapped+peering; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 5991KB/s rd, 27191B/s wr, 184op/s; 524306/2001103 degraded (26.201%); recovering 17 o/s, 72511KB/s 2013-05-23 22:21:20.129830 mon.0 [INF] pgmap v579299: 1152 pgs: 327 active+remapped+wait_backfill, 4 active+recovering+remapped, 490 active+remapped, 15 active+remapped+backfilling, 312 active+recovery_wait+remapped, 4 remapped+peering; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 810KB/s rd, 1041B/s wr, 53op/s; 524306/2001103 degraded (26.201%); recovering 0 o/s, 3125KB/s 2013-05-23 22:21:21.352791 mon.0 [INF] pgmap v579300: 1152 pgs: 327 active+remapped+wait_backfill, 4 active+recovering+remapped, 490 active+remapped, 15 active+remapped+backfilling, 312 active+recovery_wait+remapped, 4 remapped+peering; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 6592B/s rd, 941B/s wr, 2op/s; 524306/2001103 degraded (26.201%) 2013-05-23 22:21:21.846490 mon.0 [INF] osdmap e5891: 7 osds: 7 up, 5 in 2013-05-23 22:21:21.929143 mon.0 [INF] pgmap v579301: 1152 pgs: 327 active+remapped+wait_backfill, 4 active+recovering+remapped, 491 active+remapped, 15 active+remapped+backfilling, 312 active+recovery_wait+remapped, 3 remapped+peering; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 12633B/s rd, 4op/s; 524909/2001103 degraded (26.231%) 2013-05-23 22:21:24.671823 mon.0 [INF] pgmap v579302: 1152 pgs: 327 active+remapped+wait_backfill, 4 active+recovering+remapped, 491 active+remapped, 15 active+remapped+backfilling, 312 active+recovery_wait+remapped, 3 remapped+peering; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 524909/2001103 degraded (26.231%) 2013-05-23 22:21:27.338984 mon.0 [INF] pgmap v579303: 1152 pgs: 329 active+remapped+wait_backfill, 4 active+recovering+remapped, 488 active+remapped, 16 active+remapped+backfilling, 312 active+recovery_wait+remapped, 3 remapped+peering; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 454KB/s rd, 16617B/s wr, 6op/s; 524909/2001103 degraded (26.231%) 2013-05-23 22:21:28.654917 mon.0 [INF] pgmap v579304: 1152 pgs: 335 active+remapped+wait_backfill, 4 active+recovering+remapped, 488 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 15EB/s rd, 22246B/s wr, 15Eop/s; 505907/2001103 degraded (25.281%); recovering 0 o/s, 1059KB/s 2013-05-23 22:21:32.995155 mon.0 [INF] pgmap v579305: 1152 pgs: 335 active+remapped+wait_backfill, 4 active+recovering+remapped, 488 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 15EB/s rd, 6170B/s wr, 15Eop/s; 505907/2001103 degraded (25.281%); recovering 0 o/s, 1028KB/s 2013-05-23 22:21:34.138978 mon.0 [INF] pgmap v579306: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 4617KB/s rd, 2904B/s wr, 68op/s; 505907/2001103 degraded (25.281%) 2013-05-23 22:21:35.314756 mon.0 [INF] pgmap v579307: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 5271KB/s rd, 5987B/s wr, 217op/s; 505907/2001103 degraded (25.281%) 2013-05-23 22:21:36.448399 mon.0 [INF] pgmap v579308: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 9527KB/s rd, 42266B/s wr, 898op/s; 505907/2001103 degraded (25.281%) 2013-05-23 22:21:37.573235 mon.0 [INF] pgmap v579309: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 6839KB/s rd, 34318B/s wr, 712op/s; 505907/2001103 degraded (25.281%) 2013-05-23 22:21:39.573535 mon.0 [INF] pgmap v579310: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 2645KB/s rd, 1291B/s wr, 358op/s; 505907/2001103 degraded (25.281%) 2013-05-23 22:21:40.674481 mon.0 [INF] pgmap v579311: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 3040KB/s rd, 19605B/s wr, 995op/s; 505907/2001103 degraded (25.281%) 2013-05-23 22:21:41.724134 mon.0 [INF] pgmap v579312: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 2491KB/s rd, 26645B/s wr, 1614op/s; 505907/2001103 degraded (25.281%) 2013-05-23 22:21:42.896646 mon.0 [INF] pgmap v579313: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 773KB/s rd, 987op/s; 505907/2001103 degraded (25.281%) 2013-05-23 22:21:45.595096 mon.0 [INF] pgmap v579314: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 3817KB/s rd, 11112B/s wr, 572op/s; 505907/2001103 degraded (25.281%) 2013-05-23 22:21:46.302113 mon.0 [INF] pgmap v579315: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 7398KB/s rd, 24340B/s wr, 551op/s; 505907/2001103 degraded (25.281%) 2013-05-23 22:21:47.427169 mon.0 [INF] pgmap v579316: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 6948KB/s rd, 19266B/s wr, 528op/s; 505907/2001103 degraded (25.281%) 2013-05-23 22:21:50.281238 mon.0 [INF] pgmap v579317: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 1289KB/s rd, 117op/s; 505895/2001103 degraded (25.281%); recovering 3 o/s, 9703KB/s 2013-05-23 22:21:51.656723 mon.0 [INF] pgmap v579318: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 1469KB/s rd, 5376B/s wr, 161op/s; 505895/2001103 degraded (25.281%); recovering 3 o/s, 8608KB/s 2013-05-23 22:21:54.879750 mon.0 [INF] osdmap e5892: 7 osds: 7 up, 5 in 2013-05-23 22:21:54.965426 mon.0 [INF] pgmap v579319: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 840KB/s rd, 3870B/s wr, 84op/s; 505895/2001103 degraded (25.281%) 2013-05-23 22:21:57.382340 mon.0 [INF] osdmap e5893: 7 osds: 7 up, 5 in 2013-05-23 22:21:57.653260 mon.0 [INF] pgmap v579321: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 7965KB/s rd, 29174B/s wr, 60op/s; 505834/2001103 degraded (25.278%); recovering 23 o/s, 98272KB/s 2013-05-23 22:21:59.668051 mon.0 [INF] pgmap v579322: 1152 pgs: 336 active+remapped+wait_backfill, 4 active+recovering+remapped, 487 active+remapped, 13 active+remapped+backfilling, 312 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 4161KB/s rd, 15800B/s wr, 31op/s; 505832/2001103 degraded (25.278%); recovering 9 o/s, 40629KB/s 2013-05-23 22:22:00.770541 mon.0 [INF] pgmap v579323: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 2659KB/s rd, 11565B/s wr, 23op/s; 505819/2001008 degraded (25.278%); recovering 4 o/s, 16706KB/s 2013-05-23 22:22:01.930878 mon.0 [INF] pgmap v579324: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 3668KB/s rd, 15956B/s wr, 32op/s; 505819/2001008 degraded (25.278%); recovering 4 o/s, 19501KB/s 2013-05-23 22:22:04.629195 mon.0 [INF] pgmap v579325: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 1592KB/s rd, 1061B/s wr, 11op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:05.802016 mon.0 [INF] pgmap v579326: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 5056KB/s rd, 20224B/s wr, 37op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:07.002243 mon.0 [INF] pgmap v579327: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 8192KB/s rd, 31045B/s wr, 61op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:08.427964 mon.0 [INF] pgmap v579328: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 3812KB/s rd, 26op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:09.570272 mon.0 [INF] pgmap v579329: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 1281KB/s rd, 149op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:10.704368 mon.0 [INF] pgmap v579330: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 1307KB/s rd, 13310B/s wr, 589op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:11.897727 mon.0 [INF] pgmap v579331: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 1462KB/s rd, 14660B/s wr, 676op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:12.997848 mon.0 [INF] pgmap v579332: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 312KB/s rd, 400op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:14.598238 mon.0 [INF] pgmap v579333: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 2803KB/s rd, 1481B/s wr, 291op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:15.882606 mon.0 [INF] pgmap v579334: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 8180KB/s rd, 39434B/s wr, 325op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:17.066565 mon.0 [INF] pgmap v579335: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 10731KB/s rd, 44240B/s wr, 728op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:20.033174 mon.0 [INF] pgmap v579336: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 3003KB/s rd, 1039B/s wr, 530op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:22.460824 mon.0 [INF] pgmap v579337: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 187KB/s rd, 234op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:23.611995 mon.0 [INF] pgmap v579338: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 705KB/s rd, 889op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:24.763288 mon.0 [INF] pgmap v579339: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 1687KB/s rd, 1133B/s wr, 1064op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:25.825436 mon.0 [INF] pgmap v579340: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 8617KB/s rd, 29149B/s wr, 586op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:26.986661 mon.0 [INF] pgmap v579341: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 11851KB/s rd, 27410B/s wr, 622op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:29.603336 mon.0 [INF] pgmap v579342: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 3586KB/s rd, 123op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:30.770318 mon.0 [INF] pgmap v579343: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 817KB/s rd, 8op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:34.644922 mon.0 [INF] pgmap v579344: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 594KB/s rd, 2456B/s wr, 6op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:35.987896 mon.0 [INF] pgmap v579345: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 3506KB/s rd, 26598B/s wr, 28op/s; 505819/2001008 degraded (25.278%) 2013-05-23 22:22:37.612784 mon.0 [INF] pgmap v579346: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18808 GB / 26035 GB avail; 9064KB/s rd, 45324B/s wr, 69op/s; 505815/2001008 degraded (25.278%); recovering 2 o/s, 9064KB/s 2013-05-23 22:22:40.297764 mon.0 [INF] pgmap v579347: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 3269KB/s rd, 925B/s wr, 22op/s; 505767/2001008 degraded (25.276%); recovering 12 o/s, 48142KB/s 2013-05-23 22:22:41.711128 mon.0 [INF] pgmap v579348: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 1080KB/s rd, 4164B/s wr, 10op/s; 505714/2001008 degraded (25.273%); recovering 26 o/s, 102MB/s 2013-05-23 22:22:42.640586 mon.0 [INF] pgmap v579349: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 12378B/s rd, 7073B/s wr, 5op/s; 505685/2001008 degraded (25.272%); recovering 36 o/s, 143MB/s 2013-05-23 22:22:44.732866 mon.0 [INF] pgmap v579350: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 1220KB/s rd, 1220B/s wr, 9op/s; 505670/2001008 degraded (25.271%); recovering 13 o/s, 52765KB/s 2013-05-23 22:22:46.408617 mon.0 [INF] pgmap v579351: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7227 GB used, 18807 GB / 26035 GB avail; 4515KB/s rd, 23954B/s wr, 58op/s; 505642/2001008 degraded (25.269%); recovering 12 o/s, 51501KB/s 2013-05-23 22:22:47.606129 mon.0 [INF] pgmap v579352: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7228 GB used, 18807 GB / 26035 GB avail; 7318KB/s rd, 27364B/s wr, 195op/s; 505565/2001008 degraded (25.266%); recovering 37 o/s, 147MB/s 2013-05-23 22:22:59.970218 mon.0 [INF] mon.0 calling new monitor election 2013-05-23 22:23:05.919451 mon.0 [INF] osdmap e5893: 7 osds: 7 up, 5 in 2013-05-23 22:23:05.919720 mon.0 [INF] monmap e1: 3 mons at {0=10.6.0.1:6789/0,2=10.6.0.3:6789/0,3=10.6.0.4:6789/0} 2013-05-23 22:23:06.706306 mon.0 [INF] mon.0 calling new monitor election 2013-05-23 22:23:01.214512 mon.1 [INF] mon.2 calling new monitor election 2013-05-23 22:23:06.499619 mon.2 [INF] mon.3 calling new monitor election 2013-05-23 22:23:06.986826 mon.0 [INF] mon.0@0 won leader election with quorum 0,1,2 2013-05-23 22:23:07.652562 mon.0 [INF] pgmap v579353: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7228 GB used, 18807 GB / 26035 GB avail; 3160KB/s rd, 210op/s; 505564/2001008 degraded (25.265%); recovering 19 o/s, 78267KB/s 2013-05-23 22:23:07.652702 mon.0 [INF] mdsmap e1: 0/0/1 up 2013-05-23 22:23:07.709873 mon.0 [INF] osdmap e5893: 7 osds: 7 up, 5 in 2013-05-23 22:23:07.710112 mon.0 [INF] monmap e1: 3 mons at {0=10.6.0.1:6789/0,2=10.6.0.3:6789/0,3=10.6.0.4:6789/0} 2013-05-23 22:23:09.033109 mon.0 [INF] pgmap v579354: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7229 GB used, 18805 GB / 26035 GB avail; 1912KB/s rd, 6654B/s wr, 327op/s; 505266/2001008 degraded (25.251%); recovering 14 o/s, 58180KB/s 2013-05-23 22:23:09.940226 mon.0 [INF] osdmap e5894: 7 osds: 7 up, 5 in 2013-05-23 22:23:10.005978 mon.0 [INF] pgmap v579355: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7230 GB used, 18805 GB / 26035 GB avail; 3256KB/s rd, 10422B/s wr, 495op/s; 505007/2001008 degraded (25.238%); recovering 28 o/s, 112MB/s 2013-05-23 22:23:10.574552 mon.0 [INF] osdmap e5895: 7 osds: 7 up, 5 in 2013-05-23 22:23:10.599039 mon.0 [INF] pgmap v579356: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7230 GB used, 18805 GB / 26035 GB avail; 17398KB/s rd, 32043B/s wr, 1673op/s; 504987/2001008 degraded (25.237%); recovering 136 o/s, 544MB/s 2013-05-23 22:23:11.536068 mon.0 [INF] osdmap e5896: 7 osds: 7 up, 5 in 2013-05-23 22:23:11.635574 mon.0 [INF] pgmap v579357: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7230 GB used, 18805 GB / 26035 GB avail; 4290KB/s rd, 29op/s; 504987/2001008 degraded (25.237%); recovering 10 o/s, 42902KB/s 2013-05-23 22:23:12.788201 mon.0 [INF] osdmap e5897: 7 osds: 7 up, 5 in 2013-05-23 22:23:12.813536 mon.0 [INF] pgmap v579358: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7230 GB used, 18805 GB / 26035 GB avail; 504987/2001008 degraded (25.237%) 2013-05-23 22:23:16.260017 mon.0 [INF] pgmap v579359: 1152 pgs: 336 active+remapped+wait_backfill, 5 active+recovering+remapped, 488 active+remapped, 12 active+remapped+backfilling, 311 active+recovery_wait+remapped; 2336 GB data, 7230 GB used, 18804 GB / 26035 GB avail; 951B/s wr, 0op/s; 504986/2001008 degraded (25.237%); recovering 0 o/s, 951KB/s 2013-05-23 22:23:18.018008 mon.0 [INF] pgmap v579360: 1152 pgs: 257 active+remapped+wait_backfill, 7 active+recovering+remapped, 521 active+remapped, 12 active+remapped+backfilling, 245 active+recovery_wait+remapped, 110 remapped+peering; 2336 GB data, 7230 GB used, 18804 GB / 26035 GB avail; 15EB/s rd, 31675B/s wr, 15Eop/s; 421903/2001008 degraded (21.085%); recovering 18 o/s, 73260KB/s 2013-05-23 22:23:20.980355 mon.0 [INF] pgmap v579361: 1152 pgs: 257 active+remapped+wait_backfill, 7 active+recovering+remapped, 521 active+remapped, 12 active+remapped+backfilling, 245 active+recovery_wait+remapped, 110 remapped+peering; 2336 GB data, 7230 GB used, 18804 GB / 26035 GB avail; 15EB/s rd, 33587B/s wr, 15Eop/s; 421903/2001008 degraded (21.085%); recovering 18 o/s, 74969KB/s 2013-05-23 22:23:22.407144 mon.0 [INF] pgmap v579362: 1152 pgs: 225 active+remapped+wait_backfill, 7 active+recovering+remapped, 599 active+remapped, 12 active+remapped+backfilling, 235 active+recovery_wait+remapped, 74 remapped+peering; 2336 GB data, 7231 GB used, 18804 GB / 26035 GB avail; 15EB/s rd, 1989B/s wr, 15Eop/s; 434934/2001008 degraded (21.736%) 2013-05-23 22:23:23.917485 mon.0 [INF] pgmap v579363: 1152 pgs: 231 active+remapped+wait_backfill, 7 active+recovering+remapped, 612 active+remapped, 12 active+remapped+backfilling, 235 active+recovery_wait+remapped, 55 remapped+peering; 2336 GB data, 7231 GB used, 18804 GB / 26035 GB avail; 15EB/s rd, 15Eop/s; 443885/2001008 degraded (22.183%) 2013-05-23 22:23:25.965650 mon.0 [INF] pgmap v579364: 1152 pgs: 231 active+remapped+wait_backfill, 7 active+recovering+remapped, 612 active+remapped, 12 active+remapped+backfilling, 235 active+recovery_wait+remapped, 55 remapped+peering; 2336 GB data, 7230 GB used, 18804 GB / 26035 GB avail; 7351B/s rd, 1050B/s wr, 2op/s; 443885/2001008 degraded (22.183%) 2013-05-23 22:23:27.039721 mon.0 [INF] pgmap v579365: 1152 pgs: 258 active+remapped+wait_backfill, 7 active+recovering+remapped, 575 active+remapped, 12 active+remapped+backfilling, 266 active+recovery_wait+remapped, 34 remapped+peering; 2336 GB data, 7230 GB used, 18804 GB / 26035 GB avail; 3610B/s wr, 0op/s; 453793/2001008 degraded (22.678%) 2013-05-23 22:23:28.328157 mon.0 [INF] pgmap v579366: 1152 pgs: 268 active+remapped+wait_backfill, 7 active+recovering+remapped, 570 active+remapped, 12 active+remapped+backfilling, 266 active+recovery_wait+remapped, 29 remapped+peering; 2336 GB data, 7230 GB used, 18804 GB / 26035 GB avail; 13077B/s rd, 28022B/s wr, 9op/s; 454485/2001008 degraded (22.713%) 2013-05-23 22:23:31.729081 mon.0 [INF] pgmap v579367: 1152 pgs: 269 active+remapped+wait_backfill, 7 active+recovering+remapped, 569 active+remapped, 12 active+remapped+backfilling, 266 active+recovery_wait+remapped, 29 remapped+peering; 2336 GB data, 7231 GB used, 18804 GB / 26035 GB avail; 12436B/s rd, 30201B/s wr, 9op/s; 454485/2001008 degraded (22.713%) 2013-05-23 22:23:32.050315 mon.0 [INF] pgmap v579368: 1152 pgs: 288 active+remapped+wait_backfill, 7 active+recovering+remapped, 539 active+remapped, 12 active+remapped+backfilling, 281 active+recovery_wait+remapped, 25 remapped+peering; 2336 GB data, 7231 GB used, 18804 GB / 26035 GB avail; 6308B/s wr, 1op/s; 456745/2001008 degraded (22.826%) 2013-05-23 22:23:33.192194 mon.0 [INF] pgmap v579369: 1152 pgs: 321 active+remapped+wait_backfill, 7 active+recovering+remapped, 512 active+remapped, 12 active+remapped+backfilling, 296 active+recovery_wait+remapped, 4 remapped+peering; 2336 GB data, 7231 GB used, 18804 GB / 26035 GB avail; 15025B/s rd, 2146B/s wr, 6op/s; 470125/2001008 degraded (23.494%)