Ceph

h1. Deploying Ceph with Chef

{{toc}}

h2. %{color:black}Installing Chef%

Chef defines three types of entities:

* *Chef Nodes*: Run 'chef-client', which installs and manages software.

* *Chef Server*: Interacts with 'chef-client' on Chef nodes.

* *Chef Workstation* : Manages the Chef server.

See "'Chef Architecture Introduction'":http://docs.chef.io/ for details.

h3. Create a 'chef' User

The 'chef-client' command requires the proper privileges to install and manage

installations. On each Chef node, we recommend creating a 'chef' user with 

full 'root' privileges. For example: 

<pre>

ssh user@chef-node

sudo useradd -d /home/chef -m chef

sudo passwd chef

</pre>

To provide full privileges, add the following to '/etc/sudoers.d/chef'. :

<pre>

echo "chef ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/chef

sudo chmod 0440 /etc/sudoers.d/chef

</pre>

If you are using a version of 'sudo' that doesn't support includes, you will

need to add the following to the '/etc/sudoers' file:

<pre>

chef ALL = (root) NOPASSWD:ALL

</pre>

> .. important: Do not change the file permissions on '/etc/sudoers'. Use a

>   suitable tool such as 'visudo'.

h4. Generate SSH Keys for Chef Clients

Chef's 'knife' tool can run 'ssh'. To streamline deployments, we 

recommend generating an SSH key pair without a passphrase for your 

Chef nodes and copying the public key(s) to your Chef nodes so that you 

can connect to them from your workstation using 'ssh' from 'knife'

without having to provide a password. To generate a key pair without 

a passphrase, execute the following on your Chef workstation. : 

<pre>

ssh-keygen

Generating public/private key pair.

Enter file in which to save the key (/ceph-admin/.ssh/id_rsa): 

Enter passphrase (empty for no passphrase): 

Enter same passphrase again: 

Your identification has been saved in /ceph-admin/.ssh/id_rsa.

Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub.

</pre>

You may use RSA or DSA keys. Once you generate your keys, copy them to each 

OSD host. For example: 

<pre>

ssh-copy-id chef@your-node

</pre>

Consider modifying your '~/.ssh/config' file so that it defaults to 

logging in as 'chef' when no username is specified. : 

<pre>

Host myserver01

Hostname myserver01.fqdn-or-ip-address.com

User chef

Host myserver02

Hostname myserver02.fqdn-or-ip-address.com

User chef

</pre>

h4. Installing Ruby

Chef requires you to install Ruby. Use the version applicable to your current 

Linux distribution and install Ruby on all of your hosts. :

<pre>

sudo apt-get update

sudo apt-get install ruby

</pre>

Installing Chef and Chef Server on a Server

If you plan on hosting your Chef Server at Opscode you may skip this step, 

but you must make a note of the the fully qualified domain name or IP address

of your Chef Server for 'knife' and 'chef-client'.

First, add Opscode packages to your APT configuration. For example: 

<pre>

sudo tee /etc/apt/sources.list.d/chef.list << EOF

deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main  

deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main

EOF

</pre>

Next, you must request keys so that APT can verify the packages. Copy

and paste the following line into your command line: 

<pre>

sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A

</pre>

The key is only used by 'apt', so remove it from the 'root' keyring by

typing 'Y' when prompted to delete it.

Install the Opscode keyring, Chef and Chef server on the host designated

as your Chef Server. :

<pre>

sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef chef-server

</pre>

Enter the fully qualified domain name or IP address for your Chef server. For example:

<pre>

http://fqdn-or-ip-address.com:4000

</pre>

The Chef server installer will prompt you to enter a temporary password. Enter

a temporary password (*e.g.,* 'foo') and proceed with the installation. 

> .. tip: When prompted for a temporary password, you may press **OK**.

>   The installer wants you to re-enter the password to confirm it. To 

>   re-enter the password, you must press the **ESC** key.

Once the installer finishes and activates the Chef server, you may enter the 

fully qualified domain name or IP address in a browser to launch the 

Chef web UI. For example: 

<pre>

http://fqdn-or-ip-address.com:4000

</pre>

The Chef web UI will prompt you to enter the username and password.

<pre>

- login: 'admin'

- password: 'foo'

</pre>

Once you have entered the temporary password, the Chef web UI will prompt you

to enter a new password.

h4. Install Chef on all Remaining Hosts

Install Chef on all Chef Nodes and on the Chef Workstation (if it is not the 

same host as the Chef Server).

First, add Opscode packages to your APT configuration. For example: 

<pre>

sudo tee /etc/apt/sources.list.d/chef.list << EOF

deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main  

deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main

EOF

</pre>

Next, you must request keys so that APT can verify the packages. Copy

and paste the following line into your command line: 

<pre>

sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A

</pre>

The key is only used by 'apt', so remove it from the 'root' keyring by

typing 'Y' when prompted to delete it.

Install the Opscode keyring and Chef on all hosts other than the Chef Server. :

<pre>

sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef

</pre>

Enter the fully qualified domain name or IP address for your Chef server. 

For example:

<pre>

http://fqdn-or-ip-address.com:4000

</pre>

h4. Configuring Knife

Once you complete the Chef server installation, install 'knife' on the your

Chef Workstation. If the Chef server is a remote host, use 'ssh' to connect. : 

<pre>

ssh chef@fqdn-or-ip-address.com

</pre>

In the '/home/chef' directory, create a hidden Chef directory. : 

<pre>

mkdir -p ~/.chef

</pre>

The server generates validation and web UI certificates with read/write 

permissions for the user that installed the Chef server. Copy them from the

'/etc/chef' directory to the '~/.chef' directory. Then, change their 

ownership to the current user. :

<pre>

sudo cp /etc/chef/validation.pem /etc/chef/webui.pem ~/.chef && sudo chown $(id -u):$(id -g) ~/.chef/*.pem

</pre>

From the current user's home directory, configure 'knife' with an initial 

API client. : 

<pre>

knife configure -i

</pre>

The configuration will prompt you for inputs. Answer accordingly: 

# Where should I put the config file? [~/.chef/knife.rb]* Press **Enter** to accept the default value.

# Please enter the chef server URL:* If you are installing the client on the same host as the server, enter 'http://localhost:4000'. Otherwise, enter an appropriate URL for the server.

# Please enter a clientname for the new client:* Press **Enter** to accept the default value.

# Please enter the existing admin clientname:* Press **Enter** to accept the default value.

# Please enter the location of the existing admin client's private key:* Override the default value so that it points to the '.chef' directory.  (*e.g.,* '/home/chef/.chef/webui.pem')

# Please enter the validation clientname:* Press **Enter** to accept  the default value.

# Please enter the location of the validation key:* Override the  default value so that it points to the '.chef' directory. (*e.g.,* '/home/chef/.chef/validation.pem')

# Please enter the path to a chef repository (or leave blank):* Leave the entry field blank and press **Enter**.

h4. Add a Cookbook Path

Add 'cookbook_path' to the '~/.chef/knife.rb' configuration file

on your Chef workstation. For example:

<pre>

cookbook_path '/home/{user-name}/chef-cookbooks/'

</pre>

Then create the path if it doesn't already exist. :

<pre>

mkdir /home/{user-name}/chef-cookbooks

</pre>

This is where you will store local copies of cookbooks before uploading

them to the Chef server.

h4. Copy 'validation.pem' to Nodes

Copy the '/etc/chef/validation.pem' file from your Chef server to

each Chef Node. In a command line shell on the Chef Server, for each node, 

replace '{nodename}' in the following line with the node's host name and 

execute it. :

<pre>

sudo cat /etc/chef/validation.pem | ssh {nodename} "exec sudo tee /etc/chef/validation.pem >/dev/null"

</pre>

h4. Run 'chef-client' on each Chef Node

Run the 'chef-client' on each Chef Node so that the nodes

register with the Chef server. : 

<pre>

ssh chef-node

sudo chef-client

</pre>

h4. Verify Nodes

Verify that you have setup all the hosts you want to use as 

Chef nodes. : 

<pre>

knife node list

</pre>

A list of the nodes you've configured should appear.

Read on for information on using Chef to deploy your Ceph cluster.

> Chef Architecture Introduction: http://wiki.opscode.com/display/chef...e+Introduction

> Chef Server at Opscode: http://www.opscode.com/hosted-chef/

> Installing Chef Client on Ubuntu or Debian: http://wiki.opscode.com/display/chef...untu+or+Debian

> Installing Chef Server on Debian or Ubuntu using Packages: http://wiki.opscode.com/display/chef...using+Packages

> Knife Bootstrap: http://wiki.opscode.com/display/chef/Knife+Bootstrap

h2. %{color:black}Deploying Ceph%

Now it's time to deploy Ceph.   For Chef installation instructions, see above.

h3. Clone the Required Cookbooks

To get the cookbooks for Ceph, clone them from git.:

<pre>

cd ~/chef-cookbooks

git clone https://github.com/opscode-cookbooks/apache2.git

git clone https://github.com/ceph/ceph-cookbooks.git ceph

</pre>

h4. Add the Required Cookbook Paths

If you added a default cookbook path when you installed Chef, 'knife'

may be able to upload the cookbook you've cloned to your cookbook path

directory without further configuration. If you used a different path, 

or if the cookbook repository you cloned has a different tree structure, 

add the required cookbook path to your 'knife.rb' file. The 

'cookbook_path' setting takes a string or an array of strings. 

For example, you can replace a string path with an array of string paths:

<pre>

cookbook_path '/home/{user-name}/chef-cookbooks/'

</pre> 

Becomes:

<pre>

cookbook_path [

'/home/{user-name}/chef-cookbooks/', 

'/home/{user-name}/chef-cookbooks/{another-directory}/',

'/some/other/path/to/cookbooks/'

]

</pre>

h4. Install the Cookbooks

To install Ceph, you must upload the Ceph cookbooks and the Apache cookbooks

(for use with RADOSGW) to your Chef server. : 

<pre>

knife cookbook upload apache2 ceph

</pre>

h4. Configure your Ceph Environment

The Chef server can support installation of software for multiple environments.

The environment you create for Ceph requires an 'fsid', the secret for

your monitor(s) if you are running Ceph with 'cephx' authentication, and

the host name (i.e., short name) for your monitor hosts.

> .. tip: Open an empty text file to hold the following values until you create

>    your Ceph environment.

For the filesystem ID, use 'uuidgen' from the 'uuid-runtime' package to 

generate a unique identifier.

<pre>

uuidgen -r

</pre>

For the monitor(s) secret(s), use 'ceph-authtool' to generate the secret(s):

<pre>

sudo apt-get update

sudo apt-get install ceph-common

ceph-authtool /dev/stdout --name=mon. --gen-key  

</pre>

The secret is the value to the right of "key =", and should look something 

like this:

<pre>

AQBAMuJPINJgFhAAziXIrLvTvAz4PRo5IK/Log==

</pre>

To create an environment for Ceph, set a command line editor. For example:

<pre>

export EDITOR=vim

</pre>

Then, use 'knife' to create an environment.

<pre>

knife environment create {env-name}

</pre>

For example:

<pre>

knife environment create Ceph

</pre>

A JSON file will appear. Perform the following steps: 

# Enter a description for the environment. 

# In '"default_attributes": {}', add '"ceph" : {}'.

# Within '"ceph" : {}', add '"monitor-secret":'.

# Immediately following '"monitor-secret":' add the key you generated within quotes, followed by a comma.

# Within '"ceph":{}' and following the 'monitor-secret' key-value pair, add '"config": {}'

# Within '"config": {}' add '"fsid":'.

# Immediately following '"fsid":', add the unique identifier you generated within quotes, followed by a comma.

# Within '"config": {}' and following the 'fsid' key-value pair, add '"mon_initial_members":'

# Immediately following '"mon_initial_members":', enter the initial monitor host names.

You may also set Ceph settings within '"config": {}'.

For example:

<pre>

    "default_attributes" : {

        "ceph": {

            "monitor-secret": "{replace-with-generated-secret}",

            "config": {

                "fsid": "{replace-with-generated-uuid}",

                "mon_initial_members": "{replace-with-monitor-hostname(s)}",

                "global": {

                    "public network": "xxx.xxx.xxx.xxx/yy",

                    "cluster network": "xxx.xxx.xxx.xxx/yy"

                },

                "osd": {

                    "osd journal size": "1000"

                }

            }

        }

    }

</pre>

Will generate the following ceph.conf:

<pre>

    [global]

        fsid = <fsid>

        mon initial members = X,Y,Z

        mon host = ipX:port, ipY:port, ipZ:port ;mon host is auto generated

        public network = xxx.xxx.xxx.xxx/yy

        cluster network = xxx.xxx.xxx.xxx/yy

    [osd]

        osd journal size = 1000

</pre>

Advanced users (i.e., developers and QA) may also add '"branch": "{branch}"'

to '"ceph": {}'. Valid values are 'stable', 'testing', 'dev'.

You can specify which stable release (e.g. argonaut, bobtail) or which dev

branch to use with '"version": "{version}"' within '"ceph": {}'.

If 'version' is not specified for 'stable', the latest stable release

will be used. 'testing' does not require 'version'.

The Ceph cookbook will, by default, use the official Ceph repositories. If you wish to use your own repositories you can specify them, in the Ceph environment, within '"ceph": {}' like so:

<pre>

"platform_family": {

    "branch": {

        "repository": "mylocalcephrepo"

        "repository_key": "mylocalcephrepokey"

    }

}

</pre>

platform_family can be:

* rhel: For RHEL and CentOS

* debian: For Debian and Ubuntu

* suse: For OpenSUSE and SLES

* fedora: For Fedora

branch can be:

* stable

* testing

* dev

h4. Configure the Roles

Navigate to the Ceph cookbooks directory. : 

<pre>

cd ~/chef-cookbooks/ceph

</pre>

Create roles for OSDs, monitors, metadata servers, and RADOS Gateways from

their respective role files. :

<pre>

knife role from file roles/ceph-osd.rb

knife role from file roles/ceph-mon.rb

knife role from file roles/ceph-mds.rb

knife role from file roles/ceph-radosgw.rb

</pre>

h4. Configure Nodes

You must configure each node you intend to include in your Ceph cluster. 

Identify nodes for your Ceph cluster. :

<pre>

knife node list

</pre>

> .. note: for each host where you installed Chef and executed 'chef-client', 

>   the Chef server should have a minimal node configuration. You can create

>   additional nodes with 'knife node create {node-name}'.

For each node you intend to use in your Ceph cluster, configure the node 

as follows: 

<pre>

knife node edit {node-name}

</pre>

The node configuration should appear in your text editor. Change the 

'chef_environment' value to 'Ceph' (or whatever name you set for your

Ceph environment). 

In the 'run_list', add '"recipe[ceph:repo]",' to all nodes as

the first setting, so that Chef can install or update the necessary packages. 

Then, add at least one of: 

<pre>

"role[ceph-mon]"

"role[ceph-osd]"

"role[ceph-mds]"

"role[ceph-radosgw]"

</pre>

If you add more than one role, separate them with a comma. Run 'hostname'

on your command line, and replace the '{hostname}' setting of the 'name' 

key to the host name for the node. :

<pre>

{

  "chef_environment": "Ceph",

  "name": "{hostname}",

  "normal": {

    "tags": [

    ]

  },

  "run_list": [

    "role[ceph-mon]",

    "role[ceph-mds]"

  ]

}

</pre>

h4. Deploy OSDs

Configuring a node with an OSD role tells Chef that the node will run at

least one OSD. However, you may run many OSDs on one host. For example, 

you may run one 'ceph-osd' daemon for each data disk on the system. 

To tell Chef to deploy OSDs, edit the node and add the following

within '"normal": {}':

<pre>

    "ceph": {

        "osd_devices": [

            {

                "device": "/dev/...",

                "journal": "/dev/..."

            },

            {

                "device": "/dev/...",

                "dmcrypt": true

            }

        ]

    }

</pre>

Supported values are 'device', 'journal', 'dmcrypt' (deactivated by default).

> .. note: dmcrypt is only supported starting with Cuttlefish

h4. Run 'chef-client' on each Node

Once you have completed the preceding steps, you must run 'chef-client' 

on each node. For example:

<pre>

sudo chef-client

</pre>

h4. Proceed to Operating the Cluster

Once you complete the deployment, you may begin operating your cluster.

See "'Operating a Cluster'":http://ceph.com/docs/master/rados/operations/operating/ for details.

> Managing Cookbooks with Knife: http://wiki.opscode.com/display/chef...oks+With+Knife