Project

General

Profile

Actions
History

Wiki » Guides »

Deploying Ceph with Chef

Installing Chef

Chef defines three types of entities:

  • Chef Nodes: Run 'chef-client', which installs and manages software.
  • Chef Server: Interacts with 'chef-client' on Chef nodes.
  • Chef Workstation : Manages the Chef server.

See 'Chef Architecture Introduction' for details.

Create a 'chef' User

The 'chef-client' command requires the proper privileges to install and manage
installations. On each Chef node, we recommend creating a 'chef' user with
full 'root' privileges. For example:

ssh user@chef-node
sudo useradd -d /home/chef -m chef
sudo passwd chef

To provide full privileges, add the following to '/etc/sudoers.d/chef'. :

echo "chef ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/chef
sudo chmod 0440 /etc/sudoers.d/chef

If you are using a version of 'sudo' that doesn't support includes, you will
need to add the following to the '/etc/sudoers' file:

chef ALL = (root) NOPASSWD:ALL

.. important: Do not change the file permissions on '/etc/sudoers'. Use a

suitable tool such as 'visudo'.

Generate SSH Keys for Chef Clients

Chef's 'knife' tool can run 'ssh'. To streamline deployments, we
recommend generating an SSH key pair without a passphrase for your
Chef nodes and copying the public key(s) to your Chef nodes so that you
can connect to them from your workstation using 'ssh' from 'knife'
without having to provide a password. To generate a key pair without
a passphrase, execute the following on your Chef workstation. :

ssh-keygen
Generating public/private key pair.
Enter file in which to save the key (/ceph-admin/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /ceph-admin/.ssh/id_rsa.
Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub.

You may use RSA or DSA keys. Once you generate your keys, copy them to each
OSD host. For example:

ssh-copy-id chef@your-node

Consider modifying your '~/.ssh/config' file so that it defaults to
logging in as 'chef' when no username is specified. :

Host myserver01
Hostname myserver01.fqdn-or-ip-address.com
User chef
Host myserver02
Hostname myserver02.fqdn-or-ip-address.com
User chef

Installing Ruby

Chef requires you to install Ruby. Use the version applicable to your current
Linux distribution and install Ruby on all of your hosts. :

sudo apt-get update
sudo apt-get install ruby

Installing Chef and Chef Server on a Server

If you plan on hosting your Chef Server at Opscode you may skip this step,
but you must make a note of the the fully qualified domain name or IP address
of your Chef Server for 'knife' and 'chef-client'.

First, add Opscode packages to your APT configuration. For example:

sudo tee /etc/apt/sources.list.d/chef.list << EOF
deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main  
deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main
EOF

Next, you must request keys so that APT can verify the packages. Copy
and paste the following line into your command line:

sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A

The key is only used by 'apt', so remove it from the 'root' keyring by
typing 'Y' when prompted to delete it.

Install the Opscode keyring, Chef and Chef server on the host designated
as your Chef Server. :

sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef chef-server

Enter the fully qualified domain name or IP address for your Chef server. For example:

http://fqdn-or-ip-address.com:4000

The Chef server installer will prompt you to enter a temporary password. Enter
a temporary password (e.g., 'foo') and proceed with the installation.

.. tip: When prompted for a temporary password, you may press OK.

The installer wants you to re-enter the password to confirm it. To

re-enter the password, you must press the ESC key.

Once the installer finishes and activates the Chef server, you may enter the
fully qualified domain name or IP address in a browser to launch the
Chef web UI. For example:

http://fqdn-or-ip-address.com:4000

The Chef web UI will prompt you to enter the username and password.

- login: 'admin'
- password: 'foo'

Once you have entered the temporary password, the Chef web UI will prompt you
to enter a new password.

Install Chef on all Remaining Hosts

Install Chef on all Chef Nodes and on the Chef Workstation (if it is not the
same host as the Chef Server).

First, add Opscode packages to your APT configuration. For example:

sudo tee /etc/apt/sources.list.d/chef.list << EOF
deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main  
deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main
EOF

Next, you must request keys so that APT can verify the packages. Copy
and paste the following line into your command line:

sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A

The key is only used by 'apt', so remove it from the 'root' keyring by
typing 'Y' when prompted to delete it.

Install the Opscode keyring and Chef on all hosts other than the Chef Server. :

sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef

Enter the fully qualified domain name or IP address for your Chef server.
For example:

http://fqdn-or-ip-address.com:4000

Configuring Knife

Once you complete the Chef server installation, install 'knife' on the your
Chef Workstation. If the Chef server is a remote host, use 'ssh' to connect. :

ssh chef@fqdn-or-ip-address.com

In the '/home/chef' directory, create a hidden Chef directory. :

mkdir -p ~/.chef

The server generates validation and web UI certificates with read/write
permissions for the user that installed the Chef server. Copy them from the
'/etc/chef' directory to the '~/.chef' directory. Then, change their
ownership to the current user. :

sudo cp /etc/chef/validation.pem /etc/chef/webui.pem ~/.chef && sudo chown $(id -u):$(id -g) ~/.chef/*.pem

From the current user's home directory, configure 'knife' with an initial
API client. :

knife configure -i

The configuration will prompt you for inputs. Answer accordingly:

  1. Where should I put the config file? [~/.chef/knife.rb]* Press Enter to accept the default value.
  2. Please enter the chef server URL:* If you are installing the client on the same host as the server, enter 'http://localhost:4000'. Otherwise, enter an appropriate URL for the server.
  3. Please enter a clientname for the new client:* Press Enter to accept the default value.
  4. Please enter the existing admin clientname:* Press Enter to accept the default value.
  5. Please enter the location of the existing admin client's private key:* Override the default value so that it points to the '.chef' directory. (e.g., '/home/chef/.chef/webui.pem')
  6. Please enter the validation clientname:* Press Enter to accept the default value.
  7. Please enter the location of the validation key:* Override the default value so that it points to the '.chef' directory. (e.g., '/home/chef/.chef/validation.pem')
  8. Please enter the path to a chef repository (or leave blank):* Leave the entry field blank and press Enter.

Add a Cookbook Path

Add 'cookbook_path' to the '~/.chef/knife.rb' configuration file
on your Chef workstation. For example:

cookbook_path '/home/{user-name}/chef-cookbooks/'

Then create the path if it doesn't already exist. :

mkdir /home/{user-name}/chef-cookbooks

This is where you will store local copies of cookbooks before uploading
them to the Chef server.

Copy 'validation.pem' to Nodes

Copy the '/etc/chef/validation.pem' file from your Chef server to
each Chef Node. In a command line shell on the Chef Server, for each node,
replace '{nodename}' in the following line with the node's host name and
execute it. :

sudo cat /etc/chef/validation.pem | ssh {nodename} "exec sudo tee /etc/chef/validation.pem >/dev/null"

Run 'chef-client' on each Chef Node

Run the 'chef-client' on each Chef Node so that the nodes
register with the Chef server. :

ssh chef-node
sudo chef-client

Verify Nodes

Verify that you have setup all the hosts you want to use as
Chef nodes. :

knife node list

A list of the nodes you've configured should appear.

Read on for information on using Chef to deploy your Ceph cluster.

Chef Architecture Introduction: http://wiki.opscode.com/display/chef...e+Introduction

Chef Server at Opscode: http://www.opscode.com/hosted-chef/

Installing Chef Client on Ubuntu or Debian: http://wiki.opscode.com/display/chef...untu+or+Debian

Installing Chef Server on Debian or Ubuntu using Packages: http://wiki.opscode.com/display/chef...using+Packages

Knife Bootstrap: http://wiki.opscode.com/display/chef/Knife+Bootstrap

Deploying Ceph

Now it's time to deploy Ceph. For Chef installation instructions, see above.

Clone the Required Cookbooks

To get the cookbooks for Ceph, clone them from git.:

cd ~/chef-cookbooks
git clone https://github.com/opscode-cookbooks/apache2.git
git clone https://github.com/ceph/ceph-cookbooks.git ceph

Add the Required Cookbook Paths

If you added a default cookbook path when you installed Chef, 'knife'
may be able to upload the cookbook you've cloned to your cookbook path
directory without further configuration. If you used a different path,
or if the cookbook repository you cloned has a different tree structure,
add the required cookbook path to your 'knife.rb' file. The
'cookbook_path' setting takes a string or an array of strings.
For example, you can replace a string path with an array of string paths:

cookbook_path '/home/{user-name}/chef-cookbooks/'

Becomes: 
cookbook_path [
'/home/{user-name}/chef-cookbooks/', 
'/home/{user-name}/chef-cookbooks/{another-directory}/',
'/some/other/path/to/cookbooks/'
]

Install the Cookbooks

To install Ceph, you must upload the Ceph cookbooks and the Apache cookbooks
(for use with RADOSGW) to your Chef server. :

knife cookbook upload apache2 ceph

Configure your Ceph Environment

The Chef server can support installation of software for multiple environments.
The environment you create for Ceph requires an 'fsid', the secret for
your monitor(s) if you are running Ceph with 'cephx' authentication, and
the host name (i.e., short name) for your monitor hosts.

.. tip: Open an empty text file to hold the following values until you create
your Ceph environment.

For the filesystem ID, use 'uuidgen' from the 'uuid-runtime' package to
generate a unique identifier.

uuidgen -r

For the monitor(s) secret(s), use 'ceph-authtool' to generate the secret(s):

sudo apt-get update
sudo apt-get install ceph-common
ceph-authtool /dev/stdout --name=mon. --gen-key

The secret is the value to the right of "key =", and should look something
like this:

AQBAMuJPINJgFhAAziXIrLvTvAz4PRo5IK/Log==

To create an environment for Ceph, set a command line editor. For example:

export EDITOR=vim

Then, use 'knife' to create an environment.

knife environment create {env-name}

For example:

knife environment create Ceph

A JSON file will appear. Perform the following steps:

  1. Enter a description for the environment.
  2. In '"default_attributes": {}', add '"ceph" : {}'.
  3. Within '"ceph" : {}', add '"monitor-secret":'.
  4. Immediately following '"monitor-secret":' add the key you generated within quotes, followed by a comma.
  5. Within '"ceph":{}' and following the 'monitor-secret' key-value pair, add '"config": {}'
  6. Within '"config": {}' add '"fsid":'.
  7. Immediately following '"fsid":', add the unique identifier you generated within quotes, followed by a comma.
  8. Within '"config": {}' and following the 'fsid' key-value pair, add '"mon_initial_members":'
  9. Immediately following '"mon_initial_members":', enter the initial monitor host names.

You may also set Ceph settings within '"config": {}'.

For example:


    "default_attributes" : {
        "ceph": {
            "monitor-secret": "{replace-with-generated-secret}",
            "config": {
                "fsid": "{replace-with-generated-uuid}",
                "mon_initial_members": "{replace-with-monitor-hostname(s)}",
                "global": {
                    "public network": "xxx.xxx.xxx.xxx/yy",
                    "cluster network": "xxx.xxx.xxx.xxx/yy" 
                },
                "osd": {
                    "osd journal size": "1000" 
                }
            }
        }
    }

Will generate the following ceph.conf:

    [global]
        fsid = <fsid>
        mon initial members = X,Y,Z
        mon host = ipX:port, ipY:port, ipZ:port ;mon host is auto generated
        public network = xxx.xxx.xxx.xxx/yy
        cluster network = xxx.xxx.xxx.xxx/yy
    [osd]
        osd journal size = 1000

Advanced users (i.e., developers and QA) may also add '"branch": "{branch}"'
to '"ceph": {}'. Valid values are 'stable', 'testing', 'dev'.
You can specify which stable release (e.g. argonaut, bobtail) or which dev
branch to use with '"version": "{version}"' within '"ceph": {}'.
If 'version' is not specified for 'stable', the latest stable release
will be used. 'testing' does not require 'version'.

The Ceph cookbook will, by default, use the official Ceph repositories. If you wish to use your own repositories you can specify them, in the Ceph environment, within '"ceph": {}' like so:

"platform_family": {
    "branch": {
        "repository": "mylocalcephrepo" 
        "repository_key": "mylocalcephrepokey" 
    }
}
platform_family can be:
  • rhel: For RHEL and CentOS
  • debian: For Debian and Ubuntu
  • suse: For OpenSUSE and SLES
  • fedora: For Fedora
    branch can be:
  • stable
  • testing
  • dev

Configure the Roles

Navigate to the Ceph cookbooks directory. :

cd ~/chef-cookbooks/ceph

Create roles for OSDs, monitors, metadata servers, and RADOS Gateways from
their respective role files. :

knife role from file roles/ceph-osd.rb
knife role from file roles/ceph-mon.rb
knife role from file roles/ceph-mds.rb
knife role from file roles/ceph-radosgw.rb

Configure Nodes

You must configure each node you intend to include in your Ceph cluster.
Identify nodes for your Ceph cluster. :

knife node list

.. note: for each host where you installed Chef and executed 'chef-client',

the Chef server should have a minimal node configuration. You can create

additional nodes with 'knife node create {node-name}'.

For each node you intend to use in your Ceph cluster, configure the node
as follows:

knife node edit {node-name}

The node configuration should appear in your text editor. Change the
'chef_environment' value to 'Ceph' (or whatever name you set for your
Ceph environment).

In the 'run_list', add '"recipe[ceph:repo]",' to all nodes as
the first setting, so that Chef can install or update the necessary packages.
Then, add at least one of:

"role[ceph-mon]" 
"role[ceph-osd]" 
"role[ceph-mds]" 
"role[ceph-radosgw]"

If you add more than one role, separate them with a comma. Run 'hostname'
on your command line, and replace the '{hostname}' setting of the 'name'
key to the host name for the node. :

{
  "chef_environment": "Ceph",
  "name": "{hostname}",
  "normal": {
    "tags": [
    ]
  },
  "run_list": [
    "role[ceph-mon]",
    "role[ceph-mds]" 
  ]
}

Deploy OSDs

Configuring a node with an OSD role tells Chef that the node will run at
least one OSD. However, you may run many OSDs on one host. For example,
you may run one 'ceph-osd' daemon for each data disk on the system.
To tell Chef to deploy OSDs, edit the node and add the following
within '"normal": {}':

    "ceph": {
        "osd_devices": [
            {
                "device": "/dev/...",
                "journal": "/dev/..." 
            },
            {
                "device": "/dev/...",
                "dmcrypt": true
            }
        ]
    }

Supported values are 'device', 'journal', 'dmcrypt' (deactivated by default).

.. note: dmcrypt is only supported starting with Cuttlefish

Run 'chef-client' on each Node

Once you have completed the preceding steps, you must run 'chef-client'
on each node. For example:

sudo chef-client

Proceed to Operating the Cluster

Once you complete the deployment, you may begin operating your cluster.
See 'Operating a Cluster' for details.

Managing Cookbooks with Knife: http://wiki.opscode.com/display/chef...oks+With+Knife

Updated by Jessica Mack almost 9 years ago · 1 revisions