Deploying Ceph with Chef¶
- Table of contents
- Deploying Ceph with Chef
Installing Chef¶
Chef defines three types of entities:
- Chef Nodes: Run 'chef-client', which installs and manages software.
- Chef Server: Interacts with 'chef-client' on Chef nodes.
- Chef Workstation : Manages the Chef server.
See 'Chef Architecture Introduction' for details.
Create a 'chef' User¶
The 'chef-client' command requires the proper privileges to install and manage
installations. On each Chef node, we recommend creating a 'chef' user with
full 'root' privileges. For example:
ssh user@chef-node sudo useradd -d /home/chef -m chef sudo passwd chef
To provide full privileges, add the following to '/etc/sudoers.d/chef'. :
echo "chef ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/chef sudo chmod 0440 /etc/sudoers.d/chef
If you are using a version of 'sudo' that doesn't support includes, you will
need to add the following to the '/etc/sudoers' file:
chef ALL = (root) NOPASSWD:ALL
.. important: Do not change the file permissions on '/etc/sudoers'. Use a
suitable tool such as 'visudo'.
Generate SSH Keys for Chef Clients¶
Chef's 'knife' tool can run 'ssh'. To streamline deployments, we
recommend generating an SSH key pair without a passphrase for your
Chef nodes and copying the public key(s) to your Chef nodes so that you
can connect to them from your workstation using 'ssh' from 'knife'
without having to provide a password. To generate a key pair without
a passphrase, execute the following on your Chef workstation. :
ssh-keygen Generating public/private key pair. Enter file in which to save the key (/ceph-admin/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /ceph-admin/.ssh/id_rsa. Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub.
You may use RSA or DSA keys. Once you generate your keys, copy them to each
OSD host. For example:
ssh-copy-id chef@your-node
Consider modifying your '~/.ssh/config' file so that it defaults to
logging in as 'chef' when no username is specified. :
Host myserver01 Hostname myserver01.fqdn-or-ip-address.com User chef Host myserver02 Hostname myserver02.fqdn-or-ip-address.com User chef
Installing Ruby¶
Chef requires you to install Ruby. Use the version applicable to your current
Linux distribution and install Ruby on all of your hosts. :
sudo apt-get update sudo apt-get install ruby
Installing Chef and Chef Server on a Server
If you plan on hosting your Chef Server at Opscode you may skip this step,
but you must make a note of the the fully qualified domain name or IP address
of your Chef Server for 'knife' and 'chef-client'.
First, add Opscode packages to your APT configuration. For example:
sudo tee /etc/apt/sources.list.d/chef.list << EOF deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main EOF
Next, you must request keys so that APT can verify the packages. Copy
and paste the following line into your command line:
sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A
The key is only used by 'apt', so remove it from the 'root' keyring by
typing 'Y' when prompted to delete it.
Install the Opscode keyring, Chef and Chef server on the host designated
as your Chef Server. :
sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef chef-server
Enter the fully qualified domain name or IP address for your Chef server. For example:
http://fqdn-or-ip-address.com:4000
The Chef server installer will prompt you to enter a temporary password. Enter
a temporary password (e.g., 'foo') and proceed with the installation.
.. tip: When prompted for a temporary password, you may press OK.
The installer wants you to re-enter the password to confirm it. To
re-enter the password, you must press the ESC key.
Once the installer finishes and activates the Chef server, you may enter the
fully qualified domain name or IP address in a browser to launch the
Chef web UI. For example:
http://fqdn-or-ip-address.com:4000
The Chef web UI will prompt you to enter the username and password.
- login: 'admin' - password: 'foo'
Once you have entered the temporary password, the Chef web UI will prompt you
to enter a new password.
Install Chef on all Remaining Hosts¶
Install Chef on all Chef Nodes and on the Chef Workstation (if it is not the
same host as the Chef Server).
First, add Opscode packages to your APT configuration. For example:
sudo tee /etc/apt/sources.list.d/chef.list << EOF deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main EOF
Next, you must request keys so that APT can verify the packages. Copy
and paste the following line into your command line:
sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A
The key is only used by 'apt', so remove it from the 'root' keyring by
typing 'Y' when prompted to delete it.
Install the Opscode keyring and Chef on all hosts other than the Chef Server. :
sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef
Enter the fully qualified domain name or IP address for your Chef server.
For example:
http://fqdn-or-ip-address.com:4000
Configuring Knife¶
Once you complete the Chef server installation, install 'knife' on the your
Chef Workstation. If the Chef server is a remote host, use 'ssh' to connect. :
ssh chef@fqdn-or-ip-address.com
In the '/home/chef' directory, create a hidden Chef directory. :
mkdir -p ~/.chef
The server generates validation and web UI certificates with read/write
permissions for the user that installed the Chef server. Copy them from the
'/etc/chef' directory to the '~/.chef' directory. Then, change their
ownership to the current user. :
sudo cp /etc/chef/validation.pem /etc/chef/webui.pem ~/.chef && sudo chown $(id -u):$(id -g) ~/.chef/*.pem
From the current user's home directory, configure 'knife' with an initial
API client. :
knife configure -i
The configuration will prompt you for inputs. Answer accordingly:
- Where should I put the config file? [~/.chef/knife.rb]* Press Enter to accept the default value.
- Please enter the chef server URL:* If you are installing the client on the same host as the server, enter 'http://localhost:4000'. Otherwise, enter an appropriate URL for the server.
- Please enter a clientname for the new client:* Press Enter to accept the default value.
- Please enter the existing admin clientname:* Press Enter to accept the default value.
- Please enter the location of the existing admin client's private key:* Override the default value so that it points to the '.chef' directory. (e.g., '/home/chef/.chef/webui.pem')
- Please enter the validation clientname:* Press Enter to accept the default value.
- Please enter the location of the validation key:* Override the default value so that it points to the '.chef' directory. (e.g., '/home/chef/.chef/validation.pem')
- Please enter the path to a chef repository (or leave blank):* Leave the entry field blank and press Enter.
Add a Cookbook Path¶
Add 'cookbook_path' to the '~/.chef/knife.rb' configuration file
on your Chef workstation. For example:
cookbook_path '/home/{user-name}/chef-cookbooks/'
Then create the path if it doesn't already exist. :
mkdir /home/{user-name}/chef-cookbooks
This is where you will store local copies of cookbooks before uploading
them to the Chef server.
Copy 'validation.pem' to Nodes¶
Copy the '/etc/chef/validation.pem' file from your Chef server to
each Chef Node. In a command line shell on the Chef Server, for each node,
replace '{nodename}' in the following line with the node's host name and
execute it. :
sudo cat /etc/chef/validation.pem | ssh {nodename} "exec sudo tee /etc/chef/validation.pem >/dev/null"
Run 'chef-client' on each Chef Node¶
Run the 'chef-client' on each Chef Node so that the nodes
register with the Chef server. :
ssh chef-node sudo chef-client
Verify Nodes¶
Verify that you have setup all the hosts you want to use as
Chef nodes. :
knife node list
A list of the nodes you've configured should appear.
Read on for information on using Chef to deploy your Ceph cluster.
Chef Architecture Introduction: http://wiki.opscode.com/display/chef...e+Introduction
Chef Server at Opscode: http://www.opscode.com/hosted-chef/
Installing Chef Client on Ubuntu or Debian: http://wiki.opscode.com/display/chef...untu+or+Debian
Installing Chef Server on Debian or Ubuntu using Packages: http://wiki.opscode.com/display/chef...using+Packages
Knife Bootstrap: http://wiki.opscode.com/display/chef/Knife+Bootstrap
Deploying Ceph¶
Now it's time to deploy Ceph. For Chef installation instructions, see above.
Clone the Required Cookbooks¶
To get the cookbooks for Ceph, clone them from git.:
cd ~/chef-cookbooks git clone https://github.com/opscode-cookbooks/apache2.git git clone https://github.com/ceph/ceph-cookbooks.git ceph
Add the Required Cookbook Paths¶
If you added a default cookbook path when you installed Chef, 'knife'
may be able to upload the cookbook you've cloned to your cookbook path
directory without further configuration. If you used a different path,
or if the cookbook repository you cloned has a different tree structure,
add the required cookbook path to your 'knife.rb' file. The
'cookbook_path' setting takes a string or an array of strings.
For example, you can replace a string path with an array of string paths:
cookbook_path '/home/{user-name}/chef-cookbooks/'
Becomes:
cookbook_path [ '/home/{user-name}/chef-cookbooks/', '/home/{user-name}/chef-cookbooks/{another-directory}/', '/some/other/path/to/cookbooks/' ]
Install the Cookbooks¶
To install Ceph, you must upload the Ceph cookbooks and the Apache cookbooks
(for use with RADOSGW) to your Chef server. :
knife cookbook upload apache2 ceph
Configure your Ceph Environment¶
The Chef server can support installation of software for multiple environments.
The environment you create for Ceph requires an 'fsid', the secret for
your monitor(s) if you are running Ceph with 'cephx' authentication, and
the host name (i.e., short name) for your monitor hosts.
.. tip: Open an empty text file to hold the following values until you create
your Ceph environment.
For the filesystem ID, use 'uuidgen' from the 'uuid-runtime' package to
generate a unique identifier.
uuidgen -r
For the monitor(s) secret(s), use 'ceph-authtool' to generate the secret(s):
sudo apt-get update sudo apt-get install ceph-common ceph-authtool /dev/stdout --name=mon. --gen-key
The secret is the value to the right of "key =", and should look something
like this:
AQBAMuJPINJgFhAAziXIrLvTvAz4PRo5IK/Log==
To create an environment for Ceph, set a command line editor. For example:
export EDITOR=vim
Then, use 'knife' to create an environment.
knife environment create {env-name}
For example:
knife environment create Ceph
A JSON file will appear. Perform the following steps:
- Enter a description for the environment.
- In '"default_attributes": {}', add '"ceph" : {}'.
- Within '"ceph" : {}', add '"monitor-secret":'.
- Immediately following '"monitor-secret":' add the key you generated within quotes, followed by a comma.
- Within '"ceph":{}' and following the 'monitor-secret' key-value pair, add '"config": {}'
- Within '"config": {}' add '"fsid":'.
- Immediately following '"fsid":', add the unique identifier you generated within quotes, followed by a comma.
- Within '"config": {}' and following the 'fsid' key-value pair, add '"mon_initial_members":'
- Immediately following '"mon_initial_members":', enter the initial monitor host names.
You may also set Ceph settings within '"config": {}'.
For example:
"default_attributes" : { "ceph": { "monitor-secret": "{replace-with-generated-secret}", "config": { "fsid": "{replace-with-generated-uuid}", "mon_initial_members": "{replace-with-monitor-hostname(s)}", "global": { "public network": "xxx.xxx.xxx.xxx/yy", "cluster network": "xxx.xxx.xxx.xxx/yy" }, "osd": { "osd journal size": "1000" } } } }
Will generate the following ceph.conf:
[global] fsid = <fsid> mon initial members = X,Y,Z mon host = ipX:port, ipY:port, ipZ:port ;mon host is auto generated public network = xxx.xxx.xxx.xxx/yy cluster network = xxx.xxx.xxx.xxx/yy [osd] osd journal size = 1000
Advanced users (i.e., developers and QA) may also add '"branch": "{branch}"'
to '"ceph": {}'. Valid values are 'stable', 'testing', 'dev'.
You can specify which stable release (e.g. argonaut, bobtail) or which dev
branch to use with '"version": "{version}"' within '"ceph": {}'.
If 'version' is not specified for 'stable', the latest stable release
will be used. 'testing' does not require 'version'.
The Ceph cookbook will, by default, use the official Ceph repositories. If you wish to use your own repositories you can specify them, in the Ceph environment, within '"ceph": {}' like so:
"platform_family": { "branch": { "repository": "mylocalcephrepo" "repository_key": "mylocalcephrepokey" } }platform_family can be:
- rhel: For RHEL and CentOS
- debian: For Debian and Ubuntu
- suse: For OpenSUSE and SLES
- fedora: For Fedora
branch can be: - stable
- testing
- dev
Configure the Roles¶
Navigate to the Ceph cookbooks directory. :
cd ~/chef-cookbooks/ceph
Create roles for OSDs, monitors, metadata servers, and RADOS Gateways from
their respective role files. :
knife role from file roles/ceph-osd.rb knife role from file roles/ceph-mon.rb knife role from file roles/ceph-mds.rb knife role from file roles/ceph-radosgw.rb
Configure Nodes¶
You must configure each node you intend to include in your Ceph cluster.
Identify nodes for your Ceph cluster. :
knife node list
.. note: for each host where you installed Chef and executed 'chef-client',
the Chef server should have a minimal node configuration. You can create
additional nodes with 'knife node create {node-name}'.
For each node you intend to use in your Ceph cluster, configure the node
as follows:
knife node edit {node-name}
The node configuration should appear in your text editor. Change the
'chef_environment' value to 'Ceph' (or whatever name you set for your
Ceph environment).
In the 'run_list', add '"recipe[ceph:repo]",' to all nodes as
the first setting, so that Chef can install or update the necessary packages.
Then, add at least one of:
"role[ceph-mon]" "role[ceph-osd]" "role[ceph-mds]" "role[ceph-radosgw]"
If you add more than one role, separate them with a comma. Run 'hostname'
on your command line, and replace the '{hostname}' setting of the 'name'
key to the host name for the node. :
{ "chef_environment": "Ceph", "name": "{hostname}", "normal": { "tags": [ ] }, "run_list": [ "role[ceph-mon]", "role[ceph-mds]" ] }
Deploy OSDs¶
Configuring a node with an OSD role tells Chef that the node will run at
least one OSD. However, you may run many OSDs on one host. For example,
you may run one 'ceph-osd' daemon for each data disk on the system.
To tell Chef to deploy OSDs, edit the node and add the following
within '"normal": {}':
"ceph": { "osd_devices": [ { "device": "/dev/...", "journal": "/dev/..." }, { "device": "/dev/...", "dmcrypt": true } ] }
Supported values are 'device', 'journal', 'dmcrypt' (deactivated by default).
.. note: dmcrypt is only supported starting with Cuttlefish
Run 'chef-client' on each Node¶
Once you have completed the preceding steps, you must run 'chef-client'
on each node. For example:
sudo chef-client
Proceed to Operating the Cluster¶
Once you complete the deployment, you may begin operating your cluster.
See 'Operating a Cluster' for details.
Managing Cookbooks with Knife: http://wiki.opscode.com/display/chef...oks+With+Knife
Updated by Jessica Mack almost 9 years ago · 1 revisions