Create a Scalable and Resilient Object Gateway with Ceph and VirtualBox » History » Version 10
Jessica Mack, 06/23/2015 02:40 AM
1 | 1 | Jessica Mack | h1. Create a Scalable and Resilient Object Gateway with Ceph and VirtualBox |
---|---|---|---|
2 | |||
3 | {{toc}} |
||
4 | |||
5 | h3. Introducing the Ceph Object Gateway |
||
6 | |||
7 | Ceph is a highly reliable distributed storage system, with self-healing and self-managing characteristics. One of its unique characteristics is its unified storage interface, which supports object storage, block device storage and file system storage all in the same Ceph cluster. Of course, it's also open source, so you can freely download and experiment with it at your leisure. |
||
8 | 6 | Jessica Mack | The "Ceph Object Gateway":http://ceph.com/docs/master/radosgw/ provides a way to host scalable data storage "buckets", similar to those provided by "Amazon Simple Storage Service (Amazon S3)":http://aws.amazon.com/s3/ and "OpenStack Swift":http://docs.openstack.org/developer/swift/. These objects are accessible via a REST API, making them ideal for cloud-based applications, big data storage and processing, and many other use cases. And because the underlying cluster infrastructure is managed by Ceph, fault-tolerance and scalability are guaranteed. |
9 | 1 | Jessica Mack | Setting up a Ceph object gateway can be a little complex, especially if you're unfamiliar with how scalable object storage works. That's where this tutorial comes in. Over the next few pages, I'll walk you through the process of setting up a Ceph-based object gateway and adding data to it. We'll set up the cluster using VirtualBox, so you'll get a chance to see Ceph's object storage features in action in a "real" environment where you have total control, but which doesn't cost you anything to run or scale out with new nodes. |
10 | Sounds good? Keep reading. |
||
11 | |||
12 | h3. Assumptions and Requirements |
||
13 | |||
14 | 6 | Jessica Mack | For this tutorial, I'll be using "VirtualBox":https://www.virtualbox.org/, which provides an easy way to set up independent virtual servers, with "CentOS":http://www.centos.org/ as the operating system for the virtual servers. VirtualBox is available for Windows, Linux, Macintosh, and Solaris hosts. I'll make the following assumptions: |
15 | * You have a working knowledge of CentOS, VirtualBox and VirtualBox networking. |
||
16 | * You have downloaded and installed the latest version of VirtualBox. |
||
17 | * You have either already configured 5 virtual CentOS servers, or you have downloaded an ISO installation image for the latest version of CentOS (CentOS 7.0 at the time of writing). These servers must be using kernel version 3.10 or later |
||
18 | * You're familiar with installing software using the yum, the CentOS package manager. |
||
19 | * You’re familiar with SSH-based authentication. |
||
20 | * You're familiar with object storage in the cloud. |
||
21 | |||
22 | 1 | Jessica Mack | In case you’re not familiar with the above topics, look in the “Read More” section at the end of this tutorial, which has links to relevant guides. |
23 | To set up a Ceph storage cluster with VirtualBox, here are the steps you'll follow: |
||
24 | 6 | Jessica Mack | # Create cluster nodes |
25 | # Install the Ceph deployment toolkit |
||
26 | # Configure authentication between cluster nodes |
||
27 | # Configure and activate a cluster monitor |
||
28 | # Prepare and activate OSDs |
||
29 | # Verify cluster health |
||
30 | # Test the cluster |
||
31 | # Install the Ceph object gateway |
||
32 | # Configure the Ceph object gateway |
||
33 | # Start working with buckets and objects |
||
34 | |||
35 | 1 | Jessica Mack | The next sections will walk you through these steps in detail. |
36 | |||
37 | h3. Step 1: Create Cluster Nodes |
||
38 | |||
39 | If you already have 5 virtual CentOS servers configured and talking to each other, you can skip this step. If not, you must first create the virtual servers that will make up your Ceph cluster. To do this: |
||
40 | |||
41 | 6 | Jessica Mack | 1. Launch VirtualBox and use the _Machine -> New_ menu to create a new virtual server. |
42 | |||
43 | 1 | Jessica Mack | !image1.jpg! |
44 | 2 | Jessica Mack | |
45 | 6 | Jessica Mack | 2. Keeping in mind that you will need 5 virtual servers running simultaneously, calculate the available RAM on the host system and set the server memory accordingly. |
46 | 1 | Jessica Mack | |
47 | !image2.jpg! |
||
48 | |||
49 | 6 | Jessica Mack | 3. Add a virtual hard drive of at least 10 GB. |
50 | 1 | Jessica Mack | |
51 | !image3.jpg! |
||
52 | |||
53 | 6 | Jessica Mack | 4. Ensure that you have an IDE controller with a virtual CD/DVD drive (to enable CentOS installation) and at least two network adapters, one NAT (to enable download of required software) and one bridged adapter or internal network adapter (for internal communication between the cluster nodes). |
54 | 5. Once the server basics are defined, install CentOS to the server using the ISO installation image. Ensure that your kernel version is at least 3.10 or later. |
||
55 | 6. Once the installation process is complete, log in to the server and configure the second network interface with a static IP address, by editing the appropriate template file in the _/etc/sysconfig/network-scripts/_ directory. Here's a sample of what the interface configuration might look like: |
||
56 | |||
57 | p(. @HWADDR=08:00:27:AE:14:41 |
||
58 | 1 | Jessica Mack | TYPE=Ethernet |
59 | BOOTPROTO=static |
||
60 | DEFROUTE=yes |
||
61 | PEERDNS=yes |
||
62 | PEERROUTES=yes |
||
63 | IPV4_FAILURE_FATAL=no |
||
64 | IPV6INIT=yes |
||
65 | IPV6_AUTOCONF=yes |
||
66 | IPV6_DEFROUTE=yes |
||
67 | IPV6_PEERDNS=yes |
||
68 | IPV6_PEERROUTES=yes |
||
69 | IPV6_FAILURE_FATAL=no |
||
70 | NAME=enp0s8 |
||
71 | UUID=5fc74119-1ab2-4c0c-9aa1-284fd484e6c6 |
||
72 | ONBOOT=no |
||
73 | IPADDR=192.168.1.25 |
||
74 | NETMASK=255.255.255.0 |
||
75 | GATEWAY=192.168.1.1 |
||
76 | DNS1=192.168.1.1 |
||
77 | 6 | Jessica Mack | DNS2=8.8.8.8@ |
78 | |||
79 | Should any of the above steps be unfamiliar to you, refer to the "VirtualBox manual":https://www.virtualbox.org/manual/UserManual.html, especially the "VirtualBox networking guide":https://www.virtualbox.org/manual/ch06.html, and to the networking section of the "CentOS deployment guide":http://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-networkscripts-interfaces.html. |
||
80 | |||
81 | Repeat this process until you have 5 virtual servers. Of these, identify one as the cluster administration node and assign it the hostname _admin-node_. The remaining servers may be identified with hostnames such as _node1_, _node2_, and so on. Here's an example of what the final cluster might look like (note that you should obviously modify the IP addresses to match your local network settings). |
||
82 | 1 | Jessica Mack | |
83 | 6 | Jessica Mack | |*Server host name*|*IP address*|*Purpose*| |
84 | |admin-node|192.168.1.25|Administration node for cluster| |
||
85 | |node1| 192.168.1.26|Monitor| |
||
86 | |node2| 192.168.1.27|OSD daemon| |
||
87 | |node3| 192.168.1.28|OSD daemon| |
||
88 | |node4| 192.168.1.29|Object gateway host / PHP client| |
||
89 | 1 | Jessica Mack | |
90 | 6 | Jessica Mack | Before proceeding to the next step, ensure that all the servers are accessible by pinging them using their host names. If you don't have a local DNS server, add the host names and IP addresses to each server's _/etc/hosts_ file to ease network access. |
91 | |||
92 | 1 | Jessica Mack | h3. Step 2: Install the Ceph Deployment Toolkit |
93 | |||
94 | The next step is to install the Ceph deployment toolkit on the administration node. This toolkit will help install Ceph on the nodes in the cluster, as well as prepare and activate the cluster. |
||
95 | 6 | Jessica Mack | 1. Log in to the administration node as the root user. |
96 | 2. Add the package to the yum repository by creating a new file at /etc/yum.repos.d/ceph.repo with the following content: |
||
97 | |||
98 | p(. @[ceph-noarch] |
||
99 | 1 | Jessica Mack | name=Ceph noarch packages |
100 | baseurl=http://ceph.com/rpm-firefly/el7/noarch |
||
101 | enabled=1 |
||
102 | gpgcheck=1 |
||
103 | type=rpm-md |
||
104 | 6 | Jessica Mack | gpgkey=https://ceph.com/git/?p=ceph.git;a=b...ys/release.asc@ |
105 | |||
106 | 3. Update the repository. |
||
107 | @shell> yum update@ |
||
108 | 4. Install the Ceph deployment toolkit. |
||
109 | @shell> yum install ceph-deploy@ |
||
110 | 1 | Jessica Mack | |
111 | !image4.jpg! |
||
112 | |||
113 | h3. Step 3: Configure Authentication between Cluster Nodes |
||
114 | |||
115 | 6 | Jessica Mack | Now, you need to create a _ceph_ user on each server in the cluster, including the administration node. This user account will handle performing cluster-related operations on each node. Perform the following steps on each of the 5 virtual servers: |
116 | 1. Log in as the _root_ user. |
||
117 | 2. Create a _ceph_ user account. |
||
118 | @shell> useradd ceph |
||
119 | shell> passwd ceph@ |
||
120 | 3. Give the _ceph_ user account root privileges with _sudo_. |
||
121 | @shell> echo "ceph ALL = (root) NOPASSWD:ALL" | tee /etc/sudoers.d/ceph |
||
122 | shell> chmod 0440 /etc/sudoers.d/ceph@ |
||
123 | 4. Disable 'requiretty' for the _ceph_ user. |
||
124 | @shell> sudo visudo@ |
||
125 | 5. In the resulting file, locate the line containing |
||
126 | @Defaults requiretty@ |
||
127 | 2 | Jessica Mack | and change it to read |
128 | 6 | Jessica Mack | @Defaults:ceph !requiretty@ |
129 | |||
130 | 1 | Jessica Mack | Now, set up passphraseless SSH between the nodes: |
131 | 6 | Jessica Mack | 1. Log in to the administration node as the _ceph_ user. |
132 | 2. Generate an SSH key for the administration node. |
||
133 | @shell> ssh-keygen@ |
||
134 | 3 | Jessica Mack | |
135 | !image52.jpg! |
||
136 | |||
137 | 6 | Jessica Mack | 3. Copy the generated public key to the _ceph_ user account of all the nodes in the cluster. |
138 | @shell> ssh-copy-id ceph@node1 |
||
139 | 3 | Jessica Mack | shell> ssh-copy-id ceph@node2 |
140 | shell> ssh-copy-id ceph@node3 |
||
141 | shell> ssh-copy-id ceph@node4 |
||
142 | 6 | Jessica Mack | shell> ssh-copy-id ceph@admin-node@ |
143 | 1 | Jessica Mack | |
144 | !image62.jpg! |
||
145 | |||
146 | 6 | Jessica Mack | 4. Test that the _ceph_ user on the administration node can log in to any other node as _ceph_ using SSH and without providing a password. |
147 | @shell> ssh ceph@node1@ |
||
148 | 1 | Jessica Mack | |
149 | !image72.jpg! |
||
150 | |||
151 | 6 | Jessica Mack | 5. Modify the administration node's SSH configuration file so that it can easily log in to each node as the ceph user. Create the _/home/ceph/.ssh/config_ file with the following lines: |
152 | |||
153 | p(. @Host node1 |
||
154 | 1 | Jessica Mack | Hostname node1 |
155 | User ceph |
||
156 | Host node2 |
||
157 | Hostname node2 |
||
158 | User ceph |
||
159 | Host node3 |
||
160 | Hostname node3 |
||
161 | User ceph |
||
162 | 3 | Jessica Mack | Host node4 |
163 | Hostname node4 |
||
164 | 1 | Jessica Mack | User ceph |
165 | Host admin-node |
||
166 | Hostname admin-node |
||
167 | 6 | Jessica Mack | User ceph@ |
168 | 1 | Jessica Mack | |
169 | 6 | Jessica Mack | 6. Change the permissions of the _/home/ceph/.ssh/config_ file. |
170 | @shell> chmod 0400 ~/.ssh/config@ |
||
171 | 7. Test that the _ceph_ user on the administration node can log in to any other node using SSH and without providing a password or username. |
||
172 | @shell> ssh node1@ |
||
173 | |||
174 | 1 | Jessica Mack | !image82.jpg! |
175 | |||
176 | Finally, create a directory on the administration node to store cluster information, such as configuration files and keyrings. |
||
177 | 6 | Jessica Mack | @shell> mkdir my-cluster |
178 | shell> cd my-cluster@ |
||
179 | 1 | Jessica Mack | You're now ready to begin preparing and activating the cluster! |
180 | |||
181 | 3 | Jessica Mack | h3. Step 4: Configure and Activate a Cluster Monitor |
182 | |||
183 | 1 | Jessica Mack | A Ceph storage cluster consists of two types of daemons: |
184 | 6 | Jessica Mack | * Monitors maintain copies of the cluster map |
185 | * Object Storage Daemons (OSD) store data as objects on storage nodes |
||
186 | |||
187 | Apart from this, other actors in a Ceph storage cluster include metadata servers and clients such as Ceph block devices, Ceph object gateways or Ceph filesystems. "Read more about Ceph’s architecture":http://ceph.com/docs/master/architecture/. |
||
188 | |||
189 | All the commands in this and subsequent sections are to be run when logged in as the _ceph_ user on the administration node, from the _my-cluster/_ directory. Ensure that you are directly logged in as _ceph_ and are not using root with _su - ceph_. |
||
190 | |||
191 | 1 | Jessica Mack | A minimal system will have at least one monitor and two OSD daemons for data replication. |
192 | 6 | Jessica Mack | 1. Begin by setting up a Ceph monitor on _node1_ with the Ceph deployment toolkit. |
193 | @shell> ceph-deploy new node1@ |
||
194 | 3 | Jessica Mack | This will define the name of the initial monitor node and create a default Ceph configuration file and monitor keyring in the current directory. |
195 | |||
196 | !image92.jpg! |
||
197 | 1 | Jessica Mack | |
198 | 6 | Jessica Mack | 2. Change the number of replicas in the Ceph configuration file at _/home/ceph/my-cluster/ceph.conf_ from 3 to 2 so that Ceph can achieve a stable state with just two OSDs. Add the following line in the [global] section: |
199 | @osd pool default size = 2 |
||
200 | osd pool default min size = 2@ |
||
201 | 3. In the same file, set the OSD journal size. A good general setting is 10 GB; however, since this is a simulation, you can use a smaller amount such as 4 GB. Add the following line in the [global] section: |
||
202 | @osd journal size = 4000@ |
||
203 | 4. In the same file, set the default number of placement groups for a pool. Since we’ll have less than 5 OSDs, 128 placement groups per pool should suffice. Add the following line in the [global] section: |
||
204 | @osd pool default pg num = 128@ |
||
205 | 5. Install Ceph on each node in the cluster, including the administration node. |
||
206 | @shell> ceph-deploy install admin-node node1 node2 node3 node4@ |
||
207 | 1 | Jessica Mack | The Ceph deployment toolkit will now go to work installing Ceph on each node. Here's an example of what you will see during the installation process. |
208 | |||
209 | !image102.jpg! |
||
210 | |||
211 | 6 | Jessica Mack | Create the Ceph monitor on _node1_ and gather the initial keys. |
212 | @shell> ceph-deploy mon create-initial node1@ |
||
213 | 4 | Jessica Mack | |
214 | 1 | Jessica Mack | !image112.jpg! |
215 | 4 | Jessica Mack | |
216 | h3. Step 5: Prepare and Activate OSDs |
||
217 | |||
218 | 7 | Jessica Mack | The next set is to prepare and activate Ceph OSDs. We'll need a minimum of 2 OSDs, and these should be set up on _node2_ and _node3_, as it's not recommended to mix monitors and OSD daemons on the same host. To begin, set up an OSD on _node2_ as follows: |
219 | 1. Log into _node2_ as the _ceph_ user. |
||
220 | @shell> ssh node2@ |
||
221 | 2. Create a directory for the OSD daemon. |
||
222 | @shell> sudo mkdir /var/local/osd@ |
||
223 | 3. Log out of _node2_. Then, from the administrative node, prepare and activate the OSD. |
||
224 | @shell> ceph-deploy osd prepare node2:/var/local/osd@ |
||
225 | 4 | Jessica Mack | |
226 | !image122.jpg! |
||
227 | |||
228 | 7 | Jessica Mack | @shell> ceph-deploy osd activate node2:/var/local/osd@ |
229 | 1 | Jessica Mack | |
230 | !image132.jpg! |
||
231 | |||
232 | 7 | Jessica Mack | Repeat the above steps for _node3_. |
233 | 1 | Jessica Mack | At this point, the OSD daemons have been created and the storage cluster is ready. |
234 | 4 | Jessica Mack | |
235 | h3. Step 6: Verify Cluster Health |
||
236 | 1 | Jessica Mack | |
237 | Copy the configuration file and admin keyring from the administration node to all the nodes in the cluster. |
||
238 | 7 | Jessica Mack | @shell> ceph-deploy admin admin-node node1 node2 node3 node4@ |
239 | 1 | Jessica Mack | |
240 | !image142.jpg! |
||
241 | |||
242 | 7 | Jessica Mack | Log in to each node as the _ceph_ user and change the permissions of the admin keyring. |
243 | @shell> ssh node1 |
||
244 | shell> sudo chmod +r /etc/ceph/ceph.client.admin.keyring@ |
||
245 | You should now be able to check cluster health from any node in the cluster with the _ceph status_ command. Ideally, you want to see the status _active + clean_, as that indicates the cluster is operating normally. |
||
246 | @shell> ceph status@ |
||
247 | 1 | Jessica Mack | |
248 | !image152.jpg! |
||
249 | |||
250 | h3. Step 7: Test the Cluster |
||
251 | |||
252 | You can now perform a simple test to see the distributed Ceph storage cluster in action, by writing a file on one node and retrieving it on another: |
||
253 | 7 | Jessica Mack | 1. Log in to _node1_ as the _ceph_ user. |
254 | @shell> ssh node1@ |
||
255 | 2. Create a new file with some dummy data. |
||
256 | @shell> echo "Hello world" > /tmp/hello.txt@ |
||
257 | 3. Data is stored in Ceph within storage pools, which are logical groups in which to organize your data. By default, a Ceph storage cluster has 3 pools - _data_, _metadata_ and _rbd_ - and it's also possible to create your own custom pools. In this case, copy the file to the _data_ pool with the _rados put_ command and assign it a name. |
||
258 | @shell> rados put hello-object /tmp/hello.txt --pool data@ |
||
259 | |||
260 | 1 | Jessica Mack | To verify that the Ceph storage cluster stored the object: |
261 | 7 | Jessica Mack | 1. Log in to _node2_ as the _ceph_ user. |
262 | 2. Check that the file exists in the cluster's _data_ storage pool with the _rados ls_ command. |
||
263 | @shell> rados ls --pool data@ |
||
264 | 3. Copy the file out of the storage cluster to a local directory with the _rados get_ command and verify its contents |
||
265 | @shell> rados get hello-object /tmp/hello.txt --pool data |
||
266 | shell> cat hello.txt@ |
||
267 | 1 | Jessica Mack | |
268 | !image162.jpg! |
||
269 | |||
270 | h3. Step 8: Install the Ceph Object Gateway |
||
271 | |||
272 | 7 | Jessica Mack | Now that the cluster is operating, it’s time to do something with it. First, you must install and configure an Apache Web server with FastCGI on _node4_, as described below. |
273 | |||
274 | 1. Log into _node4_ as the _ceph_ user. |
||
275 | @shell> ssh node4@ |
||
276 | 2. Install Apache and FastCGI from the Ceph repositories. To do this, you need to first install the _yum priorities_ plugin, then add the repositories to your _yum_ repository list. |
||
277 | @shell> sudo yum install yum-plugin-priorities@ |
||
278 | 3. Edit the _/etc/yum/pluginconf.d/priorities.conf_ file and ensure it looks like this: |
||
279 | @[main] |
||
280 | enabled = 1@ |
||
281 | 4. Create a file at _/etc/yum.repos.d/ceph-apache.repo_ and fill it with the following content: |
||
282 | @[apache2-ceph-noarch] |
||
283 | 1 | Jessica Mack | name=Apache noarch packages for Ceph |
284 | 7 | Jessica Mack | baseurl=http://gitbuilder.ceph.com/apache2-rpm-centos7-x86_64-basic/ref/master |
285 | 1 | Jessica Mack | enabled=1 |
286 | priority=2 |
||
287 | gpgcheck=1 |
||
288 | type=rpm-md |
||
289 | 7 | Jessica Mack | gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc |
290 | 1 | Jessica Mack | [apache2-ceph-source] |
291 | name=Apache source packages for Ceph |
||
292 | 7 | Jessica Mack | baseurl=http://gitbuilder.ceph.com/apache2-rpm-centos7-x86_64-basic/ref/master |
293 | 1 | Jessica Mack | enabled=0 |
294 | priority=2 |
||
295 | gpgcheck=1 |
||
296 | type=rpm-md |
||
297 | 7 | Jessica Mack | gpgkey=https://git.ceph.com/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc@ |
298 | 5. Create a file at /etc/yum.repos.d/ceph-fastcgi.repo and fill it with the following content: |
||
299 | @[fastcgi-ceph-basearch] |
||
300 | 1 | Jessica Mack | name=FastCGI basearch packages for Ceph |
301 | 8 | Jessica Mack | baseurl=http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/ |
302 | 1 | Jessica Mack | enabled=1 |
303 | priority=2 |
||
304 | gpgcheck=1 |
||
305 | type=rpm-md |
||
306 | 8 | Jessica Mack | gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc |
307 | 1 | Jessica Mack | [fastcgi-ceph-noarch] |
308 | name=FastCGI noarch packages for Ceph |
||
309 | 8 | Jessica Mack | baseurl=http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master |
310 | 1 | Jessica Mack | enabled=1 |
311 | priority=2 |
||
312 | gpgcheck=1 |
||
313 | type=rpm-md |
||
314 | 8 | Jessica Mack | gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc |
315 | 1 | Jessica Mack | [fastcgi-ceph-source] |
316 | name=FastCGI source packages for Ceph |
||
317 | 8 | Jessica Mack | baseurl=http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master |
318 | 1 | Jessica Mack | enabled=0 |
319 | priority=2 |
||
320 | gpgcheck=1 |
||
321 | type=rpm-md |
||
322 | 8 | Jessica Mack | gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc@ |
323 | 7 | Jessica Mack | 6. Update the repository and install Apache and FastCGI. |
324 | @shell> sudo yum update |
||
325 | shell> sudo yum install httpd mod_fastcgi@ |
||
326 | 7. Edit the _/etc/httpd/conf/httpd.conf_ file and modify the ServerName directive to reflect the server's host name. Uncomment the line if needed. |
||
327 | @ServerName node4@ |
||
328 | 8. Review the files in the _/etc/httpd/conf.modules.d/*_ directory to ensure that Apache's URL rewriting and FastCGI modules are enabled. You should find the following entries in the files: |
||
329 | @LoadModule rewrite_module modules/mod_rewrite.so |
||
330 | LoadModule fastcgi_module modules/mod_fastcgi.so@ |
||
331 | 1 | Jessica Mack | In case these entries don't exist, add them to the end of the /etc/httpd/conf/httpd.conf file. |
332 | 7 | Jessica Mack | 9. Restart Apache. |
333 | @shell> sudo service httpd restart@ |
||
334 | |||
335 | Amazon S3 lets you refer to buckets using subdomains, such as _http://mybucket.s3.amazonaws.com_. You can also accomplish this with Ceph, but you must first install a local DNS server like dnsmasq and add support for wildcard subdomains. Follow these steps: |
||
336 | |||
337 | 1. Log into _node4_ as the _ceph_ user. |
||
338 | @shell> ssh node4@ |
||
339 | 2. Install _dnsmasq_. |
||
340 | @shell> yum install dnsmasq@ |
||
341 | 3. Edit the _dnsmasq_ configuration file at _/etc/dnsmasq.conf_ and add the following line to the end of the file: |
||
342 | @address=/.node4/192.168.1.29@ |
||
343 | 4. Save the file and restart _dnsmasq_. |
||
344 | @shell> sudo service dnsmasq restart@ |
||
345 | 5. If necessary, update the _/etc/resolv.conf_ file on the client host so that it knows about the new DNS server. |
||
346 | @nameserver 192.168.1.29@ |
||
347 | You should now be able to successfully ping any subdomain of _*.node4_, such as _mybucket.node4_ or _example.node4_, as shown in the image below. |
||
348 | 1 | Jessica Mack | |
349 | !image172.png! |
||
350 | |||
351 | 7 | Jessica Mack | TIP: If you're not able to configure wildcard subdomains, you can also simply decide a list of subdomains you wish to use and then add them as static entries to the client system's _/etc/hosts_ file. Ensure that the entries resolve to the _node4_ virtual host. |
352 | |||
353 | The final step is to install _radosgw_ on _node4_: |
||
354 | @shell> ssh node4 |
||
355 | shell> sudo yum install ceph-radosgw@ |
||
356 | 1 | Jessica Mack | At this point, you have a Web server running with the Ceph object gateway and FastCGI support, and subdomains that resolve to the object gateway host. |
357 | |||
358 | h3. Step 9: Configure the Ceph Object Gateway |
||
359 | |||
360 | The next step is to configure the Ceph Object Gateway daemon. Follow these steps: |
||
361 | 8 | Jessica Mack | 1. Log into the administration node as the _ceph_ user. |
362 | @shell> ssh admin-node@ |
||
363 | 2. Create a keyring for the gateway. |
||
364 | @shell> sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring |
||
365 | shell> sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring@ |
||
366 | 3. Generate a user name and key to use when accessing the gateway. For this example, the user name is _client.radosgw.gateway_. |
||
367 | @shell> sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key@ |
||
368 | 4. Add read and write capabilities to the new key: |
||
369 | @shell> sudo ceph-authtool -n client.radosgw.gateway --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring@ |
||
370 | 5. Add the new key to the storage cluster and distribute it to the object gateway node. |
||
371 | @shell> sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring |
||
372 | 1 | Jessica Mack | shell> sudo scp /etc/ceph/ceph.client.radosgw.keyring ceph@node4:/home/ceph |
373 | shell> ssh node4 |
||
374 | shell> sudo mv ceph.client.radosgw.keyring /etc/ceph/ceph.client.radosgw.keyring |
||
375 | 8 | Jessica Mack | shell> exit@ |
376 | This process should also have created a number of storage pools for the gateway. You can verify this by running the following command and verifying that the output includes various _.rgw_ pools. |
||
377 | @shell> rados lspools@ |
||
378 | 1 | Jessica Mack | |
379 | !image182.png! |
||
380 | |||
381 | 8 | Jessica Mack | 6. Change to your cluster configuration directory. |
382 | @shell> cd ~/my-cluster@ |
||
383 | 7. Edit the Ceph configuration file at _~/my-cluster/ceph/ceph.conf_ and add a new [client.radosgw.gateway] section to it, as below: |
||
384 | @[client.radosgw.gateway] |
||
385 | 1 | Jessica Mack | host = node4 |
386 | keyring = /etc/ceph/ceph.client.radosgw.keyring |
||
387 | rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock |
||
388 | log file = /var/log/radosgw/client.radosgw.gateway.log |
||
389 | rgw dns name = node4 |
||
390 | 8 | Jessica Mack | rgw print continue = false@ |
391 | 8. Transmit the new Ceph configuration file to all the other nodes in the cluster. |
||
392 | @shell> ceph-deploy config push admin-node node1 node2 node3 node4@ |
||
393 | 9. Log into _node4_ as the _ceph_ user. |
||
394 | @shell> ssh node4@ |
||
395 | 10. Add a Ceph object gateway script, by creating a file at _/var/www/html/s3gw.fcgi_ with the following content: |
||
396 | @#!/bin/sh |
||
397 | exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway@ |
||
398 | 11. Change the permissions of the script to make it executable. |
||
399 | @shell> sudo chmod +x /var/www/html/s3gw.fcgi@ |
||
400 | 12. Create a data directory for the _radosgw_ daemon. |
||
401 | @shell> sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.gateway@ |
||
402 | 13. Add a gateway configuration file, by creating a file at _/etc/httpd/conf.d/rgw.conf_ and filling it with the following content: |
||
403 | 9 | Jessica Mack | @FastCgiExternalServer /var/www/html/s3gw.fcgi -socket /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock@ |
404 | <pre> |
||
405 | 1 | Jessica Mack | <VirtualHost *:80> |
406 | ServerName node4 |
||
407 | ServerAlias *.node4 |
||
408 | ServerAdmin admin@localhost |
||
409 | DocumentRoot /var/www/html |
||
410 | RewriteEngine On |
||
411 | RewriteRule ^/(.*) /s3gw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] |
||
412 | <IfModule mod_fastcgi.c> |
||
413 | <Directory /var/www/html> |
||
414 | Options +ExecCGI |
||
415 | AllowOverride All |
||
416 | SetHandler fastcgi-script |
||
417 | Order allow,deny |
||
418 | Allow from all |
||
419 | AuthBasicAuthoritative Off |
||
420 | </Directory> |
||
421 | </IfModule> |
||
422 | AllowEncodedSlashes On |
||
423 | ErrorLog /var/log/httpd/error.log |
||
424 | CustomLog /var/log/httpd/access.log combined |
||
425 | ServerSignature Off |
||
426 | </VirtualHost> |
||
427 | <VirtualHost *:443> |
||
428 | ServerName node4 |
||
429 | ServerAlias *.node4 |
||
430 | ServerAdmin admin@localhost |
||
431 | DocumentRoot /var/www/html |
||
432 | RewriteEngine On |
||
433 | RewriteRule ^/(.*) /s3gw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] |
||
434 | <IfModule mod_fastcgi.c> |
||
435 | <Directory /var/www/html> |
||
436 | Options +ExecCGI |
||
437 | AllowOverride All |
||
438 | SetHandler fastcgi-script |
||
439 | Order allow,deny |
||
440 | Allow from all |
||
441 | AuthBasicAuthoritative Off |
||
442 | </Directory> |
||
443 | 5 | Jessica Mack | </IfModule> |
444 | AllowEncodedSlashes On |
||
445 | ErrorLog /var/log/httpd/error.log |
||
446 | 1 | Jessica Mack | CustomLog /var/log/httpd/access.log combined |
447 | ServerSignature Off |
||
448 | SSLEngine on |
||
449 | SSLCertificateFile /etc/apache2/ssl/apache.crt |
||
450 | SSLCertificateKeyFile /etc/apache2/ssl/apache.key |
||
451 | SetEnv SERVER_PORT_SECURE 443 |
||
452 | 9 | Jessica Mack | </VirtualHost> |
453 | </pre> |
||
454 | |||
455 | 8 | Jessica Mack | 14. Edit the _/etc/httpd/conf.d/fastcgi.conf_ file and ensure that the line referencing the FastCgiWrapper looks like this: |
456 | @FastCgiWrapper off@ |
||
457 | 15. Restart the Apache server, followed by the radosgw daemon. |
||
458 | @shell> sudo service httpd restart |
||
459 | shell> sudo /etc/init.d/ceph-radosgw restart@ |
||
460 | 5 | Jessica Mack | You can quickly test that the object gateway is running by sending an HTTP GET request to the Web server, as shown below: |
461 | 1 | Jessica Mack | |
462 | !image192.png! |
||
463 | |||
464 | At this point, your Ceph object gateway is running and you can begin using it. |
||
465 | |||
466 | h3. Step 10: Start Working with Buckets and Objects |
||
467 | |||
468 | Before you can begin using the Ceph object gateway, you must create a user account. |
||
469 | 10 | Jessica Mack | 1. Log in to _node4_ as the _ceph_ user. |
470 | @shell> ssh admin-node@ |
||
471 | 2. Create a new user account using the _radosgw-admin_ command. In this example, the user is named 'john'. |
||
472 | @shell> radosgw-admin user create --uid=john --display-name="Example User"@ |
||
473 | 1 | Jessica Mack | Here's an example of what you should see. Note the access key and secret key in the output, as you will need this to access the object gateway from another client. |
474 | |||
475 | !image202.png! |
||
476 | |||
477 | You can also verify that the user was created with the following command: |
||
478 | 10 | Jessica Mack | @shell> radosgw-admin user info --uid=john@ |
479 | While you can interact with the object gateway directly over HTTP, by sending authenticated GET, PUT and DELETE requests to the gateway endpoints, an easier way is with "Amazon's AWS SDK":http://aws.amazon.com/developers/. This SDK includes classes and constructs to help you work with buckets and objects in Amazon S3. Since the Ceph object gateway is S3-compatible, you can use the same SDK to interact with it as well. |
||
480 | The AWS SDK is available for multiple programming languages. In the examples that follow, I'll use the "AWS SDK for PHP":http://aws.amazon.com/sdk-for-php/, but you will find code examples for other languages as well on the "AWS developer website":http://aws.amazon.com/developers/getting-started/. |
||
481 | 1. Log in to _node4_ (which will now also double as the client node) as the _root_ user and install PHP and related tools. |
||
482 | @shell> sudo yum install php curl php-curl@ |
||
483 | 2. Create a working directory for your PHP files. Download Composer, the PHP dependency manager, into this directory. |
||
484 | @shell> cd /tmp |
||
485 | 1 | Jessica Mack | shell> mkdir ceph |
486 | shell> cd ceph |
||
487 | 10 | Jessica Mack | shell> curl -sS https://getcomposer.org/installer | php@ |
488 | 3. Create a _composer.json_ file in the working directory and fill it with the following content: |
||
489 | <pre> |
||
490 | 1 | Jessica Mack | { |
491 | "require": { |
||
492 | "aws/aws-sdk-php": "2.*" |
||
493 | } |
||
494 | } |
||
495 | 10 | Jessica Mack | </pre> |
496 | 4. Download the AWS SDK for PHP and related dependencies using Composer: |
||
497 | @shell> cd /tmp/ceph |
||
498 | shell> php composer.phar install@ |
||
499 | 1 | Jessica Mack | You can now begin interacting with your object gateway using PHP. For example, here's a simple PHP script to create a new bucket: |
500 | 10 | Jessica Mack | <pre><code class="php"> |
501 | 1 | Jessica Mack | <?php |
502 | // create-bucket.php |
||
503 | // autoload files |
||
504 | require 'vendor/autoload.php'; |
||
505 | use Aws\S3\S3Client; |
||
506 | // instantiate S3 client |
||
507 | $s3 = S3Client::factory(array( |
||
508 | 5 | Jessica Mack | 'key' => 'YOUR_ACCESS_KEY', |
509 | 1 | Jessica Mack | 'secret' => 'YOUR_SECRET_KEY', |
510 | 'endpoint' => 'http://node4' |
||
511 | 5 | Jessica Mack | )); |
512 | // create bucket |
||
513 | 1 | Jessica Mack | try { |
514 | $s3->createBucket(array('Bucket' => 'mybucket')); |
||
515 | echo "Bucket created \n"; |
||
516 | } catch (Aws\S3\Exception\S3Exception $e) { |
||
517 | echo "Request failed: $e"; |
||
518 | } |
||
519 | 10 | Jessica Mack | </code></pre> |
520 | 1 | Jessica Mack | This script begins by initializing the Composer auto-loader and an instance of the S3Client object. The object is provided with the access key and secret for the user created earlier, and a custom endpoint points to the object gateway Web server. |
521 | 10 | Jessica Mack | The S3Client object provides a number of methods to create and manage buckets and objects. One of these is the _createBucket()_ method, which accepts a bucket name and generates the necessary PUT request to create a new bucket in the object gateway. |
522 | 1 | Jessica Mack | You can run this script at the console as follows: |
523 | 10 | Jessica Mack | @shell> php create-bucket.php@ |
524 | 1 | Jessica Mack | Here's an example of what the output might look like: |
525 | |||
526 | !image212.png! |
||
527 | |||
528 | 10 | Jessica Mack | You can also create a bucket and then add a file to it as an object, using the client object's _upload()_ method. Here's an example: |
529 | <pre><code class="php"> |
||
530 | 1 | Jessica Mack | <?php |
531 | // create-bucket-object.php |
||
532 | // autoload files |
||
533 | require 'vendor/autoload.php'; |
||
534 | use Aws\S3\S3Client; |
||
535 | // instantiate S3 client |
||
536 | $s3 = S3Client::factory(array( |
||
537 | 'key' => 'YOUR_ACCESS_KEY', |
||
538 | 'secret' => 'YOUR_SECRET_KEY', |
||
539 | 'endpoint' => 'http://node4' |
||
540 | )); |
||
541 | // create bucket and upload file to it |
||
542 | try { |
||
543 | $s3->createBucket(array('Bucket' => 'myotherbucket')); |
||
544 | $s3->upload('myotherbucket', 'test.tgz', file_get_contents('/tmp/test.tgz'), 'public-read'); |
||
545 | echo 'Bucket and object created'; |
||
546 | } catch (Aws\S3\Exception\S3Exception $e) { |
||
547 | echo "Request failed: $e"; |
||
548 | } |
||
549 | 10 | Jessica Mack | </code></pre> |
550 | Of course, you can also list all the buckets and objects available to the authenticated user with the _listBuckets()_ and _listObjects()_ methods: |
||
551 | <pre><code class="php"> |
||
552 | 1 | Jessica Mack | <?php |
553 | // list-bucket-contents.php |
||
554 | // autoload files |
||
555 | require 'vendor/autoload.php'; |
||
556 | use Aws\S3\S3Client; |
||
557 | // instantiate S3 client |
||
558 | $s3 = S3Client::factory(array( |
||
559 | 'key' => 'YOUR_ACCESS_KEY', |
||
560 | 'secret' => 'YOUR_SECRET_KEY', |
||
561 | 'endpoint' => 'http://node4' |
||
562 | )); |
||
563 | 5 | Jessica Mack | // create bucket and upload file to it |
564 | try { |
||
565 | $bucketsColl = $s3->listBuckets(); |
||
566 | 1 | Jessica Mack | foreach ($bucketsColl['Buckets'] as $bucket) { |
567 | echo $bucket['Name'] . "\n"; |
||
568 | $objColl = $s3->listObjects(array('Bucket' => $bucket['Name'])); |
||
569 | if ($objColl['Contents']) { |
||
570 | foreach ($objColl['Contents'] as $obj) { |
||
571 | echo '- ' . $obj['Key'] . "\n"; |
||
572 | } |
||
573 | } |
||
574 | } |
||
575 | } catch (Aws\S3\Exception\S3Exception $e) { |
||
576 | echo "Request failed: $e"; |
||
577 | } |
||
578 | 10 | Jessica Mack | </code></pre> |
579 | 1 | Jessica Mack | Here's an example of what the output might look like: |
580 | |||
581 | !image222.png! |
||
582 | |||
583 | 10 | Jessica Mack | Of course, you can do a lot more with the AWS SDK for PHP. "Refer to the reference documentation":http://docs.aws.amazon.com/aws-sdk-php/latest/ for a complete list of methods and example code. |
584 | 1 | Jessica Mack | |
585 | h3. Conclusion |
||
586 | |||
587 | As this tutorial has illustrated, Ceph makes it easy to set up a standards-compliant object gateway for your applications or users, with all the benefits of a resilient, infinitely scalable underlying storage cluster. |
||
588 | 10 | Jessica Mack | The simple object gateway you created here with VirtualBox is just the tip of the iceberg: you can transition your object gateway to the cloud and run it in "federated mode":http://ceph.com/docs/master/radosgw/federated-config/ across regions and zones for even greater flexibility, and because the Ceph object gateway is also "Swift-compliant":http://ceph.com/docs/master/radosgw/swift/, you can maximize compatibility for OpenStack users without any changes to your existing infrastructure. And of course, you can also use the underlying object storage cluster for fault-tolerant "Ceph block devices":http://ceph.com/docs/master/rados/ or the POSIX-compliant "CephFS filesystem":http://ceph.com/docs/master/cephfs/. |
589 | 1 | Jessica Mack | The bottom line: Ceph's unique architecture gives you improved performance and flexibility without any loss in reliability and security. And it's open source, so you can experiment with it, improve it and use it without worrying about vendor lock-in. You can't get any better than that! |
590 | |||
591 | h3. Read More |
||
592 | |||
593 | "Introduction to Ceph":http://ceph.com/docs/master/start/intro/ |
||
594 | "Ceph Architecture":http://ceph.com/docs/master/architecture/ |
||
595 | "Getting Started With Ceph":http://www.inktank.com/resource/getting-started-with-ceph-miroslav-klivansky/ |
||
596 | "Introduction to Ceph & OpenStack":http://www.inktank.com/resource/introduction-to-ceph-openstack-miroslav-klivansky/ |
||
597 | "Managing A Distributed Storage System At Scale":http://www.inktank.com/resource/managing-a-distributed-storage-system-at-scale-sage-weil/ |
||
598 | "Scaling Storage With Ceph":http://www.inktank.com/resource/scaling-storage-with-ceph-ross-turk/ |
||
599 | "Ceph API Documentation":http://ceph.com/docs/master/api/ |