Create a Scalable and Resilient Object Gateway with Ceph and VirtualBox » History » Version 2
Jessica Mack, 06/22/2015 11:06 PM
1 | 1 | Jessica Mack | h1. Create a Scalable and Resilient Object Gateway with Ceph and VirtualBox |
---|---|---|---|
2 | |||
3 | {{toc}} |
||
4 | |||
5 | h3. Introducing the Ceph Object Gateway |
||
6 | |||
7 | Ceph is a highly reliable distributed storage system, with self-healing and self-managing characteristics. One of its unique characteristics is its unified storage interface, which supports object storage, block device storage and file system storage all in the same Ceph cluster. Of course, it's also open source, so you can freely download and experiment with it at your leisure. |
||
8 | The Ceph Object Gateway provides a way to host scalable data storage "buckets", similar to those provided by Amazon Simple Storage Service (Amazon S3) and OpenStack Swift. These objects are accessible via a REST API, making them ideal for cloud-based applications, big data storage and processing, and many other use cases. And because the underlying cluster infrastructure is managed by Ceph, fault-tolerance and scalability are guaranteed. |
||
9 | Setting up a Ceph object gateway can be a little complex, especially if you're unfamiliar with how scalable object storage works. That's where this tutorial comes in. Over the next few pages, I'll walk you through the process of setting up a Ceph-based object gateway and adding data to it. We'll set up the cluster using VirtualBox, so you'll get a chance to see Ceph's object storage features in action in a "real" environment where you have total control, but which doesn't cost you anything to run or scale out with new nodes. |
||
10 | Sounds good? Keep reading. |
||
11 | |||
12 | h3. Assumptions and Requirements |
||
13 | |||
14 | For this tutorial, I'll be using VirtualBox, which provides an easy way to set up independent virtual servers, with CentOS as the operating system for the virtual servers. VirtualBox is available for Windows, Linux, Macintosh, and Solaris hosts. I'll make the following assumptions: |
||
15 | You have a working knowledge of CentOS, VirtualBox and VirtualBox networking. |
||
16 | You have downloaded and installed the latest version of VirtualBox. |
||
17 | You have either already configured 5 virtual CentOS servers, or you have downloaded an ISO installation image for the latest version of CentOS (CentOS 7.0 at the time of writing). These servers must be using kernel version 3.10 or later |
||
18 | You're familiar with installing software using the yum, the CentOS package manager. |
||
19 | You’re familiar with SSH-based authentication. |
||
20 | You're familiar with object storage in the cloud. |
||
21 | In case you’re not familiar with the above topics, look in the “Read More” section at the end of this tutorial, which has links to relevant guides. |
||
22 | To set up a Ceph storage cluster with VirtualBox, here are the steps you'll follow: |
||
23 | Create cluster nodes |
||
24 | Install the Ceph deployment toolkit |
||
25 | Configure authentication between cluster nodes |
||
26 | Configure and activate a cluster monitor |
||
27 | Prepare and activate OSDs |
||
28 | Verify cluster health |
||
29 | Test the cluster |
||
30 | Install the Ceph object gateway |
||
31 | Configure the Ceph object gateway |
||
32 | Start working with buckets and objects |
||
33 | The next sections will walk you through these steps in detail. |
||
34 | |||
35 | h3. Step 1: Create Cluster Nodes |
||
36 | |||
37 | If you already have 5 virtual CentOS servers configured and talking to each other, you can skip this step. If not, you must first create the virtual servers that will make up your Ceph cluster. To do this: |
||
38 | Launch VirtualBox and use the Machine -> New menu to create a new virtual server. |
||
39 | |||
40 | 2 | Jessica Mack | !image1.jpg! |
41 | |||
42 | 1 | Jessica Mack | Keeping in mind that you will need 5 virtual servers running simultaneously, calculate the available RAM on the host system and set the server memory accordingly. |
43 | |||
44 | 2 | Jessica Mack | !image2.jpg! |
45 | |||
46 | 1 | Jessica Mack | Add a virtual hard drive of at least 10 GB. |
47 | |||
48 | 2 | Jessica Mack | !image3.jpg! |
49 | |||
50 | 1 | Jessica Mack | Ensure that you have an IDE controller with a virtual CD/DVD drive (to enable CentOS installation) and at least two network adapters, one NAT (to enable download of required software) and one bridged adapter or internal network adapter (for internal communication between the cluster nodes). |
51 | Once the server basics are defined, install CentOS to the server using the ISO installation image. Ensure that your kernel version is at least 3.10 or later. |
||
52 | Once the installation process is complete, log in to the server and configure the second network interface with a static IP address, by editing the appropriate template file in the /etc/sysconfig/network-scripts/ directory. Here's a sample of what the interface configuration might look like: |
||
53 | HWADDR=08:00:27:AE:14:41 |
||
54 | TYPE=Ethernet |
||
55 | BOOTPROTO=static |
||
56 | DEFROUTE=yes |
||
57 | PEERDNS=yes |
||
58 | PEERROUTES=yes |
||
59 | IPV4_FAILURE_FATAL=no |
||
60 | IPV6INIT=yes |
||
61 | IPV6_AUTOCONF=yes |
||
62 | IPV6_DEFROUTE=yes |
||
63 | IPV6_PEERDNS=yes |
||
64 | IPV6_PEERROUTES=yes |
||
65 | IPV6_FAILURE_FATAL=no |
||
66 | NAME=enp0s8 |
||
67 | UUID=5fc74119-1ab2-4c0c-9aa1-284fd484e6c6 |
||
68 | ONBOOT=no |
||
69 | IPADDR=192.168.1.25 |
||
70 | NETMASK=255.255.255.0 |
||
71 | GATEWAY=192.168.1.1 |
||
72 | DNS1=192.168.1.1 |
||
73 | DNS2=8.8.8.8 |
||
74 | Should any of the above steps be unfamiliar to you, refer to the VirtualBox manual, especially the VirtualBox networking guide, and to the networking section of the CentOS deployment guide. |
||
75 | Repeat this process until you have 5 virtual servers. Of these, identify one as the cluster administration node and assign it the hostname admin-node. The remaining servers may be identified with hostnames such as node1, node2, and so on. Here's an example of what the final cluster might look like (note that you should obviously modify the IP addresses to match your local network settings). |
||
76 | |||
77 | Server host name IP address |
||
78 | Purpose |
||
79 | admin-node 192.168.1.25 Administration node for cluster |
||
80 | node1 192.168.1.26 Monitor |
||
81 | node2 192.168.1.27 OSD daemon |
||
82 | node3 192.168.1.28 OSD daemon |
||
83 | node4 192.168.1.29 |
||
84 | Object gateway host / PHP client |
||
85 | Before proceeding to the next step, ensure that all the servers are accessible by pinging them using their host names. If you don't have a local DNS server, add the host names and IP addresses to each server's /etc/hosts file to ease network access. |
||
86 | |||
87 | h3. Step 2: Install the Ceph Deployment Toolkit |
||
88 | |||
89 | The next step is to install the Ceph deployment toolkit on the administration node. This toolkit will help install Ceph on the nodes in the cluster, as well as prepare and activate the cluster. |
||
90 | Log in to the administration node as the root user. |
||
91 | Add the package to the yum repository by creating a new file at /etc/yum.repos.d/ceph.repo with the following content: |
||
92 | [ceph-noarch] |
||
93 | name=Ceph noarch packages |
||
94 | baseurl=http://ceph.com/rpm-firefly/el7/noarch |
||
95 | enabled=1 |
||
96 | gpgcheck=1 |
||
97 | type=rpm-md |
||
98 | gpgkey=https://ceph.com/git/?p=ceph.git;a=b...ys/release.asc |
||
99 | Update the repository. |
||
100 | shell> yum update |
||
101 | Install the Ceph deployment toolkit. |
||
102 | shell> yum install ceph-deploy |
||
103 | |||
104 | 2 | Jessica Mack | !image4.jpg! |
105 | 1 | Jessica Mack | |
106 | h3. Step 3: Configure Authentication between Cluster Nodes |
||
107 | |||
108 | Now, you need to create a ceph user on each server in the cluster, including the administration node. This user account will handle performing cluster-related operations on each node. Perform the following steps on each of the 5 virtual servers: |
||
109 | Log in as the root user. |
||
110 | Create a ceph user account. |
||
111 | shell> useradd ceph |
||
112 | shell> passwd ceph |
||
113 | Give the ceph user account root privileges with sudo. |
||
114 | shell> echo "ceph ALL = (root) NOPASSWD:ALL" | tee /etc/sudoers.d/ceph |
||
115 | shell> chmod 0440 /etc/sudoers.d/ceph |
||
116 | Disable 'requiretty' for the ceph user. |
||
117 | shell> sudo visudo |
||
118 | In the resulting file, locate the line containing |
||
119 | Defaults requiretty |
||
120 | and change it to read |
||
121 | Defaults:ceph !requiretty |
||
122 | Now, set up passphraseless SSH between the nodes: |
||
123 | Log in to the administration node as the ceph user. |
||
124 | Generate an SSH key for the administration node. |
||
125 | shell> ssh-keygen |
||
126 | 2 | Jessica Mack | |
127 | !image52.jpg! |
||
128 | |||
129 | 1 | Jessica Mack | Copy the generated public key to the ceph user account of all the nodes in the cluster. |
130 | shell> ssh-copy-id ceph@node1 |
||
131 | shell> ssh-copy-id ceph@node2 |
||
132 | shell> ssh-copy-id ceph@node3 |
||
133 | shell> ssh-copy-id ceph@node4 |
||
134 | shell> ssh-copy-id ceph@admin-node |
||
135 | image6.jpg |
||
136 | Test that the ceph user on the administration node can log in to any other node as ceph using SSH and without providing a password. |
||
137 | shell> ssh ceph@node1 |
||
138 | image7.jpg |
||
139 | Modify the administration node's SSH configuration file so that it can easily log in to each node as the ceph user. Create the /home/ceph/.ssh/config file with the following lines: |
||
140 | Host node1 |
||
141 | Hostname node1 |
||
142 | User ceph |
||
143 | Host node2 |
||
144 | Hostname node2 |
||
145 | User ceph |
||
146 | Host node3 |
||
147 | Hostname node3 |
||
148 | User ceph |
||
149 | Host node4 |
||
150 | Hostname node4 |
||
151 | User ceph |
||
152 | Host admin-node |
||
153 | Hostname admin-node |
||
154 | User ceph |
||
155 | Change the permissions of the /home/ceph/.ssh/config file. |
||
156 | shell> chmod 0400 ~/.ssh/config |
||
157 | Test that the ceph user on the administration node can log in to any other node using SSH and without providing a password or username. |
||
158 | shell> ssh node1 |
||
159 | image8.jpg |
||
160 | |||
161 | Finally, create a directory on the administration node to store cluster information, such as configuration files and keyrings. |
||
162 | shell> mkdir my-cluster |
||
163 | shell> cd my-cluster |
||
164 | You're now ready to begin preparing and activating the cluster! |
||
165 | |||
166 | h3. Step 4: Configure and Activate a Cluster Monitor |
||
167 | |||
168 | A Ceph storage cluster consists of two types of daemons: |
||
169 | Monitors maintain copies of the cluster map |
||
170 | Object Storage Daemons (OSD) store data as objects on storage nodes |
||
171 | Apart from this, other actors in a Ceph storage cluster include metadata servers and clients such as Ceph block devices, Ceph object gateways or Ceph filesystems. Read more about Ceph’s architecture. |
||
172 | All the commands in this and subsequent sections are to be run when logged in as the ceph user on the administration node, from the my-cluster/ directory. Ensure that you are directly logged in as ceph and are not using root with su - ceph. |
||
173 | A minimal system will have at least one monitor and two OSD daemons for data replication. |
||
174 | Begin by setting up a Ceph monitor on node1 with the Ceph deployment toolkit. |
||
175 | shell> ceph-deploy new node1 |
||
176 | This will define the name of the initial monitor node and create a default Ceph configuration file and monitor keyring in the current directory. |
||
177 | image9.jpg |
||
178 | |||
179 | Change the number of replicas in the Ceph configuration file at /home/ceph/my-cluster/ceph.conf from 3 to 2 so that Ceph can achieve a stable state with just two OSDs. Add the following line in the [global] section: |
||
180 | osd pool default size = 2 |
||
181 | osd pool default min size = 2 |
||
182 | In the same file, set the OSD journal size. A good general setting is 10 GB; however, since this is a simulation, you can use a smaller amount such as 4 GB. Add the following line in the [global] section: |
||
183 | osd journal size = 4000 |
||
184 | In the same file, set the default number of placement groups for a pool. Since we’ll have less than 5 OSDs, 128 placement groups per pool should suffice. Add the following line in the [global] section: |
||
185 | osd pool default pg num = 128 |
||
186 | Install Ceph on each node in the cluster, including the administration node. |
||
187 | shell> ceph-deploy install admin-node node1 node2 node3 node4 |
||
188 | The Ceph deployment toolkit will now go to work installing Ceph on each node. Here's an example of what you will see during the installation process. |
||
189 | image10.jpg |
||
190 | Create the Ceph monitor on node1 and gather the initial keys. |
||
191 | shell> ceph-deploy mon create-initial node1 |
||
192 | image11.jpg |
||
193 | |||
194 | h3. Step 5: Prepare and Activate OSDs |
||
195 | |||
196 | The next set is to prepare and activate Ceph OSDs. We'll need a minimum of 2 OSDs, and these should be set up on node2 and node3, as it's not recommended to mix monitors and OSD daemons on the same host. To begin, set up an OSD on node2 as follows: |
||
197 | Log into node2 as the ceph user. |
||
198 | shell> ssh node2 |
||
199 | Create a directory for the OSD daemon. |
||
200 | shell> sudo mkdir /var/local/osd |
||
201 | Log out of node2. Then, from the administrative node, prepare and activate the OSD. |
||
202 | shell> ceph-deploy osd prepare node2:/var/local/osd |
||
203 | image12.jpg |
||
204 | shell> ceph-deploy osd activate node2:/var/local/osd |
||
205 | image13.jpg |
||
206 | Repeat the above steps for node3. |
||
207 | At this point, the OSD daemons have been created and the storage cluster is ready. |
||
208 | |||
209 | h3. Step 6: Verify Cluster Health |
||
210 | |||
211 | Copy the configuration file and admin keyring from the administration node to all the nodes in the cluster. |
||
212 | shell> ceph-deploy admin admin-node node1 node2 node3 node4 |
||
213 | image14.jpg |
||
214 | Log in to each node as the ceph user and change the permissions of the admin keyring. |
||
215 | shell> ssh node1 |
||
216 | shell> sudo chmod +r /etc/ceph/ceph.client.admin.keyring |
||
217 | You should now be able to check cluster health from any node in the cluster with the ceph status command. Ideally, you want to see the status active + clean, as that indicates the cluster is operating normally. |
||
218 | shell> ceph status |
||
219 | image15.jpg |
||
220 | |||
221 | h3. Step 7: Test the Cluster |
||
222 | |||
223 | You can now perform a simple test to see the distributed Ceph storage cluster in action, by writing a file on one node and retrieving it on another: |
||
224 | Log in to node1 as the ceph user. |
||
225 | shell> ssh node1 |
||
226 | Create a new file with some dummy data. |
||
227 | shell> echo "Hello world" > /tmp/hello.txt |
||
228 | Data is stored in Ceph within storage pools, which are logical groups in which to organize your data. By default, a Ceph storage cluster has 3 pools - data, metadata and rbd - and it's also possible to create your own custom pools. In this case, copy the file to the data pool with the rados put command and assign it a name. |
||
229 | shell> rados put hello-object /tmp/hello.txt --pool data |
||
230 | To verify that the Ceph storage cluster stored the object: |
||
231 | Log in to node2 as the ceph user. |
||
232 | Check that the file exists in the cluster's data storage pool with the rados ls command. |
||
233 | shell> rados ls --pool data |
||
234 | Copy the file out of the storage cluster to a local directory with the rados get command and verify its contents |
||
235 | shell> rados get hello-object /tmp/hello.txt --pool data |
||
236 | shell> cat hello.txt |
||
237 | image16.jpg |
||
238 | |||
239 | h3. Step 8: Install the Ceph Object Gateway |
||
240 | |||
241 | Now that the cluster is operating, it’s time to do something with it. First, you must install and configure an Apache Web server with FastCGI on node4, as described below. |
||
242 | Log into node4 as the ceph user. |
||
243 | shell> ssh node4 |
||
244 | Install Apache and FastCGI from the Ceph repositories. To do this, you need to first install the yum priorities plugin, then add the repositories to your yum repository list. |
||
245 | shell> sudo yum install yum-plugin-priorities |
||
246 | Edit the /etc/yum/pluginconf.d/priorities.conf file and ensure it looks like this: |
||
247 | [main] |
||
248 | enabled = 1 |
||
249 | Create a file at /etc/yum.repos.d/ceph-apache.repo and fill it with the following content: |
||
250 | [apache2-ceph-noarch] |
||
251 | name=Apache noarch packages for Ceph |
||
252 | baseurl=http://gitbuilder.ceph.com/apache2-r...sic/ref/master |
||
253 | enabled=1 |
||
254 | priority=2 |
||
255 | gpgcheck=1 |
||
256 | type=rpm-md |
||
257 | gpgkey=https://ceph.com/git/?p=ceph.git;a=b.../autobuild.asc |
||
258 | [apache2-ceph-source] |
||
259 | name=Apache source packages for Ceph |
||
260 | baseurl=http://gitbuilder.ceph.com/apache2-r...sic/ref/master |
||
261 | enabled=0 |
||
262 | priority=2 |
||
263 | gpgcheck=1 |
||
264 | type=rpm-md |
||
265 | gpgkey=https://ceph.com/git/?p=ceph.git;a=b.../autobuild.asc |
||
266 | Create a file at /etc/yum.repos.d/ceph-fastcgi.repo and fill it with the following content: |
||
267 | [fastcgi-ceph-basearch] |
||
268 | name=FastCGI basearch packages for Ceph |
||
269 | baseurl=http://gitbuilder.ceph.com/mod_fastc...sic/ref/master |
||
270 | enabled=1 |
||
271 | priority=2 |
||
272 | gpgcheck=1 |
||
273 | type=rpm-md |
||
274 | gpgkey=https://ceph.com/git/?p=ceph.git;a=b.../autobuild.asc |
||
275 | [fastcgi-ceph-noarch] |
||
276 | name=FastCGI noarch packages for Ceph |
||
277 | baseurl=http://gitbuilder.ceph.com/mod_fastc...sic/ref/master |
||
278 | enabled=1 |
||
279 | priority=2 |
||
280 | gpgcheck=1 |
||
281 | type=rpm-md |
||
282 | gpgkey=https://ceph.com/git/?p=ceph.git;a=b.../autobuild.asc |
||
283 | [fastcgi-ceph-source] |
||
284 | name=FastCGI source packages for Ceph |
||
285 | baseurl=http://gitbuilder.ceph.com/mod_fastc...sic/ref/master |
||
286 | enabled=0 |
||
287 | priority=2 |
||
288 | gpgcheck=1 |
||
289 | type=rpm-md |
||
290 | gpgkey=https://ceph.com/git/?p=ceph.git;a=b.../autobuild.asc |
||
291 | Update the repository and install Apache and FastCGI. |
||
292 | shell> sudo yum update |
||
293 | shell> sudo yum install httpd mod_fastcgi |
||
294 | Edit the /etc/httpd/conf/httpd.conf file and modify the ServerName directive to reflect the server's host name. Uncomment the line if needed. |
||
295 | ServerName node4 |
||
296 | Review the files in the /etc/httpd/conf.modules.d/* directory to ensure that Apache's URL rewriting and FastCGI modules are enabled. You should find the following entries in the files: |
||
297 | LoadModule rewrite_module modules/mod_rewrite.so |
||
298 | LoadModule fastcgi_module modules/mod_fastcgi.so |
||
299 | In case these entries don't exist, add them to the end of the /etc/httpd/conf/httpd.conf file. |
||
300 | Restart Apache. |
||
301 | shell> sudo service httpd restart |
||
302 | Amazon S3 lets you refer to buckets using subdomains, such as http://mybucket.s3.amazonaws.com. You can also accomplish this with Ceph, but you must first install a local DNS server like dnsmasq and add support for wildcard subdomains. Follow these steps: |
||
303 | Log into node4 as the ceph user. |
||
304 | shell> ssh node4 |
||
305 | Install dnsmasq. |
||
306 | shell> yum install dnsmasq |
||
307 | Edit the dnsmasq configuration file at /etc/dnsmasq.conf and add the following line to the end of the file: |
||
308 | address=/.node4/192.168.1.29 |
||
309 | Save the file and restart dnsmasq. |
||
310 | shell> sudo service dnsmasq restart |
||
311 | If necessary, update the /etc/resolv.conf file on the client host so that it knows about the new DNS server. |
||
312 | nameserver 192.168.1.29 |
||
313 | You should now be able to successfully ping any subdomain of *.node4, such as mybucket.node4 or example.node4, as shown in the image below. |
||
314 | |||
315 | image17.png |
||
316 | |||
317 | TIP: If you're not able to configure wildcard subdomains, you can also simply decide a list of subdomains you wish to use and then add them as static entries to the client system's /etc/hosts file. Ensure that the entries resolve to the node4 virtual host. |
||
318 | The final step is to install radosgw on node4: |
||
319 | shell> ssh node4 |
||
320 | shell> sudo yum install ceph-radosgw |
||
321 | At this point, you have a Web server running with the Ceph object gateway and FastCGI support, and subdomains that resolve to the object gateway host. |
||
322 | |||
323 | h3. Step 9: Configure the Ceph Object Gateway |
||
324 | |||
325 | The next step is to configure the Ceph Object Gateway daemon. Follow these steps: |
||
326 | Log into the administration node as the ceph user. |
||
327 | shell> ssh admin-node |
||
328 | Create a keyring for the gateway. |
||
329 | shell> sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring |
||
330 | shell> sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring |
||
331 | Generate a user name and key to use when accessing the gateway. For this example, the user name is client.radosgw.gateway. |
||
332 | shell> sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key |
||
333 | Add read and write capabilities to the new key: |
||
334 | shell> sudo ceph-authtool -n client.radosgw.gateway --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring |
||
335 | Add the new key to the storage cluster and distribute it to the object gateway node. |
||
336 | shell> sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring |
||
337 | shell> sudo scp /etc/ceph/ceph.client.radosgw.keyring ceph@node4:/home/ceph |
||
338 | shell> ssh node4 |
||
339 | shell> sudo mv ceph.client.radosgw.keyring /etc/ceph/ceph.client.radosgw.keyring |
||
340 | shell> exit |
||
341 | This process should also have created a number of storage pools for the gateway. You can verify this by running the following command and verifying that the output includes various .rgw pools. |
||
342 | shell> rados lspools |
||
343 | image18.png |
||
344 | Change to your cluster configuration directory. |
||
345 | shell> cd ~/my-cluster |
||
346 | Edit the Ceph configuration file at ~/my-cluster/ceph/ceph.conf and add a new [client.radosgw.gateway] section to it, as below: |
||
347 | [client.radosgw.gateway] |
||
348 | host = node4 |
||
349 | keyring = /etc/ceph/ceph.client.radosgw.keyring |
||
350 | rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock |
||
351 | log file = /var/log/radosgw/client.radosgw.gateway.log |
||
352 | rgw dns name = node4 |
||
353 | rgw print continue = false |
||
354 | Transmit the new Ceph configuration file to all the other nodes in the cluster. |
||
355 | shell> ceph-deploy config push admin-node node1 node2 node3 node4 |
||
356 | Log into node4 as the ceph user. |
||
357 | shell> ssh node4 |
||
358 | Add a Ceph object gateway script, by creating a file at /var/www/html/s3gw.fcgi with the following content: |
||
359 | #!/bin/sh |
||
360 | exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway |
||
361 | Change the permissions of the script to make it executable. |
||
362 | shell> sudo chmod +x /var/www/html/s3gw.fcgi |
||
363 | Create a data directory for the radosgw daemon. |
||
364 | shell> sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.gateway |
||
365 | Add a gateway configuration file, by creating a file at /etc/httpd/conf.d/rgw.conf and filling it with the following content: |
||
366 | FastCgiExternalServer /var/www/html/s3gw.fcgi -socket /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock |
||
367 | <VirtualHost *:80> |
||
368 | ServerName node4 |
||
369 | ServerAlias *.node4 |
||
370 | ServerAdmin admin@localhost |
||
371 | DocumentRoot /var/www/html |
||
372 | RewriteEngine On |
||
373 | RewriteRule ^/(.*) /s3gw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] |
||
374 | <IfModule mod_fastcgi.c> |
||
375 | <Directory /var/www/html> |
||
376 | Options +ExecCGI |
||
377 | AllowOverride All |
||
378 | SetHandler fastcgi-script |
||
379 | Order allow,deny |
||
380 | Allow from all |
||
381 | AuthBasicAuthoritative Off |
||
382 | </Directory> |
||
383 | </IfModule> |
||
384 | AllowEncodedSlashes On |
||
385 | ErrorLog /var/log/httpd/error.log |
||
386 | CustomLog /var/log/httpd/access.log combined |
||
387 | ServerSignature Off |
||
388 | </VirtualHost> |
||
389 | <VirtualHost *:443> |
||
390 | ServerName node4 |
||
391 | ServerAlias *.node4 |
||
392 | ServerAdmin admin@localhost |
||
393 | DocumentRoot /var/www/html |
||
394 | RewriteEngine On |
||
395 | RewriteRule ^/(.*) /s3gw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] |
||
396 | <IfModule mod_fastcgi.c> |
||
397 | <Directory /var/www/html> |
||
398 | Options +ExecCGI |
||
399 | AllowOverride All |
||
400 | SetHandler fastcgi-script |
||
401 | Order allow,deny |
||
402 | Allow from all |
||
403 | AuthBasicAuthoritative Off |
||
404 | </Directory> |
||
405 | </IfModule> |
||
406 | AllowEncodedSlashes On |
||
407 | ErrorLog /var/log/httpd/error.log |
||
408 | CustomLog /var/log/httpd/access.log combined |
||
409 | ServerSignature Off |
||
410 | SSLEngine on |
||
411 | SSLCertificateFile /etc/apache2/ssl/apache.crt |
||
412 | SSLCertificateKeyFile /etc/apache2/ssl/apache.key |
||
413 | SetEnv SERVER_PORT_SECURE 443 |
||
414 | </VirtualHost> |
||
415 | Edit the /etc/httpd/conf.d/fastcgi.conf file and ensure that the line referencing the FastCgiWrapper looks like this: |
||
416 | FastCgiWrapper off |
||
417 | Restart the Apache server, followed by the radosgw daemon. |
||
418 | shell> sudo service httpd restart |
||
419 | shell> sudo /etc/init.d/ceph-radosgw restart |
||
420 | You can quickly test that the object gateway is running by sending an HTTP GET request to the Web server, as shown below: |
||
421 | image19.png |
||
422 | At this point, your Ceph object gateway is running and you can begin using it. |
||
423 | |||
424 | h3. Step 10: Start Working with Buckets and Objects |
||
425 | |||
426 | Before you can begin using the Ceph object gateway, you must create a user account. |
||
427 | Log in to node4 as the ceph user. |
||
428 | shell> ssh admin-node |
||
429 | Create a new user account using the radosgw-admin command. In this example, the user is named 'john'. |
||
430 | shell> radosgw-admin user create --uid=john --display-name="Example User" |
||
431 | Here's an example of what you should see. Note the access key and secret key in the output, as you will need this to access the object gateway from another client. |
||
432 | image20.png |
||
433 | You can also verify that the user was created with the following command: |
||
434 | shell> radosgw-admin user info --uid=john |
||
435 | While you can interact with the object gateway directly over HTTP, by sending authenticated GET, PUT and DELETE requests to the gateway endpoints, an easier way is with Amazon's AWS SDK. This SDK includes classes and constructs to help you work with buckets and objects in Amazon S3. Since the Ceph object gateway is S3-compatible, you can use the same SDK to interact with it as well. |
||
436 | The AWS SDK is available for multiple programming languages. In the examples that follow, I'll use the AWS SDK for PHP, but you will find code examples for other languages as well on the AWS developer website. |
||
437 | Log in to node4 (which will now also double as the client node) as the root user and install PHP and related tools. |
||
438 | shell> sudo yum install php curl php-curl |
||
439 | Create a working directory for your PHP files. Download Composer, the PHP dependency manager, into this directory. |
||
440 | shell> cd /tmp |
||
441 | shell> mkdir ceph |
||
442 | shell> cd ceph |
||
443 | shell> curl -sS https://getcomposer.org/installer | php |
||
444 | Create a composer.json file in the working directory and fill it with the following content: |
||
445 | { |
||
446 | "require": { |
||
447 | "aws/aws-sdk-php": "2.*" |
||
448 | } |
||
449 | } |
||
450 | Download the AWS SDK for PHP and related dependencies using Composer: |
||
451 | shell> cd /tmp/ceph |
||
452 | shell> php composer.phar install |
||
453 | You can now begin interacting with your object gateway using PHP. For example, here's a simple PHP script to create a new bucket: |
||
454 | <?php |
||
455 | // create-bucket.php |
||
456 | // autoload files |
||
457 | require 'vendor/autoload.php'; |
||
458 | use Aws\S3\S3Client; |
||
459 | // instantiate S3 client |
||
460 | $s3 = S3Client::factory(array( |
||
461 | 'key' => 'YOUR_ACCESS_KEY', |
||
462 | 'secret' => 'YOUR_SECRET_KEY', |
||
463 | 'endpoint' => 'http://node4' |
||
464 | )); |
||
465 | // create bucket |
||
466 | try { |
||
467 | $s3->createBucket(array('Bucket' => 'mybucket')); |
||
468 | echo "Bucket created \n"; |
||
469 | } catch (Aws\S3\Exception\S3Exception $e) { |
||
470 | echo "Request failed: $e"; |
||
471 | } |
||
472 | This script begins by initializing the Composer auto-loader and an instance of the S3Client object. The object is provided with the access key and secret for the user created earlier, and a custom endpoint points to the object gateway Web server. |
||
473 | The S3Client object provides a number of methods to create and manage buckets and objects. One of these is the createBucket() method, which accepts a bucket name and generates the necessary PUT request to create a new bucket in the object gateway. |
||
474 | You can run this script at the console as follows: |
||
475 | shell> php create-bucket.php |
||
476 | Here's an example of what the output might look like: |
||
477 | image21.png |
||
478 | You can also create a bucket and then add a file to it as an object, using the client object's upload() method. Here's an example: |
||
479 | <?php |
||
480 | // create-bucket-object.php |
||
481 | // autoload files |
||
482 | require 'vendor/autoload.php'; |
||
483 | use Aws\S3\S3Client; |
||
484 | // instantiate S3 client |
||
485 | $s3 = S3Client::factory(array( |
||
486 | 'key' => 'YOUR_ACCESS_KEY', |
||
487 | 'secret' => 'YOUR_SECRET_KEY', |
||
488 | 'endpoint' => 'http://node4' |
||
489 | )); |
||
490 | // create bucket and upload file to it |
||
491 | try { |
||
492 | $s3->createBucket(array('Bucket' => 'myotherbucket')); |
||
493 | $s3->upload('myotherbucket', 'test.tgz', file_get_contents('/tmp/test.tgz'), 'public-read'); |
||
494 | echo 'Bucket and object created'; |
||
495 | } catch (Aws\S3\Exception\S3Exception $e) { |
||
496 | echo "Request failed: $e"; |
||
497 | } |
||
498 | Of course, you can also list all the buckets and objects available to the authenticated user with the listBuckets() and listObjects() methods: |
||
499 | <?php |
||
500 | // list-bucket-contents.php |
||
501 | // autoload files |
||
502 | require 'vendor/autoload.php'; |
||
503 | use Aws\S3\S3Client; |
||
504 | // instantiate S3 client |
||
505 | $s3 = S3Client::factory(array( |
||
506 | 'key' => 'YOUR_ACCESS_KEY', |
||
507 | 'secret' => 'YOUR_SECRET_KEY', |
||
508 | 'endpoint' => 'http://node4' |
||
509 | )); |
||
510 | // create bucket and upload file to it |
||
511 | try { |
||
512 | $bucketsColl = $s3->listBuckets(); |
||
513 | foreach ($bucketsColl['Buckets'] as $bucket) { |
||
514 | echo $bucket['Name'] . "\n"; |
||
515 | $objColl = $s3->listObjects(array('Bucket' => $bucket['Name'])); |
||
516 | if ($objColl['Contents']) { |
||
517 | foreach ($objColl['Contents'] as $obj) { |
||
518 | echo '- ' . $obj['Key'] . "\n"; |
||
519 | } |
||
520 | } |
||
521 | } |
||
522 | } catch (Aws\S3\Exception\S3Exception $e) { |
||
523 | echo "Request failed: $e"; |
||
524 | } |
||
525 | Here's an example of what the output might look like: |
||
526 | image22.png |
||
527 | Of course, you can do a lot more with the AWS SDK for PHP. Refer to the reference documentation for a complete list of methods and example code. |
||
528 | |||
529 | h3. Conclusion |
||
530 | |||
531 | As this tutorial has illustrated, Ceph makes it easy to set up a standards-compliant object gateway for your applications or users, with all the benefits of a resilient, infinitely scalable underlying storage cluster. |
||
532 | The simple object gateway you created here with VirtualBox is just the tip of the iceberg: you can transition your object gateway to the cloud and run it in federated mode across regions and zones for even greater flexibility, and because the Ceph object gateway is also Swift-compliant, you can maximize compatibility for OpenStack users without any changes to your existing infrastructure. And of course, you can also use the underlying object storage cluster for fault-tolerant Ceph block devices or the POSIX-compliant CephFS filesystem. |
||
533 | The bottom line: Ceph's unique architecture gives you improved performance and flexibility without any loss in reliability and security. And it's open source, so you can experiment with it, improve it and use it without worrying about vendor lock-in. You can't get any better than that! |
||
534 | |||
535 | h3. Read More |
||
536 | |||
537 | "Introduction to Ceph":http://ceph.com/docs/master/start/intro/ |
||
538 | "Ceph Architecture":http://ceph.com/docs/master/architecture/ |
||
539 | "Getting Started With Ceph":http://www.inktank.com/resource/getting-started-with-ceph-miroslav-klivansky/ |
||
540 | "Introduction to Ceph & OpenStack":http://www.inktank.com/resource/introduction-to-ceph-openstack-miroslav-klivansky/ |
||
541 | "Managing A Distributed Storage System At Scale":http://www.inktank.com/resource/managing-a-distributed-storage-system-at-scale-sage-weil/ |
||
542 | "Scaling Storage With Ceph":http://www.inktank.com/resource/scaling-storage-with-ceph-ross-turk/ |
||
543 | "Ceph API Documentation":http://ceph.com/docs/master/api/ |