Project

General

Profile

Tuning for All Flash Deployments » History » Version 17

Patrick McGarry, 01/14/2017 09:00 PM

1 1 Patrick McGarry
h1. Tuning for All Flash Deployments
2 1 Patrick McGarry
3 1 Patrick McGarry
Ceph Tuning and Best Practices for All Flash Intel® Xeon® Servers
4 1 Patrick McGarry
Last updated:  January 2017
5 1 Patrick McGarry
6 1 Patrick McGarry
h2. Table of Contents
7 9 Patrick McGarry
8 14 Patrick McGarry
# +*[[Tuning_for_All_Flash_Deployments#Introduction|Introduction]]*+
9 14 Patrick McGarry
# +*[[Tuning_for_All_Flash_Deployments#Ceph-Storage-Hardware-Guidelines|Ceph Storage Hardware Guidelines]]*+
10 14 Patrick McGarry
# +*[[Tuning_for_All_Flash_Deployments#Intel-Tuning-and-Optimization-Recommendations-for-Ceph|Intel Tuning and Optimization Recommendations for Ceph]]*+
11 14 Patrick McGarry
## +[[Tuning_for_All_Flash_Deployments#Server-Tuning|Server Tuning]]+
12 14 Patrick McGarry
### +[[Tuning_for_All_Flash_Deployments#Ceph-Client-Configuration|Ceph Client Configuration]]+
13 1 Patrick McGarry
### +[[Tuning_for_All_Flash_Deployments#Ceph-Storage-Node-NUMA-Tuning|Ceph Storage Node NUMA Tuning]]+
14 14 Patrick McGarry
## +[[Tuning_for_All_Flash_Deployments#Memory-Tuning|Memory Tuning]]+
15 14 Patrick McGarry
## +[[Tuning_for_All_Flash_Deployments#NVMe-SSD-partitioning|NVMe SSD partitioning]]+
16 15 Patrick McGarry
## +[[Tuning_for_All_Flash_Deployments#OS-Tuning|OS Tuning]]+ (must be done on all Ceph nodes)
17 14 Patrick McGarry
### +[[Tuning_for_All_Flash_Deployments#Kernel-Tuning|Kernel Tuning]]+
18 14 Patrick McGarry
### +[[Tuning_for_All_Flash_Deployments#Filesystem-considerations|Filesystem considerations]]+
19 14 Patrick McGarry
### +[[Tuning_for_All_Flash_Deployments#Disk-read-ahead|Disk read ahead]]+
20 14 Patrick McGarry
### +[[Tuning_for_All_Flash_Deployments#OSD-RADOS|OSD: RADOS]]+
21 14 Patrick McGarry
## +[[Tuning_for_All_Flash_Deployments#RBD-Tuning|RBD Tuning]]+
22 14 Patrick McGarry
## +[[Tuning_for_All_Flash_Deployments#RGW-Rados-Gateway-Tuning|RGW: Rados Gateway Tuning]]+
23 14 Patrick McGarry
## +[[Tuning_for_All_Flash_Deployments#Erasure-Coding-Tuning|Erasure Coding Tuning]]+
24 14 Patrick McGarry
# +*[[Tuning_for_All_Flash_Deployments#Appendix|Appendix]]*+
25 17 Patrick McGarry
# +*[[Tuning_for_All_Flash_Deployments#Sample-Cephconf|Sample Ceph.conf]]*+
26 14 Patrick McGarry
# +*[[Tuning_for_All_Flash_Deployments#Sample-sysctl-conf|Sample sysctl.conf]]*+
27 14 Patrick McGarry
# +*[[Tuning_for_All_Flash_Deployments#All-NVMe-Ceph-Cluster-Tuning-for-MySQL-workload|All-NVMe Ceph Cluster Tuning for MySQL workload]]*+
28 14 Patrick McGarry
## +[[Tuning_for_All_Flash_Deployments#Ceph-conf|Ceph.conf]]+
29 14 Patrick McGarry
## +[[Tuning_for_All_Flash_Deployments#CBT-YAML|CBT YAML]]+
30 14 Patrick McGarry
## +[[Tuning_for_All_Flash_Deployments#MySQL-configuration-file|MySQL configuration file]] (my.cnf)+
31 14 Patrick McGarry
# +*[[Tuning_for_All_Flash_Deployments#Sample-Ceph-Vendor-Solutions|Sample Ceph Vendor Solutions]]*+
32 1 Patrick McGarry
33 1 Patrick McGarry
34 1 Patrick McGarry
35 1 Patrick McGarry
36 1 Patrick McGarry
h3. Introduction 
37 1 Patrick McGarry
38 8 Patrick McGarry
Ceph is a scalable, open source, software-defined storage offering that runs on commodity hardware. Ceph has been developed from the ground up to deliver object, block, and file system storage in a single software platform that is self-managing, self-healing and has no single point of failure. Because of its highly scalable, software defined storage architecture, can be a powerful storage solution to consider. 
39 8 Patrick McGarry
This document covers Ceph tuning guidelines specifically for all flash deployments based on extensive testing by Intel with a variety of system, operating system and Ceph optimizations to achieve highest possible performance for servers with Intel® Xeon® processors and Intel® Solid State Drive Data Center (Intel® SSD DC) Series. Details of OEM system SKUs and Ceph reference architectures for targeted use-cases can be found on ceph.com web-site. 
40 8 Patrick McGarry
 
41 1 Patrick McGarry
h3. Ceph Storage Hardware Guidelines  
42 8 Patrick McGarry
43 1 Patrick McGarry
* *Standard* configuration is ideally suited for throughput oriented workloads (e.g., analytics,  DVR). Intel® SSD Data Center P3700 series is recommended to achieve best possible performance while balancing the cost.
44 8 Patrick McGarry
45 8 Patrick McGarry
| CPU | Intel® Xeon® CPU E5-2650v4 or higher |
46 8 Patrick McGarry
| Memory | Minimum of 64 GB|
47 1 Patrick McGarry
| NIC | 10GbE |
48 10 Patrick McGarry
| Disks | 1x 1.6TB P3700 + 12 x 4TB HDDs (1:12 ratio) / P3700 as Journal and caching |
49 8 Patrick McGarry
| Caching software | Intel Cache Acceleration Software for read caching, option: Intel® Rapid Storage Technology enterprise/MD4.3 |
50 8 Patrick McGarry
51 1 Patrick McGarry
* *TCO optimized* configuration provides best possible performance for performance centric workloads (e.g., database) while achieving the TCO with a mix of SATA SSDs and NVMe SSDs. 
52 8 Patrick McGarry
53 8 Patrick McGarry
| CPU | Intel® Xeon® CPU E5-2690v4 or higher |
54 8 Patrick McGarry
| Memory | 128 GB or higher |
55 8 Patrick McGarry
| NIC | Dual 10GbE 
56 8 Patrick McGarry
| Disks | 1x 800GB P3700 + 4x S3510 1.6TB |
57 8 Patrick McGarry
58 8 Patrick McGarry
* *IOPS optimized* configuration provides best performance for workloads that demand low latency using all NVMe SSD configuration. 
59 8 Patrick McGarry
60 8 Patrick McGarry
| CPU | Intel® Xeon® CPU E5-2699v4 |
61 8 Patrick McGarry
| Memory | 128 GB or higher |
62 8 Patrick McGarry
| NIC | 1x 40GbE, 4x 10GbE |
63 8 Patrick McGarry
| Disks | 4 x P3700 2TB |
64 8 Patrick McGarry
65 12 Patrick McGarry
h3. Intel Tuning and Optimization Recommendations for Ceph
66 11 Patrick McGarry
67 12 Patrick McGarry
h3. Server Tuning
68 16 Patrick McGarry
69 16 Patrick McGarry
h3. Ceph Client Configuration
70 10 Patrick McGarry
71 1 Patrick McGarry
In a balanced system configuration both client and storage node configuration need to be optimized to get the best possible cluster performance. Care needs to be taken to ensure Ceph client node server has enough CPU bandwidth to achieve optimum performance. Below graph shows the end to end performance for different client CPU configurations for block workload.
72 1 Patrick McGarry
73 8 Patrick McGarry
!1-cpu-cores-client.png!
74 1 Patrick McGarry
Figure 1: Client CPU cores and Ceph cluster impact
75 8 Patrick McGarry
76 8 Patrick McGarry
h3. Ceph Storage Node NUMA Tuning
77 8 Patrick McGarry
78 1 Patrick McGarry
In order to avoid latency, it is important to minimize inter-socket communication between NUMA nodes to service client IO as fast as possible and avoid latency penalty.  Based on extensive set of experiments conducted in Intel, it is recommended to pin Ceph OSD processes on the same CPU socket that has NVMe SSDs, HBAs and NIC devices attached. 
79 8 Patrick McGarry
80 8 Patrick McGarry
!2-numa-mode-config.png! 
81 1 Patrick McGarry
Figure 2: NUMA node configuration and OSD assignment
82 1 Patrick McGarry
83 8 Patrick McGarry
> *_Ceph startup scripts need change with setaffinity=" numactl --membind=0 --cpunodebind=0 "_
84 8 Patrick McGarry
85 1 Patrick McGarry
Below performance data shows best possible cluster throughput and lower latency when Ceph OSDs are partitioned by CPU socket to manage media connected to local CPU socket and network IO not going through QPI link. 
86 1 Patrick McGarry
87 1 Patrick McGarry
!3-numa-node-perf-vs-default-sys-config.png!
88 8 Patrick McGarry
!3.1-table.png!
89 1 Patrick McGarry
Figure 3: NUMA node performance compared to default system configuration
90 8 Patrick McGarry
91 1 Patrick McGarry
h3. Memory Tuning
92 1 Patrick McGarry
93 1 Patrick McGarry
Ceph default packages use tcmalloc. For flash optimized configurations, we found jemalloc providing best possible performance without performance degradation over time. Ceph supports jemalloc for the hammer release and later releases but you need to build with jemalloc option enabled.
94 1 Patrick McGarry
95 1 Patrick McGarry
Below graph in figure 4 shows how thread cache size impacts throughput. By tuning thread cache size, performance is comparable between TCMalloc and JEMalloc. However as shown in Figure 5 and Figure 6, TCMalloc performance degrades over time unlike JEMalloc. 
96 1 Patrick McGarry
97 1 Patrick McGarry
!4-thread-cache-size-impact-over-perf.png!
98 1 Patrick McGarry
Figure 4: Thread cache size impact over performance
99 1 Patrick McGarry
100 1 Patrick McGarry
!5.0-tcmalloc-over-time.png!
101 1 Patrick McGarry
!5.1-tcmalloc-over-time.png!
102 1 Patrick McGarry
Figure 5: TCMalloc performance in a running cluster over time
103 8 Patrick McGarry
104 1 Patrick McGarry
105 1 Patrick McGarry
!6.0-jemalloc-over-time.png!
106 8 Patrick McGarry
!6.1-jemalloc-over-time.png!
107 1 Patrick McGarry
Figure 6: JEMalloc performance in a running cluster over time
108 1 Patrick McGarry
109 8 Patrick McGarry
h3. NVMe SSD partitioning
110 14 Patrick McGarry
111 1 Patrick McGarry
It is not possible to take advantage of NVMe SSD bandwidth with single OSD.  4 is the optimum number of partitions per SSD drive that gives best possible performance. 
112 1 Patrick McGarry
113 14 Patrick McGarry
!7.0-ceph-osd-latency-with-different-ssd-partitions.png!
114 8 Patrick McGarry
Figure 7: Ceph OSD latency with different SSD partitions
115 8 Patrick McGarry
116 14 Patrick McGarry
!8-cpu-utilization-with-different-num-of-ssd-partitions.png!
117 1 Patrick McGarry
Figure 8: CPU Utilization with different #of SSD partitions 
118 8 Patrick McGarry
119 14 Patrick McGarry
120 14 Patrick McGarry
h3. OS Tuning
121 14 Patrick McGarry
122 1 Patrick McGarry
*(must be done on all Ceph nodes)*
123 14 Patrick McGarry
124 14 Patrick McGarry
h3. Kernel Tuning
125 14 Patrick McGarry
126 15 Patrick McGarry
1. Modify system control in /etc/sysctl.conf
127 14 Patrick McGarry
<pre>
128 1 Patrick McGarry
# Kernel sysctl configuration file for Red Hat Linux
129 1 Patrick McGarry
#
130 1 Patrick McGarry
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
131 1 Patrick McGarry
# sysctl.conf(5) for more details.
132 1 Patrick McGarry
133 1 Patrick McGarry
# Controls IP packet forwarding
134 1 Patrick McGarry
net.ipv4.ip_forward = 0
135 1 Patrick McGarry
136 1 Patrick McGarry
# Controls source route verification
137 1 Patrick McGarry
net.ipv4.conf.default.rp_filter = 1
138 1 Patrick McGarry
139 1 Patrick McGarry
# Do not accept source routing
140 1 Patrick McGarry
net.ipv4.conf.default.accept_source_route = 0
141 1 Patrick McGarry
142 1 Patrick McGarry
# Controls the System Request debugging functionality of the kernel
143 1 Patrick McGarry
kernel.sysrq = 0
144 1 Patrick McGarry
145 1 Patrick McGarry
# Controls whether core dumps will append the PID to the core filename.
146 1 Patrick McGarry
# Useful for debugging multi-threaded applications.
147 1 Patrick McGarry
kernel.core_uses_pid = 1
148 1 Patrick McGarry
149 1 Patrick McGarry
# disable TIME_WAIT.. wait ..
150 1 Patrick McGarry
net.ipv4.tcp_tw_recycle = 1
151 1 Patrick McGarry
net.ipv4.tcp_tw_reuse = 1
152 1 Patrick McGarry
153 1 Patrick McGarry
# Controls the use of TCP syncookies
154 1 Patrick McGarry
net.ipv4.tcp_syncookies = 0
155 1 Patrick McGarry
156 1 Patrick McGarry
# double amount of allowed conntrack
157 1 Patrick McGarry
net.netfilter.nf_conntrack_max = 2621440
158 1 Patrick McGarry
net.netfilter.nf_conntrack_tcp_timeout_established = 1800
159 1 Patrick McGarry
160 1 Patrick McGarry
# Disable netfilter on bridges.
161 1 Patrick McGarry
net.bridge.bridge-nf-call-ip6tables = 0
162 1 Patrick McGarry
net.bridge.bridge-nf-call-iptables = 0
163 1 Patrick McGarry
net.bridge.bridge-nf-call-arptables = 0
164 1 Patrick McGarry
165 1 Patrick McGarry
# Controls the maximum size of a message, in bytes
166 1 Patrick McGarry
kernel.msgmnb = 65536
167 1 Patrick McGarry
168 1 Patrick McGarry
# Controls the default maxmimum size of a mesage queue
169 1 Patrick McGarry
kernel.msgmax = 65536
170 1 Patrick McGarry
171 1 Patrick McGarry
# Controls the maximum shared segment size, in bytes
172 1 Patrick McGarry
kernel.shmmax = 68719476736
173 1 Patrick McGarry
174 1 Patrick McGarry
# Controls the maximum number of shared memory segments, in pages
175 1 Patrick McGarry
kernel.shmall = 4294967296
176 1 Patrick McGarry
</pre>
177 14 Patrick McGarry
178 15 Patrick McGarry
2. IP jumbo frames
179 1 Patrick McGarry
180 15 Patrick McGarry
    If your switch supports jumbo frames, then the larger MTU size is helpful. Our tests showed 9000 MTU improves Sequential Read/Write performance.
181 14 Patrick McGarry
182 15 Patrick McGarry
3. Set the Linux disk scheduler to cfq
183 1 Patrick McGarry
184 1 Patrick McGarry
h3. Filesystem considerations
185 1 Patrick McGarry
186 1 Patrick McGarry
Ceph is designed to be mostly filesystem agnostic–the only requirement being that the filesystem supports extended attributes (xattrs). Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying file system for: a) Internal object state b) Snapshot metadata c) RGW Access control Lists etc. Currently XFS is the recommended file system. We recommend using big inode size (default inode size is 256 bytes) when creating the file system:
187 1 Patrick McGarry
188 15 Patrick McGarry
<pre>mkfs.xfs –i size=2048 /dev/sda1</pre>
189 15 Patrick McGarry
190 14 Patrick McGarry
Setting the inode size is important, as XFS stores xattr data in the inode. If the metadata is too large to fit in the inode, a new extent is created, which can cause quite a performance problem. Upping the inode size to 2048 bytes provides enough room to write the default metadata, plus a little headroom.
191 1 Patrick McGarry
The following example mount options are recommended when using XFS:
192 15 Patrick McGarry
193 15 Patrick McGarry
<pre>mount -t xfs -o noatime,nodiratime,nobarrier,logbufs=8 /dev/sda1 /var/lib/Ceph/osd/Ceph-0</pre>
194 15 Patrick McGarry
195 1 Patrick McGarry
The following are specific recommendations for Intel SSD and Ceph. 
196 15 Patrick McGarry
197 15 Patrick McGarry
<pre>mkfs.xfs -f -K -i size=2048 -s size=4096 /dev/md0
198 15 Patrick McGarry
/bin/mount -o noatime,nodiratime,nobarrier /dev/md0 /data/mysql</pre>
199 15 Patrick McGarry
200 15 Patrick McGarry
h3. Disk read ahead
201 15 Patrick McGarry
202 1 Patrick McGarry
Read_ahead is the file prefetching technology used in the Linux operating system. It is a system call that loads a file's contents into the page cache. When a file is subsequently accessed, its contents are read from physical memory rather than from disk, which is much faster.
203 1 Patrick McGarry
204 15 Patrick McGarry
<pre>echo 2048 > /sys/block/${disk}/queue/read_ahead_kb  (default 128)</pre>
205 15 Patrick McGarry
206 15 Patrick McGarry
|. Per disk performance |. 128 |. 512 |. % |
207 15 Patrick McGarry
| Sequential Read(MB/s) | 1232 MB/s | 3251 MB/s | +163% |
208 15 Patrick McGarry
209 1 Patrick McGarry
* 6 nodes Ceph cluster, each have 20 OSD (750 GB * 7200 RPM. 2.5’’ HDD)
210 1 Patrick McGarry
211 15 Patrick McGarry
h3. OSD: RADOS
212 15 Patrick McGarry
213 1 Patrick McGarry
Tuning have significant performance impact of Ceph storage system, there are hundreds of tuning knobs for swift. We will introduce some of the most important tuning settings.
214 1 Patrick McGarry
215 15 Patrick McGarry
# Large PG/PGP number (since Cuttlefish)
216 15 Patrick McGarry
We find using large PG number per OSD (>200) will improve the performance. Also this will ease the data distribution unbalance issue
217 15 Patrick McGarry
<pre>(default to 8)
218 15 Patrick McGarry
ceph osd pool create testpool 8192 8192</pre>
219 15 Patrick McGarry
# omap data on separate disks (since Giant)
220 1 Patrick McGarry
Mounting omap directory to some separate SSD will improve the random write performance. In our testing we saw a ~20% performance improvement.
221 15 Patrick McGarry
# objecter_inflight_ops/objecter_inflight_op_bytes (since Cuttlefish)
222 1 Patrick McGarry
objecter_inflight_ops/objecter_inflight_op_bytes throttles tell objecter to throttle outgoing ops according its budget, objecter is responsible for send requests to OSD. By default tweak this parameter to 10x 
223 15 Patrick McGarry
<pre>(default to 1024/1024*1024*100)
224 1 Patrick McGarry
objecter_inflight_ops = 10240
225 15 Patrick McGarry
objecter_inflight_op_bytes = 1048576000</pre>
226 15 Patrick McGarry
# ms_dispatch_throttle_bytes (since Cuttlefish)
227 1 Patrick McGarry
ms_dispatch_throttle_bytes throttle is to throttle dispatch message size for simple messenger, by default tweak this parameter to 10x. 
228 15 Patrick McGarry
<pre>ms_dispatch_throttle_bytes = 1048576000</pre>
229 15 Patrick McGarry
# journal_queue_max_bytes/journal_queue_max_ops (since Cuttlefish)
230 1 Patrick McGarry
journal_queue_max_bytes/journal_queue_max_op throttles are to throttle inflight ops for journal, 
231 1 Patrick McGarry
If journal does not get enough budget for current op, it will block osd op thread, by default tweak this parameter to 10x.
232 15 Patrick McGarry
<pre>journal_queueu_max_ops = 3000
233 15 Patrick McGarry
journal_queue_max_bytes = 1048576000</pre>
234 15 Patrick McGarry
# filestore_queue_max_ops/filestore_queue_max_bytes (since Cuttlefish)
235 1 Patrick McGarry
filestore_queue_max_ops/filestore_queue_max_bytes throttle are used to throttle inflight ops for filestore, these throttles are checked before sending ops to journal, so if filestore does not get enough budget for current op, osd op thread will be blocked, by default tweak this parameter to 10x.
236 15 Patrick McGarry
<pre>filestore_queue_max_ops=5000
237 15 Patrick McGarry
filestore_queue_max_bytes = 1048576000</pre>
238 15 Patrick McGarry
# filestore_op_threads controls the number of filesystem operation threads that execute in parallel
239 1 Patrick McGarry
If the storage backend is fast enough and has enough queues to support parallel operations, it’s recommended to increase this parameter, given there is enough CPU head room.
240 15 Patrick McGarry
<pre>filestore_op_threads=6</pre>
241 15 Patrick McGarry
# journal_max_write_entries/journal_max_write_bytes (since Cuttlefish)
242 1 Patrick McGarry
journal_max_write_entries/journal_max_write_bytes throttle are used to throttle ops or bytes for every journal write, tweaking these two parameters maybe helpful for small write, by default tweak these two parameters to 10x
243 15 Patrick McGarry
<pre>journal_max_write_entries = 5000
244 15 Patrick McGarry
journal_max_write_bytes = 1048576000</pre>
245 15 Patrick McGarry
# osd_op_num_threads_per_shard/osd_op_num_shards (since Firefly)
246 1 Patrick McGarry
osd_op_num_shards set number of queues to cache requests , osd_op_num_threads_per_shard is    threads number for each queue,  adjusting these two parameters depends on cluster.
247 1 Patrick McGarry
After several performance tests with different settings, we concluded that default parameters provide best performance.
248 15 Patrick McGarry
# filestore_max_sync_interval (since Cuttlefish)
249 1 Patrick McGarry
filestore_max_sync_interval control the interval that sync thread flush data from memory to disk, by default filestore write data to memory and sync thread is responsible for flushing data to disk, then journal entries can be trimmed. Note that large filestore_max_sync_interval can cause performance spike. By default tweak this parameter to 10 seconds
250 15 Patrick McGarry
<pre>filestore_max_sync_interval = 10</pre>
251 15 Patrick McGarry
# ms_crc_data/ms_crc_header (since Cuttlefish)
252 1 Patrick McGarry
Disable crc computation for simple messenger, this can reduce CPU utilization
253 15 Patrick McGarry
# filestore_fd_cache_shards/filestore_fd_cache_size (since Firefly)
254 1 Patrick McGarry
filestore cache is map from objectname to fd, filestore_fd_cache_shards set number of LRU Cache,  filestore_fd_cache_size is cache size, tweak these two parameter maybe reduce lookup time of fd
255 15 Patrick McGarry
# Set debug level to 0 (since Cuttlefish)
256 1 Patrick McGarry
For an all-SSD Ceph cluster, set debug level for sub system to 0 will improve the performance.  
257 15 Patrick McGarry
<pre>debug_lockdep = 0/0
258 1 Patrick McGarry
debug_context = 0/0
259 1 Patrick McGarry
debug_crush = 0/0
260 1 Patrick McGarry
debug_buffer = 0/0
261 1 Patrick McGarry
debug_timer = 0/0
262 1 Patrick McGarry
debug_filer = 0/0
263 1 Patrick McGarry
debug_objecter = 0/0
264 1 Patrick McGarry
debug_rados = 0/0
265 1 Patrick McGarry
debug_rbd = 0/0
266 1 Patrick McGarry
debug_journaler = 0/0
267 1 Patrick McGarry
debug_objectcatcher = 0/0
268 1 Patrick McGarry
debug_client = 0/0
269 1 Patrick McGarry
debug_osd = 0/0
270 1 Patrick McGarry
debug_optracker = 0/0
271 1 Patrick McGarry
debug_objclass = 0/0
272 1 Patrick McGarry
debug_filestore = 0/0
273 1 Patrick McGarry
debug_journal = 0/0
274 1 Patrick McGarry
debug_ms = 0/0
275 1 Patrick McGarry
debug_monc = 0/0
276 1 Patrick McGarry
debug_tp = 0/0
277 1 Patrick McGarry
debug_auth = 0/0
278 1 Patrick McGarry
debug_finisher = 0/0
279 1 Patrick McGarry
debug_heartbeatmap = 0/0
280 1 Patrick McGarry
debug_perfcounter = 0/0
281 1 Patrick McGarry
debug_asok = 0/0
282 1 Patrick McGarry
debug_throttle = 0/0
283 1 Patrick McGarry
debug_mon = 0/0
284 1 Patrick McGarry
debug_paxos = 0/0
285 15 Patrick McGarry
debug_rgw = 0/0</pre>
286 1 Patrick McGarry
287 1 Patrick McGarry
288 15 Patrick McGarry
h3. RBD Tuning
289 15 Patrick McGarry
290 1 Patrick McGarry
To help achieve low latency on their RBD layer, we suggest the following, in addition to the CERN tuning referenced in ceph.com. 
291 15 Patrick McGarry
# echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor /dev/null 
292 15 Patrick McGarry
# start each ceph-osd in dedicated cgroup with dedicated cpu cores (which should be free from any other load, even the kernel one like network interrupts)
293 15 Patrick McGarry
# increase “filestore_omap_header_cache_size” • “filestore_fd_cache_size” , for better caching (16MB for each 500GB of storage)
294 1 Patrick McGarry
For disk entry in libvirt  put address to all three ceph monitors.
295 1 Patrick McGarry
296 15 Patrick McGarry
h3. RGW: Rados Gateway Tuning
297 15 Patrick McGarry
298 15 Patrick McGarry
# Disable usage/access log (since Cuttlefish)
299 15 Patrick McGarry
<pre>rgw enable ops log = false
300 1 Patrick McGarry
rgw enable usage log = false
301 15 Patrick McGarry
log file = /dev/null</pre>
302 1 Patrick McGarry
We find disabling usage/access log improves the performance.
303 15 Patrick McGarry
# Using large cache size (since Cuttlefish)
304 15 Patrick McGarry
<pre>rgw cache enabled = true
305 15 Patrick McGarry
rgw cache lru size = 100000</pre>
306 1 Patrick McGarry
Caching the hot objects improves the GET performance.
307 15 Patrick McGarry
# Using larger PG split/merge value.  (since Firefly)
308 15 Patrick McGarry
<pre>filestore_merge_threshold = 500
309 15 Patrick McGarry
filestore_split_multiple = 100</pre>
310 1 Patrick McGarry
We find PG split/merge will introduce a big overhead. Using a large value would postpone the split/merge behavior. This will help the case where lots of small files are stored in the cluster.
311 15 Patrick McGarry
# Using load balancer with multiple RGW instances (since Cuttlefish)
312 15 Patrick McGarry
!9-load-balancer-with-multi-rgw-instances.png!
313 1 Patrick McGarry
We’ve found that the RGW has some scalability issues at present. With a single RGW instance the performance is poor. Running multiple RGW instances with a load balancer (e.g., Haproxy) will greatly improve the throughput.
314 15 Patrick McGarry
# Increase the number of Rados handlers (since Hammer)
315 1 Patrick McGarry
Since Hammer it’s able to using multiple number of Rados handlers per RGW instances. Increasing this value should improve the performance.
316 15 Patrick McGarry
# Using Civetweb frontend (since Giant)
317 1 Patrick McGarry
Before Giant, Apache + Libfastcgi were the recommended settings. However libfastcgi still use the very old ‘select’ mode, which is not able to handle large amount of concurrent IO in our testing. Using Civetweb frontend would help to improve the stability.
318 15 Patrick McGarry
<pre>rgw frontends =civetweb port=80</pre>
319 15 Patrick McGarry
# Moving bucket index to SSD (since Giant)
320 1 Patrick McGarry
Bucket index updating maybe some bottleneck if there’s millions of objects in one single bucket. We’ve find moving the bucket index to SSD storage will improve the performance.
321 15 Patrick McGarry
# Bucket Index Sharding (since Hammer)
322 1 Patrick McGarry
We’ve find the bucket index sharding is a problem if there’s large amount of objects inside one bucket. However the index listing speed may be impacted.
323 1 Patrick McGarry
324 15 Patrick McGarry
h3. Erasure Coding Tuning
325 1 Patrick McGarry
326 15 Patrick McGarry
# Use larger stripe width 
327 15 Patrick McGarry
The default erasure code stripe size (4K) is not optimal, We find using a bigger value (64K) will reduce the CPU% a lot (10%+)
328 15 Patrick McGarry
<pre>osd_pool_erasure_code_stripe_width = 65536</pre>
329 15 Patrick McGarry
# Use mid-sized K
330 1 Patrick McGarry
For the Erasure Code algorithms, we find using some mid-sized K value would bring balanced results between throughput and CPU%. We recommend to use 10+4 or 8+2 mode
331 1 Patrick McGarry
332 15 Patrick McGarry
h3. Appendix
333 15 Patrick McGarry
334 15 Patrick McGarry
h3. Sample Ceph.conf 
335 15 Patrick McGarry
336 15 Patrick McGarry
<pre>[global]
337 1 Patrick McGarry
fsid = 35b08d01-b688-4b9a-947b-bc2e25719370
338 1 Patrick McGarry
mon_initial_members = gw2
339 1 Patrick McGarry
mon_host = 10.10.10.105
340 1 Patrick McGarry
filestore_xattr_use_omap = true
341 1 Patrick McGarry
auth_cluster_required = none
342 1 Patrick McGarry
auth_service_required = none
343 1 Patrick McGarry
auth_client_required = none
344 1 Patrick McGarry
debug_lockdep = 0/0
345 1 Patrick McGarry
debug_context = 0/0
346 1 Patrick McGarry
debug_crush = 0/0
347 1 Patrick McGarry
debug_buffer = 0/0
348 1 Patrick McGarry
debug_timer = 0/0
349 1 Patrick McGarry
debug_filer = 0/0
350 1 Patrick McGarry
debug_objecter = 0/0
351 1 Patrick McGarry
debug_rados = 0/0
352 1 Patrick McGarry
debug_rbd = 0/0
353 1 Patrick McGarry
debug_journaler = 0/0
354 1 Patrick McGarry
debug_objectcatcher = 0/0
355 1 Patrick McGarry
debug_client = 0/0
356 1 Patrick McGarry
debug_osd = 0/0
357 1 Patrick McGarry
debug_optracker = 0/0
358 1 Patrick McGarry
debug_objclass = 0/0
359 1 Patrick McGarry
debug_filestore = 0/0
360 1 Patrick McGarry
debug_journal = 0/0
361 1 Patrick McGarry
debug_ms = 0/0
362 1 Patrick McGarry
debug_monc = 0/0
363 1 Patrick McGarry
debug_tp = 0/0
364 1 Patrick McGarry
debug_auth = 0/0
365 1 Patrick McGarry
debug_finisher = 0/0
366 1 Patrick McGarry
debug_heartbeatmap = 0/0
367 1 Patrick McGarry
debug_perfcounter = 0/0
368 1 Patrick McGarry
debug_asok = 0/0
369 1 Patrick McGarry
debug_throttle = 0/0
370 1 Patrick McGarry
debug_mon = 0/0
371 1 Patrick McGarry
debug_paxos = 0/0
372 1 Patrick McGarry
debug_rgw = 0/0
373 1 Patrick McGarry
[mon]
374 1 Patrick McGarry
mon_pg_warn_max_per_osd=5000
375 1 Patrick McGarry
mon_max_pool_pg_num=106496
376 1 Patrick McGarry
[client]
377 1 Patrick McGarry
rbd cache = false
378 1 Patrick McGarry
[osd]
379 1 Patrick McGarry
osd mkfs type = xfs
380 1 Patrick McGarry
osd mount options xfs = rw,noatime,,nodiratime,inode64,logbsize=256k,delaylog
381 1 Patrick McGarry
osd mkfs options xfs = -f -i size=2048
382 1 Patrick McGarry
filestore_queue_max_ops=5000
383 1 Patrick McGarry
filestore_queue_max_bytes = 1048576000
384 1 Patrick McGarry
filestore_max_sync_interval = 10
385 1 Patrick McGarry
filestore_merge_threshold = 500
386 1 Patrick McGarry
filestore_split_multiple = 100
387 1 Patrick McGarry
osd_op_shard_threads = 8
388 1 Patrick McGarry
journal_max_write_entries = 5000
389 1 Patrick McGarry
journal_max_write_bytes = 1048576000
390 1 Patrick McGarry
journal_queueu_max_ops = 3000
391 1 Patrick McGarry
journal_queue_max_bytes = 1048576000
392 1 Patrick McGarry
ms_dispatch_throttle_bytes = 1048576000
393 1 Patrick McGarry
objecter_inflight_op_bytes = 1048576000
394 1 Patrick McGarry
public_network = 10.10.10.100/24
395 1 Patrick McGarry
cluster_network = 10.10.10.100/24
396 1 Patrick McGarry
397 1 Patrick McGarry
[client.radosgw.gw2-1]
398 1 Patrick McGarry
host = gw2
399 1 Patrick McGarry
keyring = /etc/ceph/ceph.client.radosgw.keyring
400 1 Patrick McGarry
rgw cache enabled = true
401 1 Patrick McGarry
rgw cache lru size = 100000
402 1 Patrick McGarry
rgw socket path = /var/run/ceph/ceph.client.radosgw.gw2-1.fastcgi.sock
403 1 Patrick McGarry
rgw thread pool size = 256
404 1 Patrick McGarry
rgw enable ops log = false
405 1 Patrick McGarry
rgw enable usage log = false
406 1 Patrick McGarry
log file = /dev/null
407 1 Patrick McGarry
rgw frontends =civetweb port=80
408 15 Patrick McGarry
rgw override bucket index max shards = 8</pre>
409 1 Patrick McGarry
410 15 Patrick McGarry
h3. Sample sysctl.conf 
411 15 Patrick McGarry
412 15 Patrick McGarry
<pre>fs.file-max = 6553600
413 1 Patrick McGarry
net.ipv4.ip_local_port_range = 1024 65000
414 1 Patrick McGarry
net.ipv4.tcp_fin_timeout = 20
415 1 Patrick McGarry
net.ipv4.tcp_max_syn_backlog = 819200
416 1 Patrick McGarry
net.ipv4.tcp_keepalive_time = 20
417 1 Patrick McGarry
kernel.msgmni = 2878
418 1 Patrick McGarry
kernel.sem = 256 32000 100 142
419 1 Patrick McGarry
kernel.shmmni = 4096
420 1 Patrick McGarry
net.core.rmem_default = 1048576
421 1 Patrick McGarry
net.core.rmem_max = 1048576
422 1 Patrick McGarry
net.core.wmem_default = 1048576
423 1 Patrick McGarry
net.core.wmem_max = 1048576
424 1 Patrick McGarry
net.core.somaxconn = 40000
425 1 Patrick McGarry
net.core.netdev_max_backlog = 300000
426 15 Patrick McGarry
net.ipv4.tcp_max_tw_buckets = 10000</pre>
427 1 Patrick McGarry
428 15 Patrick McGarry
h3. All-NVMe Ceph Cluster Tuning for MySQL workload
429 15 Patrick McGarry
430 15 Patrick McGarry
h3. Ceph.conf 
431 15 Patrick McGarry
432 15 Patrick McGarry
<pre>[global]
433 1 Patrick McGarry
        enable experimental unrecoverable data corrupting features = bluestore rocksdb
434 1 Patrick McGarry
        osd objectstore = bluestore
435 1 Patrick McGarry
        ms_type = async
436 1 Patrick McGarry
        rbd readahead disable after bytes = 0
437 1 Patrick McGarry
        rbd readahead max bytes = 4194304
438 1 Patrick McGarry
        bluestore default buffered read = true
439 1 Patrick McGarry
        auth client required = none
440 1 Patrick McGarry
        auth cluster required = none
441 1 Patrick McGarry
        auth service required = none
442 1 Patrick McGarry
        filestore xattr use omap = true
443 1 Patrick McGarry
        cluster network = 192.168.142.0/24, 192.168.143.0/24
444 1 Patrick McGarry
        private network = 192.168.144.0/24, 192.168.145.0/24
445 1 Patrick McGarry
        log file = /var/log/ceph/$name.log
446 1 Patrick McGarry
        log to syslog = false
447 1 Patrick McGarry
        mon compact on trim = false
448 1 Patrick McGarry
        osd pg bits = 8
449 1 Patrick McGarry
        osd pgp bits = 8
450 1 Patrick McGarry
        mon pg warn max object skew = 100000
451 1 Patrick McGarry
        mon pg warn min per osd = 0
452 1 Patrick McGarry
        mon pg warn max per osd = 32768
453 1 Patrick McGarry
        debug_lockdep = 0/0
454 1 Patrick McGarry
        debug_context = 0/0
455 1 Patrick McGarry
        debug_crush = 0/0
456 1 Patrick McGarry
        debug_buffer = 0/0
457 1 Patrick McGarry
        debug_timer = 0/0
458 1 Patrick McGarry
        debug_filer = 0/0
459 1 Patrick McGarry
        debug_objecter = 0/0
460 1 Patrick McGarry
        debug_rados = 0/0
461 1 Patrick McGarry
        debug_rbd = 0/0
462 1 Patrick McGarry
        debug_ms = 0/0
463 1 Patrick McGarry
        debug_monc = 0/0
464 1 Patrick McGarry
        debug_tp = 0/0
465 1 Patrick McGarry
        debug_auth = 0/0
466 1 Patrick McGarry
        debug_finisher = 0/0
467 1 Patrick McGarry
        debug_heartbeatmap = 0/0
468 1 Patrick McGarry
        debug_perfcounter = 0/0
469 1 Patrick McGarry
        debug_asok = 0/0
470 1 Patrick McGarry
        debug_throttle = 0/0
471 1 Patrick McGarry
        debug_mon = 0/0
472 1 Patrick McGarry
        debug_paxos = 0/0
473 1 Patrick McGarry
        debug_rgw = 0/0
474 1 Patrick McGarry
        perf = true
475 1 Patrick McGarry
        mutex_perf_counter = true
476 1 Patrick McGarry
        throttler_perf_counter = false
477 1 Patrick McGarry
        rbd cache = false
478 1 Patrick McGarry
[mon]
479 1 Patrick McGarry
        mon data =/home/bmpa/tmp_cbt/ceph/mon.$id
480 1 Patrick McGarry
        mon_max_pool_pg_num=166496
481 1 Patrick McGarry
        mon_osd_max_split_count = 10000
482 1 Patrick McGarry
        mon_pg_warn_max_per_osd = 10000
483 1 Patrick McGarry
[mon.a]
484 1 Patrick McGarry
        host = ft02
485 1 Patrick McGarry
        mon addr = 192.168.142.202:6789
486 1 Patrick McGarry
[osd]
487 1 Patrick McGarry
        osd_mount_options_xfs = rw,noatime,inode64,logbsize=256k,delaylog
488 1 Patrick McGarry
        osd_mkfs_options_xfs = -f -i size=2048
489 1 Patrick McGarry
        osd_op_threads = 32
490 1 Patrick McGarry
        filestore_queue_max_ops=5000
491 1 Patrick McGarry
        filestore_queue_committing_max_ops=5000
492 1 Patrick McGarry
        journal_max_write_entries=1000
493 1 Patrick McGarry
        journal_queue_max_ops=3000
494 1 Patrick McGarry
        objecter_inflight_ops=102400
495 1 Patrick McGarry
        filestore_wbthrottle_enable=false
496 1 Patrick McGarry
        filestore_queue_max_bytes=1048576000
497 1 Patrick McGarry
        filestore_queue_committing_max_bytes=1048576000
498 1 Patrick McGarry
        journal_max_write_bytes=1048576000
499 1 Patrick McGarry
        journal_queue_max_bytes=1048576000
500 1 Patrick McGarry
        ms_dispatch_throttle_bytes=1048576000
501 1 Patrick McGarry
        objecter_infilght_op_bytes=1048576000
502 1 Patrick McGarry
        osd_mkfs_type = xfs
503 1 Patrick McGarry
        filestore_max_sync_interval=10
504 1 Patrick McGarry
        osd_client_message_size_cap = 0
505 1 Patrick McGarry
        osd_client_message_cap = 0
506 1 Patrick McGarry
        osd_enable_op_tracker = false
507 1 Patrick McGarry
        filestore_fd_cache_size = 64
508 1 Patrick McGarry
        filestore_fd_cache_shards = 32
509 15 Patrick McGarry
        filestore_op_threads = 6</pre>
510 1 Patrick McGarry
511 15 Patrick McGarry
h3. CBT YAML
512 15 Patrick McGarry
513 15 Patrick McGarry
<pre>cluster:
514 1 Patrick McGarry
  user: "bmpa"
515 1 Patrick McGarry
  head: "ft01"
516 1 Patrick McGarry
  clients: ["ft01", "ft02", "ft03", "ft04", "ft05", "ft06"]
517 1 Patrick McGarry
  osds: ["hswNode01", "hswNode02", "hswNode03", "hswNode04", "hswNode05"]
518 1 Patrick McGarry
  mons:
519 1 Patrick McGarry
   ft02:
520 1 Patrick McGarry
     a: "192.168.142.202:6789"
521 1 Patrick McGarry
osds_per_node: 16
522 1 Patrick McGarry
  fs: xfs
523 1 Patrick McGarry
  mkfs_opts: '-f -i size=2048 -n size=64k'
524 1 Patrick McGarry
  mount_opts: '-o inode64,noatime,logbsize=256k'
525 1 Patrick McGarry
  conf_file: '/home/bmpa/cbt/ceph.conf'
526 1 Patrick McGarry
  use_existing: False
527 1 Patrick McGarry
  newstore_block: True
528 1 Patrick McGarry
  rebuild_every_test: False
529 1 Patrick McGarry
  clusterid: "ceph"
530 1 Patrick McGarry
iterations: 1
531 1 Patrick McGarry
  tmp_dir: "/home/bmpa/tmp_cbt"
532 1 Patrick McGarry
pool_profiles:
533 1 Patrick McGarry
    2rep:
534 1 Patrick McGarry
      pg_size: 8192
535 1 Patrick McGarry
      pgp_size: 8192
536 1 Patrick McGarry
      replication: 2
537 1 Patrick McGarry
benchmarks:
538 1 Patrick McGarry
  librbdfio:
539 1 Patrick McGarry
    time: 300
540 1 Patrick McGarry
    ramp: 300
541 1 Patrick McGarry
    vol_size: 10
542 1 Patrick McGarry
    mode: ['randrw']
543 1 Patrick McGarry
    rwmixread: [0,70,100]
544 1 Patrick McGarry
    op_size: [4096]
545 1 Patrick McGarry
    procs_per_volume: [1]
546 1 Patrick McGarry
    volumes_per_client: [10]
547 1 Patrick McGarry
    use_existing_volumes: False
548 1 Patrick McGarry
    iodepth: [4,8,16,32,64,128]
549 1 Patrick McGarry
    osd_ra: [4096]
550 1 Patrick McGarry
    norandommap: True
551 1 Patrick McGarry
    cmd_path: '/usr/local/bin/fio'
552 1 Patrick McGarry
    pool_profile: '2rep'
553 15 Patrick McGarry
    log_avg_msec: 250</pre>
554 1 Patrick McGarry
555 15 Patrick McGarry
h3. MySQL configuration file (my.cnf)
556 15 Patrick McGarry
557 15 Patrick McGarry
<pre>[client]
558 1 Patrick McGarry
port            = 3306
559 1 Patrick McGarry
socket          = /var/run/mysqld/mysqld.sock
560 1 Patrick McGarry
[mysqld_safe]
561 1 Patrick McGarry
socket          = /var/run/mysqld/mysqld.sock
562 1 Patrick McGarry
nice            = 0
563 1 Patrick McGarry
[mysqld]
564 1 Patrick McGarry
user            = mysql
565 1 Patrick McGarry
pid-file        = /var/run/mysqld/mysqld.pid
566 1 Patrick McGarry
socket          = /var/run/mysqld/mysqld.sock
567 1 Patrick McGarry
port            = 3306
568 1 Patrick McGarry
datadir         = /data
569 1 Patrick McGarry
basedir         = /usr
570 1 Patrick McGarry
tmpdir          = /tmp
571 1 Patrick McGarry
lc-messages-dir = /usr/share/mysql
572 1 Patrick McGarry
skip-external-locking
573 1 Patrick McGarry
bind-address            = 0.0.0.0
574 1 Patrick McGarry
max_allowed_packet      = 16M
575 1 Patrick McGarry
thread_stack            = 192K
576 1 Patrick McGarry
thread_cache_size       = 8
577 1 Patrick McGarry
query_cache_limit       = 1M
578 1 Patrick McGarry
query_cache_size        = 16M
579 1 Patrick McGarry
log_error = /var/log/mysql/error.log
580 1 Patrick McGarry
expire_logs_days        = 10
581 1 Patrick McGarry
max_binlog_size         = 100M
582 1 Patrick McGarry
performance_schema=off
583 1 Patrick McGarry
innodb_buffer_pool_size = 25G
584 1 Patrick McGarry
innodb_flush_method = O_DIRECT
585 1 Patrick McGarry
innodb_log_file_size=4G
586 1 Patrick McGarry
thread_cache_size=16
587 1 Patrick McGarry
innodb_file_per_table
588 1 Patrick McGarry
innodb_checksums = 0
589 1 Patrick McGarry
innodb_flush_log_at_trx_commit = 0
590 1 Patrick McGarry
innodb_write_io_threads = 8
591 1 Patrick McGarry
innodb_page_cleaners= 16
592 1 Patrick McGarry
innodb_read_io_threads = 8
593 1 Patrick McGarry
max_connections = 50000
594 1 Patrick McGarry
[mysqldump]
595 1 Patrick McGarry
quick
596 1 Patrick McGarry
quote-names
597 1 Patrick McGarry
max_allowed_packet      = 16M
598 1 Patrick McGarry
[mysql]
599 15 Patrick McGarry
!includedir /etc/mysql/conf.d/</pre>
600 1 Patrick McGarry
601 1 Patrick McGarry
602 15 Patrick McGarry
h3. Sample Ceph Vendor Solutions
603 15 Patrick McGarry
604 1 Patrick McGarry
The following are pointers to Ceph solutions, but this list is not comprehensive:
605 15 Patrick McGarry
* https://www.dell.com/learn/us/en/04/shared-content~data-sheets~en/documents~dell-red-hat-cloud-solutions.pdf 
606 15 Patrick McGarry
* http://www.fujitsu.com/global/products/computing/storage/eternus-cd/
607 15 Patrick McGarry
* http://www8.hp.com/h20195/v2/GetPDF.aspx/4AA5-2799ENW.pdf http://www8.hp.com/h20195/v2/GetPDF.aspx/4AA5-8638ENW.pdf 
608 15 Patrick McGarry
* http://www.supermicro.com/solutions/storage_ceph.cfm 
609 15 Patrick McGarry
* https://www.thomas-krenn.com/en/products/storage-systems/suse-enterprise-storage.html
610 15 Patrick McGarry
* http://www.qct.io/Solution/Software-Defined-Infrastructure/Storage-Virtualization/QCT-and-Red-Hat-Ceph-Storage-p365c225c226c230 
611 1 Patrick McGarry
612 15 Patrick McGarry
613 15 Patrick McGarry
*Notices:*
614 1 Patrick McGarry
Copyright © 2016 Intel Corporation. All rights reserved
615 1 Patrick McGarry
Intel, the Intel logo, Intel Atom, Intel Core, and Intel Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
616 1 Patrick McGarry
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.
617 1 Patrick McGarry
Intel® Hyper-Threading Technology available on select Intel® Core™ processors. Requires an Intel® HT Technology-enabled system. Consult your PC manufacturer. Performance will vary depending on the specific hardware and software used. For more information including details on which processors support HT Technology, visit http://www.intel.com/info/hyperthreading.
618 1 Patrick McGarry
Any software source code reprinted in this document is furnished under a software license and may only be used or copied in accordance with the terms of that license.
619 1 Patrick McGarry
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. 
620 1 Patrick McGarry
621 1 Patrick McGarry
A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. 
622 1 Patrick McGarry
623 1 Patrick McGarry
Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. 
624 1 Patrick McGarry
625 1 Patrick McGarry
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. 
626 1 Patrick McGarry
627 1 Patrick McGarry
Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.