Ceph

h1. A standard framework for Ceph performance profiling with latency breakdown

h3. Summary

When we working on Ceph performance evaluation and optimization, we found how to trouble shoot the bottlenecks, identify the best tuning knobs from many parameters and handle the unexpected performance regression between different releases is pretty difficult. So we'd like to propose a general framework based on CBT and existing LTTng and other components we need to add to build a user friendly framework to profile and tune Ceph performance. 

h3. Owners

* Chendi Xue (Intel)

* Jian Zhang (Intel)

* Mark Nelson (Red Hat)

h3. Interested Parties

* Andrew Shewmaker (Red Hat, UCSC)

* Danny Al-Gaaf (Deutsche Telekom)

* Name

h3. Current Status

 From our point of view, the framework should comprises three parts of work:

(1) Deployment

CBT to deploy the Ceph cluster

script to deploy lttng and zipkin.

(2) Workload generator,

Enable COSBench as a CBT Object storage plugin, Integrate COSBench into CBT as the object(rgw) workload generator.

Current proposed approach is:

Let the end user to deploy COSBench

Add a plugin model in CBT to extract the COSBench related parameters and translate it to COSBench XML files

Use the plugin to kick off COSBench to generate  the load and get the data

(3) Analyzer

Adding more trace points

Leverage blkin (LTTng + Zipkin) patch to do the Ceph latency breakdown.

h3. Detailed Description

(1)Workload generator

Cosbench is an open source Benchmarking tool developed by intel to measure Cloud Object Storage Service performance.

What we want to do is to extend CBT benchmark part, to support calling cosbench do the object(rgw) test.

(2)Analyzer

After investigating all current latency breakdown method in ceph, we think the BLKIN(LTTNG+ZIPKIN) patch seems to be the most promised approach.

Patch is first posted by Marios Kogias in August. http://www.spinics.net/lists/ceph-devel/msg19890.html

Currently owned by Andrew Shewmaker. https://github.com/agshew/ceph/tree/wip-blkin-v4

What we want to do is to

help to make the BLKIN patch bug less, stable and small overhead.

Add support in CBT to run a performance test with BLKIN.

Add more tracepoints followed BLKIN method to do "Latency Breakdown".

Exists problem

BLKIN patch is not merged to ceph master yet, so we need to rebase BLKIN patch to each new release.

Current BLKIN codes only cover the MOSDOP, MOSDSubop and MOSDRepop, may extents to other msg in the future.

Zipkin can only show the latency breakdown for one specific request. So the webpage will show thousands of requests, we need to summarize all request, add a page to show average relative latency and a timeline.

Can't link/merge client, primary and replica tracepoints in the same view. Because of multi OSD server system time delta problem, we aren't able to find a best solution yet to get the time delta by microseconds.

h3. Work items

h4. Coding tasks

Task 1:  make CBT support running cosbench test

Task 2:  BLKIN patch bug fixing.

Task 3:  script to deploy lttng and zipkin. 

h4. Build / release tasks

# Task 1

# Task 2

# Task 3

h4. Documentation tasks

# Task 1

# Task 2

# Task 3

h4. Deprecation tasks

# Task 1

# Task 2

# Task 3