Ceph

h1. Add SystemtapDtrace static markers

h3. Summary

Dtrace and SystemTap are monitoring tools, each providing a way to inspect what the processes on a computer system are doing. They both use domain-specific languages allowing a user to write scripts which:

  - filter which processes are to be observed

  - gather data from the processes of interest

  - generate reports on the data

Ceph can be built with embedded "markers" that can be observed by a SystemTap script, making it easier to monitor what the Ceph

processes on a system are doing.

h3. Owners

* Haomai Wang (UnitedStack)

* Danny Al-Gaaf (Deutsche Telekom AG)

* Marc Koderer (Deutsche Telekom AG)

* Name

h3. Interested Parties

* Guang Yang (Yahoo!)

* Saket Sinha (Google Summer of Code 2014)

h3. Current Status

h3. Detailed Description

Now MySQL, PosgreSQL, Glibc, CPython and others all support static markers which benefits much for developers. Alghough Ceph already has intern "perf counter" component which plays  a subset role. see (ceph.com/docs/master/dev/logs/#performance-counters). Compared to Systemtap/Dtrace, It obviously has some limitations, for example, we can't program it when running, lack of flexibility, need to write toolchains to display, visualization or format it from the result of "perf dump". What's more, Systemtap/Dtrace static markers focus on Ceph developers, and "perf counter" is likely designed for system admin.

Static markers in Ceph can be enabled with build progress, like:

./configure --enable-systemtap

At first, We would like to add markers in critical I/O path similliar to perf counter.  

h3. Work items

Identify our Use Case

Following tasks can be performed in the Linux by System Tap-

1. Track Scheduling Time -

       a) Peek into kernel scheduler

       b) Visualize current activity

       c) Evaluate System Load

2. Tweak I/O usage -  which process responsible for most I/O

3. Kernel Profiling - Record stack traces

4. Call graph tracing - Start tracing when a trigger is called, Visualize all functions that are invoked, how long each step takes 

h4. Coding tasks

# Add basic workable infrastruction

# Add more static markers

h4. Build / release tasks

# Task 1

# Task 2

# Task 3

h4. Documentation tasks

# Add usage and build related document

h4. Pad

http://pad.ceph.com/p/cdsgiant-syste...static-markers

http://pad.ceph.com/p/GH-systemtap-l...static-markers

h4. Resources

Systemtap/DTrace:

* http://www.ibm.com/developerworks/ai...aceprobes.html

* https://sourceware.org/systemtap/wiki/HomePage?action=show&redirect=SystemTap

* https://sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApp

LTTng:

* http://lttng.org/comparison-systemtap-and-dtrace

* https://lttng.org/files/doc/man-page...tng-ust.3.html