Project

General

Profile

Add SystemtapDtrace static markers » History » Version 2

Jessica Mack, 06/23/2015 01:51 AM

1 1 Jessica Mack
h1. Add SystemtapDtrace static markers
2 1 Jessica Mack
3 1 Jessica Mack
h3. Summary
4 1 Jessica Mack
5 1 Jessica Mack
Dtrace and SystemTap are monitoring tools, each providing a way to inspect what the processes on a computer system are doing. They both use domain-specific languages allowing a user to write scripts which:
6 1 Jessica Mack
  - filter which processes are to be observed
7 1 Jessica Mack
  - gather data from the processes of interest
8 1 Jessica Mack
  - generate reports on the data
9 1 Jessica Mack
10 1 Jessica Mack
Ceph can be built with embedded "markers" that can be observed by a SystemTap script, making it easier to monitor what the Ceph
11 1 Jessica Mack
processes on a system are doing.
12 1 Jessica Mack
 
13 1 Jessica Mack
h3. Owners
14 1 Jessica Mack
15 1 Jessica Mack
* Haomai Wang (UnitedStack)
16 1 Jessica Mack
* Danny Al-Gaaf (Deutsche Telekom AG)
17 1 Jessica Mack
* Marc Koderer (Deutsche Telekom AG)
18 1 Jessica Mack
* Name
19 1 Jessica Mack
20 1 Jessica Mack
h3. Interested Parties
21 1 Jessica Mack
22 1 Jessica Mack
* Guang Yang (Yahoo!)
23 1 Jessica Mack
* Saket Sinha (Google Summer of Code 2014)
24 2 Jessica Mack
25 2 Jessica Mack
h3. Current Status
26 1 Jessica Mack
 
27 2 Jessica Mack
h3. Detailed Description
28 1 Jessica Mack
 
29 1 Jessica Mack
Now MySQL, PosgreSQL, Glibc, CPython and others all support static markers which benefits much for developers. Alghough Ceph already has intern "perf counter" component which plays  a subset role. see (ceph.com/docs/master/dev/logs/#performance-counters). Compared to Systemtap/Dtrace, It obviously has some limitations, for example, we can't program it when running, lack of flexibility, need to write toolchains to display, visualization or format it from the result of "perf dump". What's more, Systemtap/Dtrace static markers focus on Ceph developers, and "perf counter" is likely designed for system admin.
30 1 Jessica Mack
 
31 1 Jessica Mack
Static markers in Ceph can be enabled with build progress, like:
32 1 Jessica Mack
./configure --enable-systemtap
33 1 Jessica Mack
 
34 1 Jessica Mack
At first, We would like to add markers in critical I/O path similliar to perf counter.  
35 1 Jessica Mack
36 1 Jessica Mack
h3. Work items
37 1 Jessica Mack
38 1 Jessica Mack
Identify our Use Case
39 1 Jessica Mack
Following tasks can be performed in the Linux by System Tap-
40 1 Jessica Mack
1. Track Scheduling Time -
41 1 Jessica Mack
       a) Peek into kernel scheduler
42 1 Jessica Mack
       b) Visualize current activity
43 1 Jessica Mack
       c) Evaluate System Load
44 1 Jessica Mack
2. Tweak I/O usage -  which process responsible for most I/O
45 1 Jessica Mack
3. Kernel Profiling - Record stack traces
46 1 Jessica Mack
4. Call graph tracing - Start tracing when a trigger is called, Visualize all functions that are invoked, how long each step takes 
47 1 Jessica Mack
48 1 Jessica Mack
h4. Coding tasks
49 1 Jessica Mack
50 1 Jessica Mack
# Add basic workable infrastruction
51 1 Jessica Mack
# Add more static markers
52 1 Jessica Mack
53 1 Jessica Mack
h4. Build / release tasks
54 1 Jessica Mack
55 1 Jessica Mack
# Task 1
56 1 Jessica Mack
# Task 2
57 1 Jessica Mack
# Task 3
58 1 Jessica Mack
59 1 Jessica Mack
h4. Documentation tasks
60 1 Jessica Mack
61 1 Jessica Mack
# Add usage and build related document
62 1 Jessica Mack
63 1 Jessica Mack
h4. Pad
64 1 Jessica Mack
65 1 Jessica Mack
http://pad.ceph.com/p/cdsgiant-syste...static-markers
66 1 Jessica Mack
http://pad.ceph.com/p/GH-systemtap-l...static-markers
67 1 Jessica Mack
68 1 Jessica Mack
h4. Resources
69 1 Jessica Mack
70 1 Jessica Mack
Systemtap/DTrace:
71 1 Jessica Mack
* http://www.ibm.com/developerworks/ai...aceprobes.html
72 1 Jessica Mack
* https://sourceware.org/systemtap/wiki/HomePage?action=show&redirect=SystemTap
73 1 Jessica Mack
* https://sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApp
74 1 Jessica Mack
 
75 1 Jessica Mack
LTTng:
76 1 Jessica Mack
* http://lttng.org/comparison-systemtap-and-dtrace
77 1 Jessica Mack
* https://lttng.org/files/doc/man-page...tng-ust.3.html