Project

General

Profile

RESTful API for DR Geo-Replication » History » Version 3

Jessica Mack, 06/01/2015 06:28 PM

1 1 Jessica Mack
h1. RESTful API for DR  Geo-Replication
2 1 Jessica Mack
3 1 Jessica Mack
h3. System User Special Requests
4 1 Jessica Mack
5 1 Jessica Mack
In general, a system user (a user that has the system flag set on) may operate on behalf of other users. Certain requests may either have special request params (always prefixed by 'rgwx-') or may have special header fields (prefixed by 'Rgwx-').
6 1 Jessica Mack
7 1 Jessica Mack
h4. Params:
8 1 Jessica Mack
9 1 Jessica Mack
* rgwx-uid
10 1 Jessica Mack
11 1 Jessica Mack
The effective user id for this request
12 1 Jessica Mack
 
13 1 Jessica Mack
h4. Create bucket for a non-master region
14 1 Jessica Mack
15 1 Jessica Mack
Tells the master region that a user tries to create a bucket. Can only be executed by a system user. The original bucket creation request is being forwarded under a different user context, with some extra params.
16 1 Jessica Mack
 
17 1 Jessica Mack
Syntax:
18 1 Jessica Mack
 
19 1 Jessica Mack
PUT /bucket
20 1 Jessica Mack
 
21 1 Jessica Mack
Extra Params:
22 1 Jessica Mack
 
23 1 Jessica Mack
1. rgwx-region: source region for request
24 1 Jessica Mack
2. rgwx-uid: effective user id for bucket creation
25 1 Jessica Mack
 
26 1 Jessica Mack
Response:
27 1 Jessica Mack
Response data for a successful bucket creation returns the JSON encoded bucket object metadata version and the bucket info.
28 1 Jessica Mack
 
29 1 Jessica Mack
h4. Extended object GET
30 1 Jessica Mack
31 1 Jessica Mack
Read an object, send extra metadata about object. This request extends generic object read.
32 1 Jessica Mack
 
33 1 Jessica Mack
Syntax
34 1 Jessica Mack
GET /bucket/object
35 1 Jessica Mack
 
36 1 Jessica Mack
Extra Params:
37 1 Jessica Mack
1. rgwx-prepend-metadata: whether object metadata will be prepended to object data as part of the response.
38 1 Jessica Mack
 
39 1 Jessica Mack
Extra Response Headers:
40 1 Jessica Mack
 
41 1 Jessica Mack
1.  Rgwx-Embedded-Metadata-Len: Length of prepended metadata (if requested)
42 1 Jessica Mack
 
43 1 Jessica Mack
h4. Extended object PUT
44 1 Jessica Mack
45 1 Jessica Mack
Writes an object, return extra object metadata. Extends regular object write.
46 1 Jessica Mack
 
47 1 Jessica Mack
Syntax
48 1 Jessica Mack
PUT /bucket/object
49 1 Jessica Mack
 
50 1 Jessica Mack
Extra Response Headers:
51 1 Jessica Mack
1. Rgwx-Mtime: Object modification time.
52 1 Jessica Mack
*Sync object from zone within region (intra-region copy)*
53 1 Jessica Mack
54 1 Jessica Mack
Syntax:
55 1 Jessica Mack
PUT /bucket/objcect
56 1 Jessica Mack
 
57 1 Jessica Mack
Extra Params:
58 1 Jessica Mack
1. rgwx-source-zone: the source zone for where 
59 1 Jessica Mack
2. rgwx-client-id: the id of the agent that initiated the request
60 1 Jessica Mack
3. rgwx-op-od: a unique op id, created by the agent
61 1 Jessica Mack
 
62 1 Jessica Mack
It is up for the agent to provide a meaningful client id and a unique (for that agent) op id. These will be used for handling opstate information.
63 1 Jessica Mack
 
64 3 Jessica Mack
%{color:gold}Metadata%
65 1 Jessica Mack
 
66 1 Jessica Mack
h4. List metadata
67 1 Jessica Mack
68 1 Jessica Mack
Display list of metadata sections, keys.
69 1 Jessica Mack
70 1 Jessica Mack
Syntax:
71 1 Jessica Mack
72 1 Jessica Mack
GET /admin/metadata[/section]
73 1 Jessica Mack
74 1 Jessica Mack
Response:
75 1 Jessica Mack
76 1 Jessica Mack
JSON encoded list of sections or keys.
77 1 Jessica Mack
78 1 Jessica Mack
Examples:
79 1 Jessica Mack
80 1 Jessica Mack
1. GET /admin/metadata
81 1 Jessica Mack
82 1 Jessica Mack
[
83 1 Jessica Mack
    "user"]
84 1 Jessica Mack
85 1 Jessica Mack
86 1 Jessica Mack
2. GET /admin/metadata/user
87 1 Jessica Mack
88 1 Jessica Mack
[
89 1 Jessica Mack
    "yehsad",
90 1 Jessica Mack
    "greg"]
91 1 Jessica Mack
 
92 1 Jessica Mack
h4. Get metadata info
93 1 Jessica Mack
94 1 Jessica Mack
Syntax:
95 1 Jessica Mack
96 1 Jessica Mack
GET /admin/metadata/<section>?key=<key>
97 1 Jessica Mack
98 1 Jessica Mack
Response:
99 1 Jessica Mack
100 1 Jessica Mack
JSON encoded metadata info
101 1 Jessica Mack
102 1 Jessica Mack
Examples:
103 1 Jessica Mack
104 1 Jessica Mack
GET /admin/metadata/user?key=yehsad
105 1 Jessica Mack
106 1 Jessica Mack
{ "key": "user:yehsad",
107 1 Jessica Mack
  "ver": { "ver": 3,
108 1 Jessica Mack
      "tag": "8L0en1nFM1Y+NaQrBAzgPliZ"},
109 1 Jessica Mack
  "data": { "user_id": "yehsad",
110 1 Jessica Mack
      "display_name": "yehuda",
111 1 Jessica Mack
      "email": "",
112 1 Jessica Mack
      "suspended": 0,
113 1 Jessica Mack
      "max_buckets": 1000,
114 1 Jessica Mack
      "auid": 0,
115 1 Jessica Mack
      "subusers": [
116 1 Jessica Mack
            { "id": "yehsad:yehsad",
117 1 Jessica Mack
              "permissions": "<none>"}],
118 1 Jessica Mack
      "keys": [
119 1 Jessica Mack
            { "user": "yehsad",
120 1 Jessica Mack
              "access_key": "foo",
121 1 Jessica Mack
              "secret_key": "bar"}],
122 1 Jessica Mack
      "swift_keys": [
123 1 Jessica Mack
            { "user": "yehsad:yehsad",
124 1 Jessica Mack
              "secret_key": "zxczxc"}],
125 1 Jessica Mack
      "caps": []}}
126 1 Jessica Mack
127 1 Jessica Mack
 
128 1 Jessica Mack
h4. Set metadata info
129 1 Jessica Mack
130 1 Jessica Mack
Syntax:
131 1 Jessica Mack
132 1 Jessica Mack
PUT /admin/metadata/<section>?key=<key>
133 1 Jessica Mack
134 1 Jessica Mack
<JSON encoded metadata>
135 1 Jessica Mack
136 1 Jessica Mack
Response:
137 1 Jessica Mack
138 1 Jessica Mack
Examples:
139 1 Jessica Mack
140 1 Jessica Mack
PUT /admin/metadata/user?key=yehsad
141 1 Jessica Mack
142 1 Jessica Mack
{ "key": "user:yehsad",
143 1 Jessica Mack
  "ver": { "ver": 4,
144 1 Jessica Mack
      "tag": "8L0en1nFM1Y+NaQrBAzgPliZ"},
145 1 Jessica Mack
  "data": { "user_id": "yehsad",
146 1 Jessica Mack
      "display_name": "yehuda",
147 1 Jessica Mack
      "email": "",
148 1 Jessica Mack
      "suspended": 0,
149 1 Jessica Mack
      "max_buckets": 10,
150 1 Jessica Mack
      "auid": 0,
151 1 Jessica Mack
      "subusers": [
152 1 Jessica Mack
            { "id": "yehsad:yehsad",
153 1 Jessica Mack
              "permissions": "<none>"}],
154 1 Jessica Mack
      "keys": [
155 1 Jessica Mack
            { "user": "yehsad",
156 1 Jessica Mack
              "access_key": "foo",
157 1 Jessica Mack
              "secret_key": "bar"}],
158 1 Jessica Mack
      "swift_keys": [
159 1 Jessica Mack
            { "user": "yehsad:yehsad",
160 1 Jessica Mack
              "secret_key": "zxczxc"}],
161 1 Jessica Mack
      "caps": []}}
162 1 Jessica Mack
 
163 1 Jessica Mack
164 1 Jessica Mack
h4. Remove metadata
165 1 Jessica Mack
 
166 1 Jessica Mack
Remove a metadata key
167 1 Jessica Mack
168 1 Jessica Mack
Syntax:
169 1 Jessica Mack
170 1 Jessica Mack
DELETE /admin/metadata/<section>?key=<key>
171 1 Jessica Mack
172 1 Jessica Mack
Response:
173 1 Jessica Mack
174 1 Jessica Mack
Example:
175 1 Jessica Mack
176 1 Jessica Mack
DELETE /admin/metadata/user?key=yehsad
177 1 Jessica Mack
178 1 Jessica Mack
179 1 Jessica Mack
h3. Data, Metadata Logs
180 1 Jessica Mack
181 2 Jessica Mack
h4. get log info
182 1 Jessica Mack
 
183 1 Jessica Mack
Returns info about the log. Currently that only includes the number of log shards.
184 1 Jessica Mack
 
185 1 Jessica Mack
Syntax:
186 1 Jessica Mack
 
187 1 Jessica Mack
GET /admin/log?type=<type>
188 1 Jessica Mack
 
189 1 Jessica Mack
params:
190 1 Jessica Mack
1. type: metadata, data
191 1 Jessica Mack
 
192 1 Jessica Mack
result:
193 1 Jessica Mack
 
194 1 Jessica Mack
return a JSON representation of the data log information
195 1 Jessica Mack
 
196 1 Jessica Mack
{
197 1 Jessica Mack
  num_objects = <num>
198 1 Jessica Mack
}
199 1 Jessica Mack
 
200 2 Jessica Mack
h4. ?get shard info
201 1 Jessica Mack
 
202 1 Jessica Mack
Retuns information about the current log shard state. Shard id is in the range of 0 to num_objects - 1 (num_objects can be retrieved by the get log info request).
203 1 Jessica Mack
 
204 1 Jessica Mack
Syntax:
205 1 Jessica Mack
 
206 1 Jessica Mack
GET /admin/log?info&type=<type>&id=<shard-id>
207 1 Jessica Mack
 
208 1 Jessica Mack
params:
209 1 Jessica Mack
1. type: metadata, data
210 1 Jessica Mack
2. id: shard id
211 1 Jessica Mack
 
212 1 Jessica Mack
result:
213 1 Jessica Mack
 
214 1 Jessica Mack
return a JSON representation of the data log information
215 1 Jessica Mack
 
216 1 Jessica Mack
{
217 1 Jessica Mack
  num_objects = <num>
218 1 Jessica Mack
}
219 1 Jessica Mack
 
220 1 Jessica Mack
h4. Lock log shard
221 1 Jessica Mack
 
222 1 Jessica Mack
POST /admin/log?lock
223 1 Jessica Mack
 
224 1 Jessica Mack
 
225 1 Jessica Mack
params:
226 1 Jessica Mack
 
227 1 Jessica Mack
1. type: metadata, data
228 1 Jessica Mack
2. id: shard id (0..num_objects)
229 1 Jessica Mack
3. length: time in seconds for lock to be valid
230 1 Jessica Mack
4. zone-id: locker zone id
231 1 Jessica Mack
5. locker-id: client id (e.g., agent id).
232 1 Jessica Mack
 
233 1 Jessica Mack
Returns 404 if there are no entries of the specified type in the shard.
234 1 Jessica Mack
235 1 Jessica Mack
h4. Unlock log shard
236 1 Jessica Mack
 
237 1 Jessica Mack
POST /admin/log?unlock
238 1 Jessica Mack
 
239 1 Jessica Mack
params:
240 1 Jessica Mack
 
241 1 Jessica Mack
1. type: metadata, data
242 1 Jessica Mack
2. id: shard id (0..num_objects)
243 1 Jessica Mack
3. zone-id: locker zone id
244 1 Jessica Mack
4. locker-id: unique user identifier that corresponds to a previous lock call
245 1 Jessica Mack
 
246 1 Jessica Mack
h4. list log
247 1 Jessica Mack
 
248 1 Jessica Mack
GET /admin/log?id=<shard id>
249 1 Jessica Mack
 
250 1 Jessica Mack
params:
251 1 Jessica Mack
 
252 1 Jessica Mack
1. type: metadata, data
253 1 Jessica Mack
2. id=<shard id>
254 1 Jessica Mack
3. start-time=<start time> (optional)
255 1 Jessica Mack
4. end-time=<end time> (optional)
256 1 Jessica Mack
5. marker=<start marker> (optional)
257 1 Jessica Mack
 
258 1 Jessica Mack
Need to have either one pair of start-time and end-time specified, or start-marker and end-marker specified. Markers supersede timestamps, and they correspond to the 'id' field in the log entries.
259 1 Jessica Mack
 
260 1 Jessica Mack
h4. trim data log
261 1 Jessica Mack
 
262 1 Jessica Mack
DELETE /admin/log?id=<shard id>
263 1 Jessica Mack
 
264 1 Jessica Mack
params:
265 1 Jessica Mack
 
266 1 Jessica Mack
1. type: metadata, data
267 1 Jessica Mack
2. id=<shard id>
268 1 Jessica Mack
3. start-time=<start time>
269 1 Jessica Mack
4. end-time=<end time>
270 1 Jessica Mack
5. start-marker=<start marker>
271 1 Jessica Mack
6. end-marker=<end marker> 
272 1 Jessica Mack
 
273 1 Jessica Mack
Need to have either one pair of start-time and end-time specified, or start-marker and end-marker specified. Markers supersede timestamps, and they correspond to the 'id' field in the log entries.
274 1 Jessica Mack
 
275 1 Jessica Mack
h3. Bucket Index Log
276 1 Jessica Mack
 
277 1 Jessica Mack
h4. List bucket index log
278 1 Jessica Mack
279 1 Jessica Mack
GET /admin/log?type=bucket-index
280 1 Jessica Mack
281 1 Jessica Mack
params:
282 1 Jessica Mack
283 1 Jessica Mack
1. bucket=<bucket name>
284 1 Jessica Mack
2. bucket-instance=<bucket instance id>
285 1 Jessica Mack
3. start-marker=<start-marker>
286 1 Jessica Mack
4. end-marker=<end-marker>
287 1 Jessica Mack
Notes:
288 1 Jessica Mack
 - One of either bucket or bucket-instance must be specified.
289 1 Jessica Mack
 - The marker param can be used to iterate through entries. It corresponds to the "id" field in a returned entry.
290 1 Jessica Mack
 
291 1 Jessica Mack
h4. Get bucket index log info
292 1 Jessica Mack
 
293 1 Jessica Mack
Get information about bucket index log state. Can be used to retrieve current last log entry's marker position.
294 1 Jessica Mack
295 1 Jessica Mack
GET /admin/log?info&type=bucket-index
296 1 Jessica Mack
297 1 Jessica Mack
params:
298 1 Jessica Mack
299 1 Jessica Mack
1. bucket=<bucket name>
300 1 Jessica Mack
2. bucket-instance=<bucket instance id>Notes:
301 1 Jessica Mack
 - One of either bucket or bucket-instance must be specified.
302 1 Jessica Mack
 
303 1 Jessica Mack
h4. trim bucket index log
304 1 Jessica Mack
305 1 Jessica Mack
306 1 Jessica Mack
DELETE /admin/log?type=bucket-index
307 1 Jessica Mack
308 1 Jessica Mack
params:
309 1 Jessica Mack
310 1 Jessica Mack
1. bucket=<bucket name>
311 1 Jessica Mack
2. bucket-instance=<bucket instance>
312 1 Jessica Mack
3. marker=<marker> (optional)
313 1 Jessica Mack
4. max-entries=<max entries> (optional, default is 1000)
314 1 Jessica Mack
 
315 1 Jessica Mack
Notes:
316 1 Jessica Mack
 - One of either bucket or bucket-instance must be specified.
317 1 Jessica Mack
 
318 1 Jessica Mack
h3. Replica Log
319 1 Jessica Mack
320 1 Jessica Mack
h4. Data/Metadata Replica Log
321 1 Jessica Mack
322 1 Jessica Mack
h4. Set a Worker Bound
323 1 Jessica Mack
324 1 Jessica Mack
POST /admin/replica_log?work_bound
325 1 Jessica Mack
 
326 1 Jessica Mack
params:
327 1 Jessica Mack
1) type: metadata, data
328 1 Jessica Mack
2) id: shard id
329 1 Jessica Mack
3) marker: as used for log listing
330 1 Jessica Mack
4) time: the lower bound master zone time (corresponds to the marker)
331 1 Jessica Mack
5) daemon_id: a user-provided string, which can be anything.
332 1 Jessica Mack
 
333 1 Jessica Mack
Currenly there can only be one worker bound stored for a data or metadata shard, and there are a few constraints on setting it:
334 1 Jessica Mack
daemon_id must be the same as the existing entry, if there is one
335 1 Jessica Mack
time must not be before the existing timestamp, if there is one
336 1 Jessica Mack
marker must not be empty
337 1 Jessica Mack
 
338 1 Jessica Mack
body:
339 1 Jessica Mack
the body of the request includes a JSON-encoded list of buckets which are "in progress" from before the lower bound, plus a lower bound on the time associated with each of them:
340 1 Jessica Mack
[ { "bucket": <name>, "time": <time> }, { "bucket": <name2>, "time": <time2> } ]
341 1 Jessica Mack
 
342 1 Jessica Mack
h4. Get the Worker bounds
343 1 Jessica Mack
344 1 Jessica Mack
GET /admin/replica_log?bounds
345 1 Jessica Mack
params:
346 1 Jessica Mack
1) type: metadata, data, bucket-index (see below)
347 1 Jessica Mack
2) id: shard id
348 1 Jessica Mack
 
349 1 Jessica Mack
This returns a JSON object containing a list of the lower bounds and bucket collections for each daemon_id, plus the aggregate's lowest marker and oldest time bound:
350 1 Jessica Mack
{
351 1 Jessica Mack
  "markers":
352 1 Jessica Mack
  [ { "entity": <id1>, "position_marker": <marker1>, "postion_time": <time1>, "items_in_progress": [ { "name": <name>, "timestamp":2 <time2> }, { "name": <name2>, "timestamp": <time3> } ] },
353 1 Jessica Mack
    { "entity": <id2>, "positon_marker": <marker2>, "position_time": <time4>, "items_in_progress": [ ] }
354 1 Jessica Mack
  ],
355 1 Jessica Mack
  "marker": <marker1>,
356 1 Jessica Mack
  "oldest_time": <time3>
357 1 Jessica Mack
}
358 1 Jessica Mack
 
359 1 Jessica Mack
h4. Remove a daemon's bound
360 1 Jessica Mack
361 1 Jessica Mack
DELETE /admin/replica_log?work_bound
362 1 Jessica Mack
params:
363 1 Jessica Mack
1) type: metadata, data, bucket-index (see below)
364 1 Jessica Mack
2) id: shard id
365 1 Jessica Mack
3) daemon_id: a user-provided string, which can be anything
366 1 Jessica Mack
 
367 1 Jessica Mack
h3. Bucket Index Replica Log
368 1 Jessica Mack
369 1 Jessica Mack
h4. Set a Worker Bound
370 1 Jessica Mack
371 1 Jessica Mack
POST /admin/replica_log?work_bound
372 1 Jessica Mack
params:
373 1 Jessica Mack
1) type=bucket-index
374 1 Jessica Mack
2) bucket-instance: bucket instance id
375 1 Jessica Mack
3) marker: as used for bucket log listing
376 1 Jessica Mack
4) time: the lower bound master zone time for this bucket (corresponds to the marker)
377 1 Jessica Mack
5) daemon_id: a user-provided string, which can be anything
378 1 Jessica Mack
 
379 1 Jessica Mack
Currenly there can only be one worker bound stored for a bucket-instance, and there are a few constraints on setting it:
380 1 Jessica Mack
daemon_id must be the same as the existing entry, if there is one
381 1 Jessica Mack
time must not be before the existing timestamp, if there is one
382 1 Jessica Mack
marker must not be empty
383 1 Jessica Mack
 
384 1 Jessica Mack
The request body includes a JSON list of objects in progress, which includes the time each of them became out-of-date:
385 1 Jessica Mack
[ { name:"objfoo", time:<time1>}, { name:"objbar", time:<time2>}, { name:"objbaz", time:<time3>} ]
386 1 Jessica Mack
 
387 1 Jessica Mack
h4. Get a Worker Bound
388 1 Jessica Mack
389 1 Jessica Mack
GET /admin/replica_log?bounds
390 1 Jessica Mack
params:
391 1 Jessica Mack
1) type=bucket-index
392 1 Jessica Mack
2) bucket-instance: bucket instance id
393 1 Jessica Mack
 
394 1 Jessica Mack
It returns an object containing the post data and the params from the last call for each daemon, plus an aggregate lowest bound marker and oldest_time
395 1 Jessica Mack
{
396 1 Jessica Mack
  "markers": [ { "entity": <id1>, "position_marker": <marker1>, "position_time": <time1>, "items_in_progress": [ { name:"objfoo", time:<time1>}, { name:"objbar", time:<time2>}, { name:"objbaz", time:<time3>} ],
397 1 Jessica Mack
    { "entity": <id2>, "position_marker": <marker2>, "position_time": <time2>, "items_in_progress": [] } ],
398 1 Jessica Mack
  "marker": <marker1>,
399 1 Jessica Mack
  "oldest_time": <time3>
400 1 Jessica Mack
}
401 1 Jessica Mack
 
402 1 Jessica Mack
h4. Delete a Worker's Bound
403 1 Jessica Mack
404 1 Jessica Mack
DELETE /admin/replica_log?work_bound
405 1 Jessica Mack
params:
406 1 Jessica Mack
1) type: bucket-index
407 1 Jessica Mack
2) bucket-instance: bucket instance id
408 1 Jessica Mack
3) daemon_id: a user-provided string, which can be anything
409 1 Jessica Mack
 
410 1 Jessica Mack
h3. OpState
411 1 Jessica Mack
 
412 1 Jessica Mack
h4. List entries
413 1 Jessica Mack
414 1 Jessica Mack
GET /admin/opstate
415 1 Jessica Mack
416 1 Jessica Mack
params:
417 1 Jessica Mack
418 1 Jessica Mack
1. client-id=<client-id> (optional)
419 1 Jessica Mack
2. op-id=<op-id> (optional)
420 1 Jessica Mack
3. object=<object> (optional)
421 1 Jessica Mack
4. max-entries=<max entries> (optional, default is 1000)
422 1 Jessica Mack
 
423 1 Jessica Mack
h4. Set entry
424 1 Jessica Mack
 
425 1 Jessica Mack
POST /admin/opstate
426 1 Jessica Mack
params:
427 1 Jessica Mack
1. client-id=<client-id> (required)
428 1 Jessica Mack
2. op-id=<op-id> (required)
429 1 Jessica Mack
3. object=<object> (requiredl)
430 1 Jessica Mack
4. state=<state> (required)
431 1 Jessica Mack
5. renew (optional)
432 1 Jessica Mack
if renew is specified then operation can only succeed if the state specified matches the existing state.
433 1 Jessica Mack
 
434 1 Jessica Mack
h4. Remove entry
435 1 Jessica Mack
 
436 1 Jessica Mack
DELETE /admin/opstate
437 1 Jessica Mack
params:
438 1 Jessica Mack
1. client-id=<client-id> (required)
439 1 Jessica Mack
2. op-id=<op-id> (required)
440 1 Jessica Mack
3. object=<object> (requiredl)
441 1 Jessica Mack
 
442 1 Jessica Mack
 
443 1 Jessica Mack
h3. Reading Configuration
444 1 Jessica Mack
 
445 1 Jessica Mack
At the moment, region configuration information is read-only.
446 1 Jessica Mack
 
447 1 Jessica Mack
h4. Get region map
448 1 Jessica Mack
 
449 1 Jessica Mack
Syntax:
450 1 Jessica Mack
GET /admin/config