Subtask #5862
closedFeature #4929: Erasure encoded placement group
FileStore must work with ghobjects rather than hobjects
100%
Description
vhobjects should be basically tuple<hobject_t, version_t, chunk_id_t> where chunk_id_t is probably uint8_t. A chunk_id_t of ~0 should be reserved for the case where the chunk field is unused, as for osd metadata or for Replicated pg objects. Similarly, a version_t value of ~0 should be reserved for osd metadata or Replicated pg objects. I suggest that an hobject_t value of x should implicitely convert into a vhobject_t value of (x, ~0, ~0) and be represented by the same filename as currently represents x (same for the keys generated for DBObjectMap). This suggests a natural upgrade path, values of (x, ~0, ~0) will continue to have filenames generated as they currently are. New values of (x, y, z) will have y and z appended to the end.
- Create a vhobject_t object in os/hobject.h
- Add a vhobject_t(const hobject_t &oid); implicit constructor to allow FileStore methods with vhobject_t parameters to accept hobject_t arguments.
- Convert all interfaces to use vhobject_t rather than hobject_t
- Convert internals to use vhobject_t
- Convert users of collection_list to handle vhobject_t return values
- Convert filename and KeyValueDB key generation to handle both flavors of vhobject_t
Updated by Loïc Dachary over 10 years ago
- Source changed from other to Development
Updated by David Zafman over 10 years ago
- Subject changed from FileStore must work with vhobjects rather than hobjects to FileStore must work with ghobjects rather than hobjects
Use a "generation" number (gen_t?) instead of version_t.
Use a "shard" or "slice" number (shard_t or slice_t) instead of chunk_id_t.
Updated by Loïc Dachary over 10 years ago
Here is how I understand stripe / shard|strip / chunk
- chunk : when the encoding function is called, it returns chunks of the same size.
- stripe : when an object is too large to be encoded with a single call, each set of chunks created by a call to the encoding function is called a stripe.
- shard|strip : the file that holds all chunks of a same rank for a given object.
OSD 40 OSD 33
+-------------------------+ +-------------------------+
| shard 0 - PG 10 | | shard 1 - PG 10 |
|+------ object O -------+| |+------ object O -------+|
||+---------------------+|| ||+---------------------+||
stripe||| chunk 0 ||| ||| chunk 1 ||| ...
0 ||| [0,+N) ||| ||| [0,+N) |||
||+---------------------+|| ||+---------------------+||
||+---------------------+|| ||+---------------------+||
stripe||| chunk 0 ||| ||| chunk 1 ||| ...
1 ||| [N,+N) ||| ||| [N,+N) |||
||+---------------------+|| ||+---------------------+||
||+---------------------+|| ||+---------------------+||
stripe||| chunk 0 [N*2,+len) ||| ||| chunk 1 [N*2,+len) ||| ...
2 ||+---------------------+|| ||+---------------------+||
|+-----------------------+| |+-----------------------+|
| ... | | ... |
+-------------------------+ +-------------------------+
I vote in favor of shard_t :-)
Updated by David Zafman over 10 years ago
The ceph-filestore-dump probably needs a version bump to prevent an import of an export which includes erasure coding. I wish there was a version number in the pg_info_t, but even if we add one now, the older version of ceph-filestore-dump needs to be prevented from importing from the newer pg.
Updated by Greg Farnum over 10 years ago
pg_info_t has our standard encode/decode function versioning. We've got some fairly complex ones you can examine where we conditionally encode either old or new format depending on what flags are set that you can look at if necessary.
Updated by Loïc Dachary over 10 years ago
- Status changed from New to In Progress
Updated by David Zafman over 10 years ago
- Status changed from In Progress to Resolved
- Translation missing: en.field_remaining_hours set to 0.00
Pull request #546
aba6efda13eb6ab4b96930e9cc2dbddebbe03f26
Updated by Loïc Dachary over 10 years ago
- % Done changed from 0 to 100
- Estimated time set to 0:00 h