Ceph » rbd

Here's a short list of smaller tasks I can identify right
now in order to implement this. I may create a new issue
for each, but for now I'm just going to document my list.

- Modify the trail portion of a message so instead of
being a ceph pagelist pointer, it's a "message data"
structure containing one of those pointers.
- Define accessor routines to ensure the trail data is
treated everywhere as an abstraction fully contained
within the messenger code.
- Add a type field to the message data structure, to
indicate how the data is formatted (initially just
a pagelist).
- Abstract the way write_partial_msg_pages() gets the
next page, page offset, and length to send while
processing the tail portion of the message.

At that point I need to do a little more to know the
right way to proceed, but generally:
- Make the shift from the trail being a message data
structure to being a list containing message data
structures.
- Maybe abstract how all data pages (not just the
trail) get selected in write_partial_msg_pages().
- Maybe add to the types supported by the message
data structure, e.g. page array.
- Maybe work on the "out_msg_pos" field in a ceph
connection to reflect these changes.

I've been winding my way through code this morning and
I've discovered there's something in the ceph message
protocol that may interfere with me doing what I had
intended to do.

A ceph message header contains a single "data_off" field
which is an offset into a page into which the received
data is to land. Because this is a message field it
means it is unable to represent a data offset for more
than one hunk of data--it assumes a single block of
data arrives, at the given offset.

I think it may be OK if we can just ensure both sides
of an osd client connection always use 0 for this value
in the message header, and handle the data defined by
individual ops within the osd message in a way that
takes care of whatever page offsets are required for
the type of data being sent/received.

Another thing about that field. In ceph_osdc_build_request(),
the 64-bit "off" value is subject to this:

req->r_request->hdr.data_off = cpu_to_le16(off);

I have no idea what the result (on the osd) is if the offset
is > 65536 bytes (which I am pretty sure it can be). Maybe
the hdr.data_off value is ignored on the osd side?

That field has no semantic meaning. All it does is control the alignment of the memory the OSD reads into, which will let it avoid copying the data (again) to realign if it needs to do direct-io at some point.

Ideally, it would be set so that the largest chunk of data in the request is aligned properly.

That field has no semantic meaning. All it does is
control the alignment of the memory the OSD reads
into, which will let it avoid copying the data (again)
to realign if it needs to do direct-io at some point.

Interesting. Now I understand its purpose, although the
client has to know what the page size is (or at least
what the direct I/O alignment requirements are) of the osd
(server) to actually be of benefit. (64KB should be very
safe; 4KB--and even 512B--might be sufficient.)

It seems to me the ideal case has an offset value of 0,
otherwise read/modify/write is required somewhere.

On the kernel side I think I'll be able to avoid all such
extra copies (on reads anyway). And I think could get rbd
to avoid them on writes as well.

I should have updated this before I guess.

It turns out I implemented something fairly similar to what
was originally described, but I did not use the trail of the
message. After some analysis of how the message was actually
used I learned that it was possible to kill off the trail
entirely and just use a single data item for all types of
data.

So there is a single "message data" structure in a message,
it's just not the trail.

There is a type field, and there are accessor functions
that allow the data transfer functions to treat message
data generically.

The out_msg_pos (and its incoming counterpart) are now gone
in favor of cursors that match the generic data types.

Overall I've arranged things pretty much along the lines
of what I envisioned a month ago.

Another status update, based on the history in this issue.

Sage straightened me out about the "data_off" field. It
is an optimization so that the buffer allocated for received
data can be allocated such that the data lands in a naturally
aligned position, so that large quantities of incoming data
can be submitted to the backing storage using direct I/O
to avoid an extra copy. I haven't changed any of how that
works.

The fact that "only" 16 bits worth of this alignment data
is sent over the wire is just fine.

I still don't like this line of code in ceph_osdc_build_request():
req->r_request->hdr.data_off = cpu_to_le16(off);
because "off" is a 64-bit value.

For the most part though, I would say that I'm satisfied that the
content of this issue has been completed. It's all easily
subsumed by the broader http://tracker.ceph.com/issues/3761.

My plan now is to leave the fix of that silly cpu_to_le16(off)
issue as the final thing to do on this issue, and when that's
done I'll mark it resolved.