Ceph » rbd

The new code for tracking rbd requests works, but
there are some error paths that still need to be
fleshed out.

Here are some specific cases:
- The (new) function rbd_obj_request_submit() calls
  ceph_osdc_start_request().  The cleanup for the
  case where that returns an error needs to be done.
- The (new) function rbd_image_request_submit() calls
  rbd_obj_request_submit(), and if the latter returns
  an error it is not yet properly handled.

There are a few other outstanding questions I've got
in the new code (some of which probably also apply on
the current/"old" code):
- If ceph_osd_wait_request() returns an error, is the
  reassert_version value returned in the osd request
  valid?
- When an osd request response indicates an error, is its
  reassert_version value valid?
- When an osd read request returns with an error, what
  is the meaning of the length field?  Is it valid?
- When an osd read request returns, I would like to
  validate some fields that are not now touched,
  such as the offset, payload_len, truncate_size and
  truncate_seq.
- Similarly, on a write request I'd like to perform
  whatever validation I can on the response.
- What happens if the length in a response to a
  write request is less than requested?  Does that
  indicate a zero-fill to the end of the original
  length should be done?
- There are some special zero-length requests (like
  flush).  We should deal with those.

I did some testing yesterday and found that I got I/O errors
while running xfstests. This was unexpected; I thought this
code was nearly done (and I did test it before).

I made a few minor changes, and have run tests again. This time
I got through xfstest 075 without error.

HOWEVER I just got a crash. First I got a warning in the
rcu code (weird), but that was followed less than a second later
by a crash in kmem_cache_alloc(). Based on this I suspect there's
a memory allocation/free problem somewhere.

Still looking.

I am leaving shortly for a few hours. In reviewing this
new code I find a few things that make it a little hard
make sure certain things are getting allocated and freed
at all the right times. I have some changes I'm working
through that will improve on that, but I still need to
look them over a bit more before testing with them.

Basically they involve replacing an array of object request
structures with an array of pointers to them. That makes
it easier to get rid of object requests when their last
reference is dropped. It also will make it much more
natural to have object requests (such as rbd image header
reads) stand on their own, without being connected to
an rbd image request (which is used for I/O).

I'll continue tonight and tomorrow morning.

My full test run isn't complete but I seem to have resolved
whatever problem I was hitting yesterday. I have not yet
completed the switchover to using pointers instead of structs
in the image request structure; I'm still finishing that up.

Unfortunately my system crashed after an hour or so. The
crash was in the network driver, and a little analysis
suggested it could be some sort of weird bug, so I restarted
to try again.

This time, however, I'm looking at the slab allocation stats
and it looks like there's a memory leak, at least in the
slab named "kmalloc-192", where I show over 5 million objects
and growing right now. That might be a hint at the source
of the leak. (A ceph_msg_header is 53 bytes; a ceph_msg is
208 bytes, and so on.) I'm turning on some memory debugging
I put together last month.

For some reason my tests started hanging on Friday when
I added memory debug code for catching leaks and reuses.
I went through many iterations trying to figure out what
caused this, and was surprised that even what I thought
was working was also hanging. This morning I learned
that something in the ceph "next" branch had caused this.

I think I'm back to working with what was my best and
latest version of the code before this occurred now, but
I'm a little afraid I may have back-tracked. This episode
has been quite unhelpful.

I finally looked at the memory leak test results though
and find that there is only one spot that seems to be
leaking, and it's only 24 bytes, and is not something that
happens a lot (it's the snapshot context).

This process produces a lot of output on the console so
I'm trying not to run too many tests, in order to get
complete information about allocations and frees. I
may have to do more though.

I spent the day trying to find the memory leak and finally
found it. The structure being leaked was a bio. It was
not caught by my kmalloc debug code because these are
allocated using bio_alloc(), which won't use my clever
macros.

I noticed that the "kmalloc-192" slab cache was growing
larger and larger over time. So I wanted to look at
what was consuming that slab cache.

The way I found it was through miring through raw memory
using kdb.
- I looked through all the kmem caches until I found the
data structure describing "kmalloc-192"
- I figured out that the per-node partial list linked
together a set of page structures on their lru links.
- I learned that for pages in a slab cache, the third
8-byte value in the page structure points to the
first free object in the page. That gave me a pointer
within the page.
- I looked at the 192-byte chunks of that page, and
after some head scratching and heavy duty looking
through data structures, figured out that the most
common thing there looked like a struct bio.

Given that, I looked at the rbd code and, sure enough,
each object request was given a bio structure (or list
of them) but it was not freed when the object request
completed.

I'm waiting for a build to complete so I can test it.

OK, I ran a test and got a crash. The bio built for
an object request gets handed off to an osd request.
I need to work through the ownership life cycle for
that.

But not tonight...

I found the source of my trouble, and in the process understood
a little more about some subtlety in bio reference counting
that exists between the osd client (in the libceph kernel
module) and rbd. The rbd client needs to take a reference
to the first bio in the list for any bio list it passes in
an osd request. Later, the osd client is responsible for
dropping that reference. I created a few new tracker entries
to suggest improving that.

The leak was due to resetting the osd request to NULL before
calling ceph_osdc_put_request(), which prevented that function
from dropping the bio reference.

My testing on this code is nearly complete. However, I'm going
to hold off on pushing this (along with the changes associated
with 3754, 3755 and 3877) to the testing branch for two reasons:
- I just updated the testing branch to be based on a much newer
version of the Linux kernel, and I'd like to give that change
alone one or more passes through the nightly test suite before
adding more to it.
- There are a few more known problems (3940, 3937, 3427, and
possibly 3950) that I'd like to get reviewed and committed
before I push them to the testing branch.