Ceph » Linux kernel client

There is an ugly reference counting dance that occurs with bio
pointers in the kernel osd I/O path, and it needs to get fixed.

The rbd kernel module uses bio's for I/O (the ceph file system
does not). It takes incoming I/O requests and builds a list
of bio's that describe the portion of an I/O operation that
involves a particular osd object. Each bio on the list has
a reference, and when the request to an osd completes, rbd
will call bio_chain_put() (an rbd function) to drop the
reference for each.

The rbd module will record the head of one of these bio lists
in an osd request it builds up. The libceph kernel module
will later copy that pointer from the incoming osd request to
the ceph message that will actually transfer the data over
the wire (in ceph_osdc_start_request()). The osd request in
this case is reference counted, and the initiator will own the
reference that was supplied when the osd request was created.

The rbd code is expected to have taken an extra reference to
he head bio (only--not any of the others in the list). This
reference apparently is taken on behalf of the osd request.

When the osd request message actually starts getting sent
over the wire (in ceph_osdc_start_request()), the bio pointer
from the request is copied to the outgoing message. (This
is not necessarily a problem.) Eventually the message will
get sent successfully, and a response to that message will
arrive.

When the response arrives, handle_response() will end up
calling the r_callback routine recorded for the request, or
if none, will signal the completion of the message, which
will indicate to a caller of ceph_osdc_wait_request() that
the message is complete. In either case, after handling
that completion, ceph_osdc_put_request() is expected to
be called in order to drop the reference to the osd request.

And when the last reference is dropped, ceph_osd_put_request()
will cause ceph_osdc_release_request() to be called, and that
calls bio_put() on the bio pointer in the osd request to drop
the extra reference the initiator was expected to take.

OK. So what are the problems here?
- Only the first bio in the chain has its reference dropped.
If there happened to be multiple bio's, the second and
subsequent bio's could have already been cleaned up when
rbd calls bio_chain_put().
- The rbd kernel module is responsible for taking the
reference to the bio which the libceph kernel module is
responsible for dropping. This is manageable, but not
a good design.

The first problem will be tracked here:
http://tracker.newdream.net/issues/3798

The present issue will be used to track the second problem.
To fix this, that extra reference taken for the bio (list)
should be taken by libceph, in ceph_osdc_start_request().