Ceph » rbd

Adding two things:
- this occurred during test 190 of the third consecutive pass
of xfstests with this in the teuthology yaml definition:

- workunit:
    clients:
      all:
        - rbd/map-unmap.sh
        - rbd/kernel.sh
        - misc/trivial_sync.sh
        - suites/blogbench.sh
        - suites/dbench.sh
        - suites/tiobench.sh
        - suites/fsstress.sh
        - kernel_untar_build.sh
- rbd.xfstests:
    client.0:
        count: 3

- At some point--possibly at the point this failure
occurred--a VPN server was restarted. I don't have
evidence that they're related but the failure and
the VPN restart happened within the same one hour
time window, so they could be.

I've decoded the osd request that's been provided to
rbd_osd_req_callback(). Its contents look completely
legitimate. Its r_priv pointer, which should be a
(struct obj_request *) does not contain an object
request, however. This means the object request has
probably been freed.

This osd request is marked for linger. That means the
osd client will hang onto a copy of it until rbd
unregisters it.

So I think that may point out the problem, and it could
well have arisen because of the network blip.

(I'm not completely sure about this, but here's the
theory...)

I think that the rbd device needs to take an extra
reference to this osd linger request to make sure it
hangs around until it gets unregistered. Otherwise
as soon as the request marked for linger completes,
it will be destroyed, and if the osd client indicates
it completes again the r_priv pointer will be invalid.

The thing I'm not sure about is whether or when or why
the osd client would complete the same osd request
more than once.

I'm going to look at that a bit now.

I have confirmed that every time a request registered to linger
is re-submitted the osd client will call the callback function
for that request. This was not an issue with the previous
request code, because watch requests were synchronous (with
respect to the osd client) and thus had no osd client callback
function.

The new code always has a callback, so we need to handle it.

The fix is to add a new reference to the object request every
time a CEPH_OSD_OP_WATCH request completes, except if the
request was issued to cancel the watch request. That way
the object request will stay around, but when the watch
request gets unregistered it will go away.

I have implemented this (complete with a big comment that
explains the situation) and now need to test it.

OK, with Josh's help I finally managed to reproduce the
problem intentionally to check my fix.

I'm building it now (along with a few other patches to
make the change easier to follow).

I guess I'll post it for review with a caveat that it
follows on the new request code.

I've opened a new issue that has symptoms similar to this
but not identical:
http://tracker.ceph.com/issues/3950

In the current case, the osd request seems fine, but its
osd_req->priv pointer does not seem to point to a valid
object request pointer. That suggests the object request
has been freed.

In that (3950) case, the object request pointer seems to
be valid, but its osd_req pointer does not refer back to
the osd request.

This doesn't rule out that these are slightly different
symptoms of the same problem. (But I'm running the test
with what's supposed to be a fix in place for the present
problem...)