Ceph » rbd

NOTE: I adjusted the test to add the sleep 1 until this is fixed.

We now are very careful about allowing an image to be unmapped
if it's still open, whereas previously we would allow an unmap
to proceed even if something still held it open.

It appears that this is happening here--even though the "dd"
command is using direct I/O. (My first thought was to test
using the "oflag=direct" and only then noticed it was already
present in the command...)

Still looking.

OK, I think I understand now.

It turns out that "blkid" is running again on the
device after its final close. This is probably
because the final close may indicate the device's
content has been modified, so blkid is notified so
it can go look at it again and identify its new
format.

So udev is again at fault. But this time, rather
than running "udevadm settle" after mapping the
image to make sure it's ready, we need to run
"udevadm settle before running the unmap command.

In other words, I could run reproduce this problem
by running the commands show above. But with the
"udevadm settle" inserted as shown below I could
run it undefinitely.

So that's the fix. I created http://tracker.ceph.com/issues/4186
to recommend the fix get integrated into the rbd CLI.

rbd create image -s 100
rbd map image
udevadm settle  # note: newer versions of rbd do this for you.
dd if=/dev/zero of=/dev/rbd/rbd/image oflag=direct count=10

udevadm settle    # <-- this is also required

rbd unmap /dev/rbd/rbd/image
rbd rm image