Ceph » Linux kernel client

The image im1 is on the osd.And the osd is not install on the disk.
It is install on the /var/cache/local.

[root@song /var/local]# pwd
/var/local
[root@song /var/local]# ls
osd0 osd1 osd2
[root@song /var/local]#

[root@song /var/local]# mount
/dev/mapper/vg_song-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda2 on /boot type ext4 (rw)
/dev/sda1 on /boot/efi type vfat (rw,umask=0077,shortname=winnt)
/dev/mapper/vg_song-lv_home on /home type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
vmware-vmblock on /var/run/vmblock-fuse type fuse.vmware-vmblock (rw,nosuid,nodev,default_permissions,allow_other)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)

Thank you for your reploy!

I think this is relate to the vmvare.I install the same system and ceph in the real machine.

It does not have the problem. Only in my vmvare system have the problem.

Well, that shouldn't be the case. What are the steps to reproduce on your vmware VM?

The below is the operation in my vmware.The same operation is ok on the real machine.

[root@song ~]# rbd ls -p song
im1
im2
im4
im3
im5
im6
[root@song ~]# rbd info song/im6
rbd image 'im6':
size 200 MB in 50 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1a5f2ae8944a
format: 2
features: layering
[root@song ~]# rbd map song/im6
/dev/rbd2
[root@song ~]# rbd unmap /dev/rbd2
rbd: sysfs write failed
rbd: unmap failed: (16) Device or resource busy
[root@song ~]#

i think the device may be being used by multipathd or something else.

lsof /dev/rbdx 

when doing rbd map with a process of multipathd, this issue will come up, with no doubt. The weird thing is both multipathd command and fuser command show no sign of holding rbd.

If you post a reproducer - one that starts with the installation of multipathd package, then sets up multipath and hits this issue, I'll investigate. When rbd unmap hangs, is there anything in /sys/kernel/debug/ceph/<id>/osdc?

Thank you for your answer. It is too long.The prolem above only in that vmware machine can show.
In our oter machine is not show.But thar vmware is not exit.

Ilya Dryomov wrote:

If you post a reproducer - one that starts with the installation of multipathd package, then sets up multipath and hits this issue, I'll investigate. When rbd unmap hangs, is there anything in /sys/kernel/debug/ceph/<id>/osdc?

Surely nothing. this issue bothers me for a log time. But you can modify multipath configure file by adding rbd device to blacklist.

Ilya Dryomov wrote:

If you post a reproducer - one that starts with the installation of multipathd package, then sets up multipath and hits this issue, I'll investigate. When rbd unmap hangs, is there anything in /sys/kernel/debug/ceph/<id>/osdc?

not hangs,but return

rbd: sysfs write failed
rbd: unmap failed: (16) Device or resource busy

ceph zte wrote:

Thank you for your answer. It is too long.The prolem above only in that vmware machine can show.
In our oter machine is not show.But thar vmware is not exit.

multipath can produce the same phenomenon, no matter VM or PM, Really,.

Yeah, sorry - rbd unmap returns -EBUSY, not hangs. So there is nothing in /sys/kernel/debug/ceph/<id>/osdc after it fails?

Ilya Dryomov wrote:

Yeah, sorry - rbd unmap returns -EBUSY, not hangs. So there is nothing in /sys/kernel/debug/ceph/<id>/osdc after it fails?

root@cvknode40:/sys/kernel/debug/ceph/1a016e2d-01d2-4cd3-b83d-872112e35098.client117727# rbd unmap /dev/rbd1
rbd: sysfs write failed
rbd: unmap failed: (16) Device or resource busy
root@cvknode40:/sys/kernel/debug/ceph/1a016e2d-01d2-4cd3-b83d-872112e35098.client117727# cat osdc
root@cvknode40:/sys/kernel/debug/ceph/1a016e2d-01d2-4cd3-b83d-872112e35098.client117727# vi osdc

nothing

There was a similar issue reported on the mailing list - http://www.spinics.net/lists/ceph-devel/msg27435.html. The thread is split between ceph-users and ceph-devel, so it's a bit hard to follow, but there are a couple of links in the linked message that suggest that, at least with ext4, this issue occurs without rbd or multipath being involved. I'm afraid I can't do much more without a reproducer.

Here is related bug in Ubuntu: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1581764

I can reproduce it 100% times with running multipathd

Can you paste your reproducer here, including how you set up multipath and map the rbd image?

• http://paste.openstack.org/show/510493/
• http://paste.openstack.org/show/510494/

Ubuntu 14.04.4 LTS, I'm using OpenStack + Ceph backend with Ceph Devstack plugin

A bit more details about my environment:
$ ceph -v
ceph version 0.94.7 (d56bdf93ced6b80b07397d57e3fa68fe68304432)

1. start multipathd: sudo /etc/init.d/multipath-tools start
2. map image: sudo rbd map -p volumes volume-2e4c8f0f-c088-44c7-bb20-76fce705f58b
3. try to unmap image: sudo rbd unmap -p volumes /dev/rbd2

Well, if multipathd is holding the device, as one of your pastes shows, you won't be able to unmap it - that's expected, at least from the rbd point of view.
What is the output of "multipath -ll" right before the unmap?

'sudo multipath -ll' doesn't show anything at all

What about lsof right before the unmap?

$ sudo lsof /dev/rbd1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
multipath 31388 root 10u BLK 251,0 0t0 773422 /dev/rbd1

This is likely a bug in either multipathd configuration or udev rules. Either way, not an rbd kernel client issue - multipathd shouldn't be holding random devices.
As a workaround, I'd try blacklisting rbd devices with something like

blacklist {
       devnode "^(rbd)[0-9]*" 
}

Thanks, Ilya,

This workaround works for me. So, there is no support for multipath from the rbd side, right?

Depends on what you mean by "support" - rbd CLI tool won't attempt to mess with your multipath settings ;)