Ceph

I've pushed the alternative fix in the same pull req.

I'm able to reproduce that frequently by running

sudo test/ceph-disk.sh test_activate_journal_dev 

I think what happens is the following sequence

• partition 1 is created
• partprobe called so the kernel notices (the symlink shows)
• partition table is zapped and symlink removed
• partproble is not called
• partition 1 is created
• partprobe called but the kernel thinks partition already exists and does not trigger an udev event that does not create the symlink

the only way to reset the idea the kernel has about a given device is to zap the partition table + partprobe. I think symlinks are created reliably and it does not depend on the machine load.

Hi Loic,
In this case, ceph-disk zap doesn't apply. The use-case is that you have say 4-5 partitions on a shared journal SSD, and you want to re-use only one of those partitions to become the journal for a new OSD. So we don't (and mustn't!) call ceph-disk zap on the SSD device in this case.

Instead we tried changing the guid of the journal partition, but that doesn't trigger udev reliably. So the only reliable method is not to change the guid, as in the current pull req.
Cheers, Dan

Dan van der Ster wrote:

Instead we tried changing the guid of the journal partition, but that doesn't trigger udev reliably. So the only reliable method is not to change the guid, as in the current pull req.

I'm under the impression (although I've never actually tried to prove it) that the udev event will never be called if the guid is modified, even though it makes the by-partuuid symlink obsolete. Not changing the guid (which is what you implemented at https://github.com/ceph/ceph/commit/29eb1350b4acaeabfe1d2b19efedbce22641d8cc ) works around the problem.

This should probably be a bug report against udev ?

Loic Dachary wrote:

I'm under the impression (although I've never actually tried to prove it) that the udev event will never be called if the guid is modified, even though it makes the by-partuuid symlink obsolete.

On our test cluster changing the guid did trigger udev and make the correct by-partuuid link (and also left behind the old link -- no big deal). On our prod cluster changing the guid did not trigger udev. Both are CentOS 6.6. The only difference is the activity on the journal devs.

Ok. Have you ever seen a problem under load where udev would fail to notice the creation / removal of a partition although partprobe / partx is called consistently (i.e. after each creation / removal ) ?

For OSD devices (one dev -- one OSD), I haven't observed any problems, regardless of load. (In this case, the OSD process is not running, so no processes have the device open when the partition is removed or created).

For journal devices, it really depends. Removing 1 out of 5 journal partitions (when the other 4 are still used by active OSDs) is not really doable, in my experience with CentOS 6. I never found the combination of partprobe / partx that made the OSD realize the new ptable. The only reliable method to remove/recreate partitions on a shared journal dev was to stop all OSDs using that dev, then adjust the ptable, then restart the OSDs.

Thanks for explaining, that makes me slightly less worried about the reliability of the partition/udev notification couple in the most common case :-)