Bug #15398
Updated by Loïc Dachary about 8 years ago
h3. workaround
* rm /lib/udev/rules.d/95-ceph-osd.rules
* add ceph-disk activate /dev/sda2 etc. in /etc/rc.local
h3. description
OS:
<pre>
[runsisi@hust ~]$ lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.2.1511 (Core)
Release: 7.2.1511
Codename: Core
</pre>
systemd version:
<pre>
[runsisi@hust ~]$ rpm -qa | grep systemd
systemd-219-19.el7.x86_64
systemd-devel-219-19.el7.x86_64
systemd-libs-219-19.el7.x86_64
systemd-python-219-19.el7.x86_64
systemd-sysv-219-19.el7.x86_64
</pre>
udev version, 219, of course:
<pre>
[runsisi@hust ~]$ udevadm --version
219
</pre>
OSD fs type:
<pre>
[runsisi@hust ~]$ sudo blkid /dev/sdb1
/dev/sdb1: UUID="5a3804e8-275c-4dbb-8481-37235e6fb7fa" TYPE="xfs" PARTLABEL="ceph data" PARTUUID="35be26f8-dae9-421a-9584-5b7132c1089a"
</pre>
*description:*
*1.* during system booting, udev rule for osd device, i.e, /lib/udev/rules.d/95-ceph-osd.rules, invokes /sbin/ceph-disk to activate each newly added osd device, /sbin/ceph-disk mounts the device to a temp location and then re-mount to the correct location, then starts the ceph-osd daemon;
*2.* unfortunately, the mount command executed under the udev (version 219) context always fails to mount the osd device, something needs to be clarified here though: *the mount command exits with 0, but the device is not mounted*;
*3.* what's worse is that the mount command has increased the s_active counter of xfs's in memory superblock instance by one, which prevents the device from being re-parted or any other modifications, e.g,
mount info, nothing for /dev/sdb1:
<pre>
[runsisi@hust ~]$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=98998320k,nr_inodes=24749580,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/hust-root on / type xfs (rw,relatime,attr2,inode64,noquota)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/sda1 on /boot type xfs (rw,relatime,attr2,inode64,noquota)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=19804112k,mode=700,uid=1000,gid=1000)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
</pre>
and kernel threads for /dev/sdb1 still there:
<pre>
[runsisi@hust ~]$ ps aux | grep xfs
root 572 0.0 0.0 0 0 ? S< 14:11 0:00 [xfsalloc]
root 573 0.0 0.0 0 0 ? S< 14:11 0:00 [xfs_mru_cache]
root 574 0.0 0.0 0 0 ? S< 14:11 0:00 [xfslogd]
root 575 0.0 0.0 0 0 ? S< 14:11 0:00 [xfs-data/dm-0]
root 576 0.0 0.0 0 0 ? S< 14:11 0:00 [xfs-conv/dm-0]
root 577 0.0 0.0 0 0 ? S< 14:11 0:00 [xfs-cil/dm-0]
root 579 0.0 0.0 0 0 ? S 14:11 0:02 [xfsaild/dm-0]
root 850 0.0 0.0 0 0 ? S< 14:11 0:00 [xfs-data/sda1]
root 851 0.0 0.0 0 0 ? S< 14:11 0:00 [xfs-conv/sda1]
root 852 0.0 0.0 0 0 ? S< 14:11 0:00 [xfs-cil/sda1]
root 854 0.0 0.0 0 0 ? S 14:11 0:00 [xfsaild/sda1]
root 1764 0.0 0.0 0 0 ? S< 14:12 0:00 [xfs-data/sdc1]
root 1765 0.0 0.0 0 0 ? S< 14:12 0:00 [xfs-conv/sdc1]
root 1766 0.0 0.0 0 0 ? S< 14:12 0:00 [xfs-cil/sdc1]
root 1767 0.0 0.0 0 0 ? S 14:12 0:00 [xfsaild/sdc1]
root 2356 0.0 0.0 0 0 ? S< 15:58 0:00 [xfs-data/sdb1]
root 2357 0.0 0.0 0 0 ? S< 15:58 0:00 [xfs-conv/sdb1]
root 2358 0.0 0.0 0 0 ? S< 15:58 0:00 [xfs-cil/sdb1]
root 2359 0.0 0.0 0 0 ? S 15:58 0:00 [xfsaild/sdb1]
runsisi 2362 0.0 0.0 112648 956 pts/3 S+ 15:58 0:00 grep --color=auto xfs
</pre>
*4.* this issue can be easily reproduced by creating a custom udev rule which mounts the device;
*5.* i have verified that the udev carried with CentOS 7.0.x or 7.1.x (with udev version 208) works like a charm, this issue seems only exists with newer udev.