Project

General

Profile

Bug #16351

jewel : 60-ceph-partuuid-workaround-rules still needed by debian jessie (udev 215-17)

Added by alexandre derumier almost 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
06/16/2016
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
jewel
Regression:
Yes
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

associated mail thread

Hi,
since this commit
https://github.com/ceph/ceph/commit/9f77244b8e0782921663e52005b725cca58a8753

osd disk are not mounted anymore on debian jessie ((udev 215-17) at boot

I have looked at udev log, at the 95-ceph-osd.rules is never triggered

if I add the old 60-ceph-partuuid-workaround-rules, it's working fine

ceph-disk.strace.sde1.4787.gz - /bin/systemctl --no-block restart ceph-disk@dev/sde1.service (8.77 KB) Denis Pudov, 08/05/2016 03:12 PM

ceph-disk.strace.sde1.4545.gz - /usr/sbin/ceph-disk --log-stdout -v trigger /dev/sde1 (142 KB) Denis Pudov, 08/05/2016 03:13 PM


Related issues

Copied to Ceph - Backport #16577: jewel : 60-ceph-partuuid-workaround-rules still needed by debian jessie (udev 215-17) Resolved

History

#1 Updated by Loic Dachary almost 3 years ago

  • Assignee set to Loic Dachary
  • Priority changed from Normal to Urgent

#2 Updated by Loic Dachary almost 3 years ago

  • Description updated (diff)
  • Status changed from New to In Progress

#3 Updated by Daniel Swarbrick almost 3 years ago

I ran into this bug also, and can confirm that re-adding the old 60-ceph-partuuid-workaround-rules fixes the problem.

#4 Updated by Loic Dachary almost 3 years ago

On Jessie 8.5, the rules work as expected when removing/adding a disk

udevadm monitor
...
KERNEL[399.124705] add      /devices/pci0000:00/0000:00:07.0 (pci)
KERNEL[399.128475] add      /devices/pci0000:00/0000:00:07.0/virtio3 (virtio)
UDEV  [399.131617] add      /devices/pci0000:00/0000:00:07.0 (pci)
KERNEL[399.132247] add      /devices/virtual/bdi/254:16 (bdi)
UDEV  [399.132499] add      /devices/pci0000:00/0000:00:07.0/virtio3 (virtio)
UDEV  [399.133016] add      /devices/virtual/bdi/254:16 (bdi)
KERNEL[399.142008] add      /devices/pci0000:00/0000:00:07.0/virtio3/block/vdb (block)
KERNEL[399.142033] add      /devices/pci0000:00/0000:00:07.0/virtio3/block/vdb/vdb1 (block)
KERNEL[399.142045] add      /devices/pci0000:00/0000:00:07.0/virtio3/block/vdb/vdb2 (block)
UDEV  [399.526162] add      /devices/pci0000:00/0000:00:07.0/virtio3/block/vdb (block)
UDEV  [403.304562] add      /devices/pci0000:00/0000:00:07.0/virtio3/block/vdb/vdb2 (block)
UDEV  [403.395816] add      /devices/pci0000:00/0000:00:07.0/virtio3/block/vdb/vdb1 (block)

and
ceph       938  6.4  0.4 831352 32052 ?        Ssl  13:50   0:00 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph

But it does not work when rebooting. Trying to figure out why. The fact that adding 60-ceph-partuuid-workaround-rules fixes the issue is helpful.

#5 Updated by Loic Dachary almost 3 years ago

For the record, it works on CentOS 7.2 but a user reported it failed in the same way on CentOS 7 (maybe an older version) ?

#7 Updated by Loic Dachary almost 3 years ago

The CentOS failure happens when the root file system is on LVM. When the root file system is on a regular partition, the OSD shows up as expected.

#8 Updated by alexandre derumier almost 3 years ago

Loic Dachary wrote:

The CentOS failure happens when the root file system is on LVM. When the root file system is on a regular partition, the OSD shows up as expected.

on my debian jessie setup, it is without lvm

/etc/fstab
UUID=a9309e01-0f80-475b-95e1-33a9ddb4bdff / xfs defaults 0 1

#9 Updated by Loic Dachary almost 3 years ago

  • Status changed from In Progress to Need Review

#10 Updated by Loic Dachary almost 3 years ago

  • Backport set to jewel

#11 Updated by Loic Dachary over 2 years ago

  • Status changed from Need Review to Pending Backport

#12 Updated by Loic Dachary over 2 years ago

  • Copied to Backport #16577: jewel : 60-ceph-partuuid-workaround-rules still needed by debian jessie (udev 215-17) added

#13 Updated by Denis Pudov over 2 years ago

alexandre derumier wrote:

associated mail thread

Hi,
since this commit
https://github.com/ceph/ceph/commit/9f77244b8e0782921663e52005b725cca58a8753

osd disk are not mounted anymore on debian jessie ((udev 215-17) at boot

I have looked at udev log, at the 95-ceph-osd.rules is never triggered

On my Jessie (8.5) with 4.6.0 backports kernel its triggered.
I have added strace to ceph-disk at the 95-ceph-osd.rules:

RUN+="/usr/bin/strace -tt -s 1024 -o /var/tmp/ceph-disk.strace.$name -ff /usr/sbin/ceph-disk --log-stdout -v trigger /dev/$name" 

and got four strace output per device.

At the stage of calling trigger (/bin/systemctl --no-block restart ceph-disk@dev/sde1.service) get an error from systemd (full strace at attach):

org.freedesktop.DBus.Error.InvalidArgs.*sysinit.target is queued, ignoring restart request for unit ceph-disk@dev-sde1.service

if I add the old 60-ceph-partuuid-workaround-rules, it's working fine

For me it's not fine... systemd tries many times to run ceph-osd@.service and eventually get:

авг 05 18:00:18 storage-ceph-01y systemd[1]: ceph-osd@58.service: main process exited, code=exited, status=1/FAILURE
авг 05 18:00:18 storage-ceph-01y systemd[1]: Unit ceph-osd@58.service entered failed state.
авг 05 18:00:18 storage-ceph-01y systemd[1]: ceph-osd@58.service start request repeated too quickly, refusing to start.
авг 05 18:00:18 storage-ceph-01y systemd[1]: Failed to start Ceph object storage daemon.

#15 Updated by Denis Pudov over 2 years ago

Yes, i have added 60-ceph-partuuid-workaround-rules to /lib/udev/rules.d/ and got mounts but have problems with start ceph-osd@.service.

#16 Updated by alexandre derumier over 2 years ago

The fix is
60-ceph-by-parttypeuuid.rules
https://github.com/dachary/ceph/blob/35004a628b2969d8b2f1c02155bb235165a1d809/udev/60-ceph-by-parttypeuuid.rules

not old
60-ceph-partuuid-workaround-rules

#17 Updated by Denis Pudov over 2 years ago

Alexandre, thank you!
60-ceph-by-parttypeuuid.rules works!

#18 Updated by Loic Dachary over 2 years ago

Additional pull request, for the record: https://github.com/ceph/ceph/pull/10115

#19 Updated by Loic Dachary over 2 years ago

  • Status changed from Pending Backport to Resolved

#20 Updated by joke lee over 2 years ago

it's still not mount osd after reboot when the osd disk is raid0

Also available in: Atom PDF