Bug #24903: Update 12.2.5 -> 12.2.6: block.db symlink exists but target unusable - bluestore - Ceph

Actions

Copy link

Bug #24903

closed

Update 12.2.5 -> 12.2.6: block.db symlink exists but target unusable

Added by Robert Sander almost 6 years ago. Updated over 5 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Target version:

% Done:

Source:

Community (user)

Tags:

Backport:

mimic,luminous

Regression:

Severity:

4 - irritation

Reviewed:

Affected Versions:

Ceph - v12.2.6

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

After updating from 12.2.5 to 12.2.6 BlueStore OSDs with a separate blocks.db device will not restart:

2018-07-13 12:01:56.916651 7fe93f2f4e40 0 set uid:gid to 64045:64045 (ceph:ceph)
2018-07-13 12:01:56.916690 7fe93f2f4e40 0 ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable), process ceph-osd, pid 107609
2018-07-13 12:01:56.926350 7fe93f2f4e40 0 pidfile_write: ignore empty --pid-file
2018-07-13 12:01:56.934142 7fe93f2f4e40 0 load: jerasure load: lrc load: isa
2018-07-13 12:01:56.934273 7fe93f2f4e40 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel
2018-07-13 12:01:56.934293 7fe93f2f4e40 1 bdev(0x55adeed706c0 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block
2018-07-13 12:01:56.934633 7fe93f2f4e40 1 bdev(0x55adeed706c0 /var/lib/ceph/osd/ceph-6/block) open size 107369988096 (0x18ffc00000, 102396 MB) block_size 4096 (4096 B) rotational
2018-07-13 12:01:56.934882 7fe93f2f4e40 1 bluestore(/var/lib/ceph/osd/ceph-6) _set_cache_sizes max 0.5 < ratio 0.99
2018-07-13 12:01:56.934915 7fe93f2f4e40 1 bluestore(/var/lib/ceph/osd/ceph-6) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
2018-07-13 12:01:56.934923 7fe93f2f4e40 1 bdev(0x55adeed706c0 /var/lib/ceph/osd/ceph-6/block) close
2018-07-13 12:01:57.215780 7fe93f2f4e40 1 bluestore(/var/lib/ceph/osd/ceph-6) _mount path /var/lib/ceph/osd/ceph-6
2018-07-13 12:01:57.216296 7fe93f2f4e40 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel
2018-07-13 12:01:57.216308 7fe93f2f4e40 1 bdev(0x55adeed70d80 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block
2018-07-13 12:01:57.216967 7fe93f2f4e40 1 bdev(0x55adeed70d80 /var/lib/ceph/osd/ceph-6/block) open size 107369988096 (0x18ffc00000, 102396 MB) block_size 4096 (4096 B) rotational
2018-07-13 12:01:57.217362 7fe93f2f4e40 1 bluestore(/var/lib/ceph/osd/ceph-6) _set_cache_sizes max 0.5 < ratio 0.99
2018-07-13 12:01:57.217395 7fe93f2f4e40 1 bluestore(/var/lib/ceph/osd/ceph-6) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
2018-07-13 12:01:57.218191 7fe93f2f4e40 -1 bluestore(/var/lib/ceph/osd/ceph-6) _open_db /var/lib/ceph/osd/ceph-6/block.db symlink exists but target unusable: (13) Permission denied
2018-07-13 12:01:57.218786 7fe93f2f4e40 1 bdev(0x55adeed70d80 /var/lib/ceph/osd/ceph-6/block) close
2018-07-13 12:01:57.498694 7fe93f2f4e40 -1 osd.6 0 OSD:init: unable to mount object store
2018-07-13 12:01:57.498719 7fe93f2f4e40 -1 ESC[0;31m ** ERROR: osd init failed: (13) Permission deniedESC[0m

The block.db symlink points to /dev/sdc1:

root@ceph04:/var/lib/ceph/osd/ceph-6# ls -l
total 24
lrwxrwxrwx 1 ceph ceph 93 Jul 11 12:14 block -> /dev/ceph-4926a944-c779-4154-a667-4b7be995faed/osd-block-cf3a261f-86ae-45b5-b29a-558070a139aa
lrwxrwxrwx 1 root root 9 Jul 11 12:14 block.db -> /dev/sdc1

this device is owned by ceph:

root@ceph04:/var/lib/ceph/osd/ceph-6# ls l /dev/sdc1
brw-rw--- 1 ceph ceph 8, 33 Jul 13 11:55 /dev/sdc1

I have to reboot the node to resolve this issue.

Actions

Copy link

Updated by Josh Durgin almost 6 years ago

Priority changed from Normal to Urgent

Actions

Copy link

Updated by Alfredo Deza almost 6 years ago

What is happening here is that ceph-volume was relying on bluestore to set these links, and then when bluestore changed, a code path in ceph-volume had to enable these links and
set the correct permissions.

However, ceph-volume was not using chown with the flag that also includes updating the permissions on the target as well as the underlying devices (see https://github.com/ceph/ceph/pull/22716 )

The workaround for this problem is one of these:

manually `chown -h ceph:ceph /var/lib/ceph/osd/ceph-6/block`
reboot the whole server (ceph-volume will take care of this automatically)
stop the osd, unmount /var/lib/ceph/osd/ceph-6, run `ceph-volume lvm activate --all`

Actions

Copy link