Project

General

Profile

Bug #21335

ceph-disk:osd can not get up after reboot.

Added by linghucong linghucong over 6 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
OSD
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-disk
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

when we use the ceph-disk to deploy osd, it looks ok after the deploying.

but when we reboot the host, all the osds in that host can not set up nomorly.

the ceph verion is 10.2.7

the produce like below:

one hdd disk for ceph data , for nvme disk partion for ceph journal.

the problem look like do with the nvme journal which do not contain the ceph-jounal metadate info like below.

root@ceph3:/var/lib/ceph/osd/ceph-4# uname -a
Linux ceph3 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root@ceph3:/var/lib/ceph/osd/ceph-4# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial

root@ceph3:/var/lib/ceph/osd# ll /dev/sdb
brw-rw---- 1 root disk 8, 16 Sep 11 11:06 /dev/sdb
root@ceph3:/var/lib/ceph/osd# ll /dev/nvme0n1p4
brw-rw---- 1 root root 259, 4 Sep 11 11:00 /dev/nvme0n1p4

root@ceph3:/var/lib/ceph/osd# ceph-disk prepare /dev/sdb /dev/nvme0n1p4
prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data
prepare_device: Journal /dev/nvme0n1p4 was not prepared with ceph-disk. Symlinking directly.
Setting name!
partNum is 0
REALLY setting name!
The operation has completed successfully.
meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=61047597 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0
data = bsize=4096 blocks=244190385, imaxpct=25 = sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=119233, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully

root@ceph3:/var/lib/ceph/osd# ceph-disk activate /dev/sdb1
got monmap epoch 1
added key for osd.4

the ceph-disk do not have any error in the above operation, but the osd can not set up because the jounal permission issue.

2017-09-11 11:10:43.798339 7efea25c58c0 -1 filestore(/var/lib/ceph/osd/ceph-4) mount failed to open journal /var/lib/ceph/osd/ceph-4/journal: (13) Permission denied
2017-09-11 11:10:43.798623 7efea25c58c0 -1 osd.4 0 OSD:init: unable to mount object store
2017-09-11 11:10:43.798646 7efea25c58c0 -1 [[0;31m ** ERROR: osd init failed: (13) Permission denied[[0m

then i do the below opration:
root@ceph3:/var/lib/ceph/osd/ceph-4# chown ceph:ceph /dev/nvme0n1p4
root@ceph3:/var/lib/ceph/osd/ceph-4# systemctl restart
ceph 10394 1 3 11:14 ? 00:00:01 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph

*this time everything looks all ok, the osd can nomal run. but if we reboot the host, the osd can not setup normally

i think this maybe a bug for the ceph-disk, but i am not sure if this bug had fixed?

it look like the jounal info for the nvme do not get write like below:*

root@ceph3:/var/lib/ceph/osd/ceph-4# sgdisk -i 4 /dev/nvme0n1
Partition GUID code: 0FC63DAF-8483-4772-8E79-3D69D8477DE4 (Linux filesystem)
Partition unique GUID: C72B3719-1AC9-4FE0-A0F0-2B99FD0C9899
First sector: 229443584 (at 109.4 GiB)
Last sector: 271386623 (at 129.4 GiB)
Partition size: 41943040 sectors (20.0 GiB)
Attribute flags: 0000000000000000
Partition name: ''

waiting for your reploy think you!

History

#1 Updated by linghucong linghucong over 6 years ago

I have see the ceph-disk code. It is because if the disk partion not prepare by ceph-disk,

just use the link. This is ok, but should we also need to check whether this disk have the

ceph jounal type and jounal uuid ?

LOG.warning('%s %s was not prepared with '
'ceph-disk. Symlinking directly.',
self.name.capitalize(),
getattr(self.args, self.name))
self.space_symlink = getattr(self.args, self.name)
return

#2 Updated by Sage Weil almost 3 years ago

  • Status changed from New to Closed

Also available in: Atom PDF