Project

General

Profile

Bug #50682

Pacific - OSD not starting after upgrade

Added by Greg Farnum over 1 year ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Copied from https://tracker.ceph.com/issues/50169

Using Ubuntu 20.04, none cephadm, packages from ceph repositories.

Upgraded from latest Octopus to Pacific.

After the upgrade radosgw AND osds do not start with permission errors. There are multiple users with the same issues on ceph-users mailinglist.

for OSDs:

2021-04-06T11:27:23.402+0200 7f527b714f00 0 osd.20:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2021-04-06T11:27:23.402+0200 7f527b714f00 1 bdev(0x55815696a400 /var/lib/ceph/osd/ceph-20/block) open path /var/lib/ceph/osd/ceph-20/block
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 bdev(0x55815696a400 /var/lib/ceph/osd/ceph-20/block) open open got: (1) Operation not permitted
2021-04-06T11:27:23.402+0200 7f527b714f00 0 osd.20:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (1) Operation not permitted
2021-04-06T11:27:23.402+0200 7f527b714f00 1 bluestore(/var/lib/ceph/osd/ceph-20) _mount path /var/lib/ceph/osd/ceph-20
2021-04-06T11:27:23.402+0200 7f527b714f00 0 bluestore(/var/lib/ceph/osd/ceph-20) _open_db_and_around read-only:0 repair:0
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (1) Operation not permitted
2021-04-06T11:27:23.402+0200 7f527b714f00 1 bdev(0x55815696a400 /var/lib/ceph/osd/ceph-20/block) open path /var/lib/ceph/osd/ceph-20/block
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 bdev(0x55815696a400 /var/lib/ceph/osd/ceph-20/block) open open got: (1) Operation not permitted
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 osd.20 0 OSD:init: unable to mount object store
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 [[0;31m ** ERROR: osd init failed: (1) Operation not permitted[[0m

permissions to block etc. are all correct. Utilizing manual start of the osd with as example:

/usr/bin/ceph-osd -f --cluster ceph --id 20 --setuser ceph --setgroup ceph

starts the OSD just fine. But via systemctl it is not working.

Comments from original bug:
玮文 胡 wrote:

I believe RGW and OSD have different issues and should be filed seperately.

Links to mailing-list threads:
RGW: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZQNA65QMUOPJKBPNLHAOSI5W4BXYWC73/
OSD: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/TNBGGNN6STGDKARAQTQCIPTU4KLIVJQV/

Enrico Kern wrote:

I can confirm that setting

rgw_data_log_backing = omap

is resolving the radosGW startup issues. I coupled them together here as both show the same problems (permission denied). Could create a seperate one about the OSD issue.


Related issues

Related to Ceph - Bug #50347: systemd: `ceph-osd@.service` Failed with `ProtectClock=true` Resolved
Copied from rgw - Bug #50169: Pacific - RadosGW not starting after upgrade Resolved

History

#1 Updated by Greg Farnum over 1 year ago

  • Copied from Bug #50169: Pacific - RadosGW not starting after upgrade added

#2 Updated by Neha Ojha over 1 year ago

  • Related to Bug #50347: systemd: `ceph-osd@.service` Failed with `ProtectClock=true` added

#3 Updated by Neha Ojha over 1 year ago

This issue has been fixed by https://github.com/ceph/ceph/pull/40845 and will be released in the next pacific point release.

Also available in: Atom PDF