Ceph » RADOS

Copied from https://tracker.ceph.com/issues/50169

Using Ubuntu 20.04, none cephadm, packages from ceph repositories.

Upgraded from latest Octopus to Pacific.

After the upgrade radosgw AND osds do not start with permission errors. There are multiple users with the same issues on ceph-users mailinglist.

for OSDs:

2021-04-06T11:27:23.402+0200 7f527b714f00 0 osd.20:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2021-04-06T11:27:23.402+0200 7f527b714f00 1 bdev(0x55815696a400 /var/lib/ceph/osd/ceph-20/block) open path /var/lib/ceph/osd/ceph-20/block
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 bdev(0x55815696a400 /var/lib/ceph/osd/ceph-20/block) open open got: (1) Operation not permitted
2021-04-06T11:27:23.402+0200 7f527b714f00 0 osd.20:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (1) Operation not permitted
2021-04-06T11:27:23.402+0200 7f527b714f00 1 bluestore(/var/lib/ceph/osd/ceph-20) _mount path /var/lib/ceph/osd/ceph-20
2021-04-06T11:27:23.402+0200 7f527b714f00 0 bluestore(/var/lib/ceph/osd/ceph-20) _open_db_and_around read-only:0 repair:0
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (1) Operation not permitted
2021-04-06T11:27:23.402+0200 7f527b714f00 1 bdev(0x55815696a400 /var/lib/ceph/osd/ceph-20/block) open path /var/lib/ceph/osd/ceph-20/block
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 bdev(0x55815696a400 /var/lib/ceph/osd/ceph-20/block) open open got: (1) Operation not permitted
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 osd.20 0 OSD:init: unable to mount object store
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 ^{[[0;31m ** ERROR: osd init failed: (1) Operation not permitted}[[0m

permissions to block etc. are all correct. Utilizing manual start of the osd with as example:

/usr/bin/ceph-osd -f --cluster ceph --id 20 --setuser ceph --setgroup ceph

starts the OSD just fine. But via systemctl it is not working.

Comments from original bug:
玮文 胡 wrote:

I believe RGW and OSD have different issues and should be filed seperately.

Links to mailing-list threads:
RGW: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZQNA65QMUOPJKBPNLHAOSI5W4BXYWC73/
OSD: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/TNBGGNN6STGDKARAQTQCIPTU4KLIVJQV/

Enrico Kern wrote:

I can confirm that setting

rgw_data_log_backing = omap

is resolving the radosGW startup issues. I coupled them together here as both show the same problems (permission denied). Could create a seperate one about the OSD issue.