Ceph » rgw

Using Ubuntu 20.04, none cephadm, packages from ceph repositories.

Upgraded from latest Octopus to Pacific.

After the upgrade radosgw AND osds do not start with permission errors. There are multiple users with the same issues on ceph-users mailinglist.

for OSDs:

2021-04-06T11:27:23.402+0200 7f527b714f00 0 osd.20:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2021-04-06T11:27:23.402+0200 7f527b714f00 1 bdev(0x55815696a400 /var/lib/ceph/osd/ceph-20/block) open path /var/lib/ceph/osd/ceph-20/block
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 bdev(0x55815696a400 /var/lib/ceph/osd/ceph-20/block) open open got: (1) Operation not permitted
2021-04-06T11:27:23.402+0200 7f527b714f00 0 osd.20:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (1) Operation not permitted
2021-04-06T11:27:23.402+0200 7f527b714f00 1 bluestore(/var/lib/ceph/osd/ceph-20) _mount path /var/lib/ceph/osd/ceph-20
2021-04-06T11:27:23.402+0200 7f527b714f00 0 bluestore(/var/lib/ceph/osd/ceph-20) _open_db_and_around read-only:0 repair:0
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-20/block: (1) Operation not permitted
2021-04-06T11:27:23.402+0200 7f527b714f00 1 bdev(0x55815696a400 /var/lib/ceph/osd/ceph-20/block) open path /var/lib/ceph/osd/ceph-20/block
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 bdev(0x55815696a400 /var/lib/ceph/osd/ceph-20/block) open open got: (1) Operation not permitted
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 osd.20 0 OSD:init: unable to mount object store
2021-04-06T11:27:23.402+0200 7f527b714f00 -1 ^{[[0;31m ** ERROR: osd init failed: (1) Operation not permitted}[[0m

permissions to block etc. are all correct. Utilizing manual start of the osd with as example:

/usr/bin/ceph-osd -f --cluster ceph --id 20 --setuser ceph --setgroup ceph

starts the OSD just fine. But via systemctl it is not working.

for RGW:

2021-04-06T15:39:14.923+0200 7efe51a9cb40 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process radosgw, pid 2560132
2021-04-06T15:39:14.923+0200 7efe51a9cb40 0 framework: beast
2021-04-06T15:39:14.923+0200 7efe51a9cb40 0 framework conf key: port, val: 7480
2021-04-06T15:39:14.923+0200 7efe51a9cb40 1 radosgw_Main not setting numa affinity
2021-04-06T15:39:15.107+0200 7efe51a9cb40 -1 int rgw::cls::fifo::get_meta(librados::v14_2_0::IoCtx&, const string&, std::optional<rados::cls::fifo::objv>, rados::cls::fifo::info*, uint32_t*, uint32_t*, uint64_t, optional_yield, bool):105 fifo::op::GET_META failed r=-1 tid=0
2021-04-06T15:39:15.107+0200 7efe51a9cb40 -1 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, bool):884 get_meta failed: r=-1
2021-04-06T15:39:15.107+0200 7efe51a9cb40 -1 static int RGWDataChangesFIFO::exists(ceph::common::CephContext*, librados::v14_2_0::Rados*, const rgw_pool&, bool*, bool*): unable to open FIFO: default.rgw.log/data_log.0: (1) Operation not permitted
2021-04-06T15:39:15.107+0200 7efe51a9cb40 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when checking for existing FIFO datalog backend: (1) Operation not permitted
2021-04-06T15:39:15.107+0200 7efe51a9cb40 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-1
2021-04-06T15:39:15.107+0200 7efe51a9cb40 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Operation not permitted
2021-04-06T15:39:15.107+0200 7efe51a9cb40 0 ERROR: failed to start datalog_rados service ((1) Operation not permitted
2021-04-06T15:39:15.107+0200 7efe51a9cb40 0 ERROR: failed to init services (ret=(1) Operation not permitted)
2021-04-06T15:39:15.131+0200 7efe51a9cb40 -1 Couldn't init storage provider (RADOS)

rgw also does not start with manual command invokation.