Bug #61725
openFailed preloading extblkdev plugins, libcap depedency isn't specified?
0%
Description
When deploy osd in multi-sites cluster, journal shows 'preload extblkdev plugins' (screenshot in attachment)
And after analysis, I found the reason is that 'libcap' versions are different in every node. And 'extblkdev' module is added in https://github.com/ceph/ceph/pull/46037, actually it lacks of a thing like 'version requirement' in spec. Not sure which version should be assigned.
Files
Updated by Ilya Dryomov 11 months ago
- Project changed from rbd to RADOS
- Subject changed from extblkdev: preload extblkdev plugins to Failed preloading extblkdev plugins, libcap depedency isn't specified?
- Priority changed from Normal to High
Rongqi Sun wrote:
Not sure which version should be assigned.
Definitely not RBD, I'm moving this to RADOS as the pictures don't make the source of the error clear but OSDs definitely fail to start.
Updated by Radoslaw Zarzynski 11 months ago
As as workaround maybe you could set osd_extblkdev_plugins
to empty.
Updated by Radoslaw Zarzynski 11 months ago
- Status changed from New to Need More Info
If the osd_extblkdev_plugins
tuning helps, please let us know!
Updated by Radoslaw Zarzynski 11 months ago
Last resort: https://github.com/ceph/ceph/pull/52195
Updated by Jianyun Cheng 7 months ago
Radoslaw Zarzynski wrote:
If the
osd_extblkdev_plugins
tuning helps, please let us know!
I had the same issue. Setting osd_extblkdev_plugins
to empty doesn't help. I'm using centos 8 and libcap
version is 2.26. After upgrade libcap
version to 2.48, the osds started.
Updated by Radoslaw Zarzynski 7 months ago
Hi Jianyun!
It's really weird that emptying the configurable doesn't help.
Would it be possible to have an OSD's log with verbosity bumped
up (debug_osd=20
)?
Regards,
Radek
Updated by Ben Hines 4 months ago
Radoslaw Zarzynski wrote:
Would it be possible to have an OSD's log with verbosity bumped
up (debug_osd=20
)?
We are seeing the same issue. setting debug_osd=20 does not add much more information. (unless i did it wrong?)
I'm guessing this happens on any Centos 8?
I'll have to figure out how to upgrade libpcap if that was a workaround
2024-01-19T20:19:47.074-0800 7f1b076eb640 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-19T20:19:47.074-0800 7f1b076eb640 0 ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-osd, pid 2914669 2024-01-19T20:19:47.074-0800 7f1b076eb640 0 pidfile_write: ignore empty --pid-file 2024-01-19T20:19:47.074-0800 7f1b076eb640 5 object store type is bluestore 2024-01-19T20:19:47.075-0800 7f1b076eb640 -1 Failed preloading extblkdev plugins, error code: -22
Updated by Ben Hines 4 months ago
Ben Hines wrote:
Radoslaw Zarzynski wrote:
Would it be possible to have an OSD's log with verbosity bumped
up (debug_osd=20
)?We are seeing the same issue. setting debug_osd=20 does not add much more information. (unless i did it wrong?)
I'm guessing this happens on any Centos 8?I'll have to figure out how to upgrade libpcap if that was a workaround
[...]
Edit: "dnf upgrade libcap" did indeed fix it and allow the OSD to start up.
Updated by Radoslaw Zarzynski 3 months ago
Long shot: if a library upgrade helps, this might be a problem with a dependency of the plugin. How about debugging running osd with LD_DEBUG=libs
? (https://bnikolic.co.uk/blog/linux-ld-debug.html)