Project

General

Profile

Actions
Copy link

Bug #61725

open

Failed preloading extblkdev plugins, libcap depedency isn't specified?

Added by Rongqi Sun 11 months ago. Updated 3 months ago.

Status:
Need More Info
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When deploy osd in multi-sites cluster, journal shows 'preload extblkdev plugins' (screenshot in attachment)

And after analysis, I found the reason is that 'libcap' versions are different in every node. And 'extblkdev' module is added in https://github.com/ceph/ceph/pull/46037, actually it lacks of a thing like 'version requirement' in spec. Not sure which version should be assigned.

Files

Download all files
1.png (6.41 KB) 1.png Rongqi Sun, 06/19/2023 11:55 AM
2.png (88.8 KB) 2.png Rongqi Sun, 06/19/2023 11:56 AM
Actions
Copy link
 #1

Updated by Ilya Dryomov 11 months ago

  • Project changed from rbd to RADOS
  • Subject changed from extblkdev: preload extblkdev plugins to Failed preloading extblkdev plugins, libcap depedency isn't specified?
  • Priority changed from Normal to High

Rongqi Sun wrote:

Not sure which version should be assigned.

Definitely not RBD, I'm moving this to RADOS as the pictures don't make the source of the error clear but OSDs definitely fail to start.

Actions
Copy link
 #2

Updated by Radoslaw Zarzynski 11 months ago

As as workaround maybe you could set osd_extblkdev_plugins to empty.

Actions
Copy link
 #3

Updated by Radoslaw Zarzynski 11 months ago

  • Status changed from New to Need More Info

If the osd_extblkdev_plugins tuning helps, please let us know!

Actions
Copy link
 #5

Updated by Jianyun Cheng 7 months ago

Radoslaw Zarzynski wrote:

If the osd_extblkdev_plugins tuning helps, please let us know!

I had the same issue. Setting osd_extblkdev_plugins to empty doesn't help. I'm using centos 8 and libcap version is 2.26. After upgrade libcap version to 2.48, the osds started.

Actions
Copy link
 #6

Updated by Radoslaw Zarzynski 6 months ago

Hi Jianyun!

It's really weird that emptying the configurable doesn't help.
Would it be possible to have an OSD's log with verbosity bumped
up (debug_osd=20)?

Regards,
Radek

Actions
Copy link
 #7

Updated by Ben Hines 3 months ago

Radoslaw Zarzynski wrote:

Would it be possible to have an OSD's log with verbosity bumped
up (debug_osd=20)?

We are seeing the same issue. setting debug_osd=20 does not add much more information. (unless i did it wrong?)
I'm guessing this happens on any Centos 8?

I'll have to figure out how to upgrade libpcap if that was a workaround

2024-01-19T20:19:47.074-0800 7f1b076eb640  0 set uid:gid to 167:167 (ceph:ceph)
2024-01-19T20:19:47.074-0800 7f1b076eb640  0 ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-osd, pid 2914669
2024-01-19T20:19:47.074-0800 7f1b076eb640  0 pidfile_write: ignore empty --pid-file
2024-01-19T20:19:47.074-0800 7f1b076eb640  5 object store type is bluestore
2024-01-19T20:19:47.075-0800 7f1b076eb640 -1 Failed preloading extblkdev plugins, error code: -22

Actions
Copy link
 #8

Updated by Ben Hines 3 months ago

Ben Hines wrote:

Radoslaw Zarzynski wrote:

Would it be possible to have an OSD's log with verbosity bumped
up (debug_osd=20)?

We are seeing the same issue. setting debug_osd=20 does not add much more information. (unless i did it wrong?)
I'm guessing this happens on any Centos 8?

I'll have to figure out how to upgrade libpcap if that was a workaround
[...]

Edit: "dnf upgrade libcap" did indeed fix it and allow the OSD to start up.

Actions
Copy link
 #9

Updated by Radoslaw Zarzynski 3 months ago

Long shot: if a library upgrade helps, this might be a problem with a dependency of the plugin. How about debugging running osd with LD_DEBUG=libs? (https://bnikolic.co.uk/blog/linux-ld-debug.html)

Actions
Copy link

Also available in: Atom PDF