Ceph » crimson

Likely related, centos 8 crimson builds are failing with (https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/77293/consoleFull#17838507786030192d-da7d-4544-bbc5-ba785a77a7b9)

-- Found Protobuf: /usr/lib64/libprotobuf.so;-lpthread (found suitable version "3.19.6", minimum required is "2.5.0") 
...
Error: 
 Problem 1: conflicting requests
  - nothing provides libprotobuf.so.30()(64bit) needed by ceph-osd-2:19.0.0-1826.g9ca0b5de.el8.x86_64 from Ceph

From another PR https://jenkins.ceph.com/job/ceph-pull-requests/130486/consoleFull#85719909585e1414f-af06-4588-8fed-ab5891ddc58a

Collecting protobuf<4.0dev,>=3.12.0 (from grpcio-tools==1.46.5->-r requirements.txt (line 15))
Downloading protobuf-3.20.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (679 bytes)

I'm able to build in a local centos 8 container with protobuf, protobuf-devel, protobuf-compiler 3.5.0.

The two entries above seem to be 3.20 and 3.19 -- perhaps that's the problem? Not sure where the 3.19.6 version from https://tracker.ceph.com/issues/64696#note-2 is coming from.

dmick@irvingi05:~$ dpkg -l | grep protobuf
ii libprotobuf-dev:amd64 3.12.4-1ubuntu7.22.04.1 amd64 protocol buffers C++ library (development files) and proto files
ii libprotobuf-lite23:amd64 3.12.4-1ubuntu7.22.04.1 amd64 protocol buffers C++ library (lite version)
ii libprotobuf23:amd64 3.12.4-1ubuntu7.22.04.1 amd64 protocol buffers C++ library
ii protobuf-compiler 3.12.4-1ubuntu7.22.04.1 amd64 compiler for protocol buffer definition files
ii protobuf-compiler-grpc 1.30.2-3build6 amd64 high performance general RPC framework - protobuf plugin
ii python3-protobuf 3.12.4-1ubuntu7.22.04.1 amd64 Python 3 bindings for protocol buffers

I'm trying to create a jammy environment to reproduce this.

From dmick@irvingi05,

$ protoc --version
libprotoc 25.1

/usr/bin/protoc -> protoc-25.1.0

ii protobuf-compiler 3.12.4-1ubuntu7.22.04.1 amd64 c

This does not match my container which has the same package, but is not a symlink and

main/build [sjust/main] _ protoc --version
libprotoc 3.12.4

as you'd really expect.

From what we can tell, someone tampered with /usr/bin/protoc on 3 jammy hosts:

irvingi08: -rwxr-xr-x 1 root root 18672 Mar  9  2023 /usr/bin/protoc
irvingi05: lrwxrwxrwx 1 root root 13 Feb 24 21:00 /usr/bin/protoc -> protoc-25.1.0
braggi17: -rwxr-xr-x 1 root root 18672 Mar  9  2023 /usr/bin/protoc
braggi12: -rwxr-xr-x 1 root root 18672 Mar  9  2023 /usr/bin/protoc
braggi15: -rwxr-xr-x 1 root root 18672 Mar 17  2022 /usr/bin/protoc
braggi02: -rwxr-xr-x 1 root root 18672 Mar 17  2022 /usr/bin/protoc
braggi10: -rwxr-xr-x 1 root root 18672 Mar 17  2022 /usr/bin/protoc
braggi09: lrwxrwxrwx 1 root root 13 Feb 24 20:58 /usr/bin/protoc -> protoc-25.1.0
braggi16: -rwxr-xr-x 1 root root 18672 Mar 17  2022 /usr/bin/protoc
irvingi06: -rwxr-xr-x 1 root root 18672 Mar 17  2022 /usr/bin/protoc
adami07: -rwxr-xr-x 1 root root 18672 Mar 17  2022 /usr/bin/protoc
braggi11: -rwxr-xr-x 1 root root 18672 Mar  9  2023 /usr/bin/protoc
adami08: lrwxrwxrwx 1 root root 13 Feb 26 09:22 /usr/bin/protoc -> protoc-25.1.0
braggi03: -rwxr-xr-x 1 root root 18672 Mar  9  2023 /usr/bin/protoc
omani004: -rwxr-xr-x 1 root root 14352 Mar  9  2023 /usr/bin/protoc
braggi13: -rwxr-xr-x 1 root root 18672 Mar 17  2022 /usr/bin/protoc
omani009: -rwxr-xr-x 1 root root 14352 Mar  9  2023 /usr/bin/protoc
confusa09: -rwxr-xr-x 1 root root 14352 Mar  9  2023 /usr/bin/protoc
confusa11: -rwxr-xr-x 1 root root 14352 Mar  9  2023 /usr/bin/protoc

irvingi05: libprotoc 25.1
braggi12: libprotoc 3.12.4
braggi15: libprotoc 3.12.4
braggi17: libprotoc 3.12.4
irvingi08: libprotoc 3.12.4
irvingi06: libprotoc 3.12.4
braggi09: libprotoc 25.1
braggi02: libprotoc 3.12.4
braggi16: libprotoc 3.12.4
braggi10: libprotoc 3.12.4
adami08: libprotoc 25.1
adami07: libprotoc 3.12.4
braggi11: libprotoc 3.12.4
braggi03: libprotoc 3.12.4
confusa11: libprotoc 3.12.4
omani004: libprotoc 3.12.4
confusa09: libprotoc 3.12.4
omani009: libprotoc 3.12.4
braggi13: libprotoc 3.12.4

irvingi05: -rwxr-xr-x 1 root root 26225768 Feb 24 21:00 /usr/bin/protoc-25.1.0
braggi09: -rwxr-xr-x 1 root root 26233856 Feb 24 20:57 /usr/bin/protoc-25.1.0
confusa09: -rwxr-xr-x 1 root root 27409488 Feb 24 16:01 /usr/bin/protoc-25.1.0

Ah, interesting. Were you able to fix those hosts? Or do we need someone from Infrastructure to take a look?

Yes, those hosts don't have this problem anymore. I think the current issue is something different.

https://github.com/ceph/ceph/pull/55444 added protobuf-devel and protobuf-compiler as build requirements for a new version of seastar, but did not appear to add runtime dependencies, and also did not note the runtime dependencies for the containerized version. protobuf appears not to be available for centos8 or 9, so we've added a COPR built by Ken Dreyer to handle the build-time dependencies, but it's not clear to me how the package was intended to be installed at runtime for either normal bare-metal installs or container installs.

Samuel Just wrote:

From what we can tell, someone tampered with /usr/bin/protoc on 3 jammy hosts:

[...]

[...]
specifically

[...]

adami01 has the same issue

Indeed, the libprotobuf in the copr repo is libprotobuf.so.30, the version that the built packages require...that is, it's a runtime dependency

adami01 has the same issue

No it doesn't:
[dmick@adami01 ~]$ ls /usr/bin/protoc*
/usr/bin/protoc

I'm going to open another bug about the protoc/libprotobuf-dev damage. This bug is now about the missing runtime dependency.

Please use https://tracker.ceph.com/issues/64907 for further discussion of "wrong versions of protobuf files installed"

Collating data:

8stream
protobuf:

AppStream contains protobuf-compiler 3.5.0-15.el8
depends on libprotobuf.so.15 which is in
protobuf-0:3.5.0-{13,15}.el8.x86_64
Powertools contains protobuf-devel-0:3.5.0-{13,15].el8.x86_64

grpc:
8stream does not contain grpc packages, hence https://copr.fedorainfracloud.org/coprs/ceph/grpc/, which contains among other things
grpc-cpp-0:1.46.7-22.el8.x86_64, which depends on
libprotoc.so.30()(64bit), in
protobuf-compiler-0:3.19.6-6.el8.x86_64, also in the copr

9stream:
protobuf:

appstream contains protobuf-3.14.0-13.el9, which depends on
libprotobuf.so.25()(64bit), which is in
protobuf-0:3.14.0-{8.9,13}.el9.x86_64
protobuf-devel is in crb as protobuf-3.14.0-{8,9,13}.el9

grpc:
epel9 contains all the grpc packages (grpc, grpc-cli, grpc-cpp, grpc-devel)
all are version 0:1.46.7-10.el9
grpc-cpp-0:1.46.7-10.el9.x86_64 depends on
libprotobuf.so.25, in
protobuf-3.14.0-13.el9.x86_64, from appstream

So the problem with using the copr for stream8 is that its packages require libprotobuf.so.30, which is later than even epel9 requires. I don't know how one might fix the copr build to depend on libprotobuf.so.25.

jammy has libprotobuf.so.23.0.4 from libprotobuf23, protobuf-compiler-3.12.4 depends on it

python3-grpcio-1.30.2-3build6 is a BuildRequires, and it depends on python3-protobuf (>= 3.11.4~), which resolves to python3-protobuf-3.12.4-1ubuntu7.22.04.1, which depends on libprotobuf23 (>= 3.12.4), which resolves to libprotobuf23-3.12.4-1ubuntu7.22.04.1.

I think I made a mistake when I chose the protobuf version in https://copr.fedorainfracloud.org/coprs/ceph/grpc/ . I forked the Fedora Rawhide version of protobuf 3.19.6 instead of forking the el9 version (3.14.0). I will try rebuilding grpc against the older version.

This will likely fix the (I know now) centos8-only issue. I saw the copr for el9 in the build and mistakenly thought there was grpc stuff in there too, but there's not.

Of course our next step is to leave c8 behind, but that will surely take some incremental fixing.

I made the changes in a temporary repo (https://copr.fedorainfracloud.org/coprs/ktdreyer/grpc/), and the packages install ok on el8. It has libprotobuf.so.25. I'm going to copy these to the ceph/grpc copr now.

https://copr.fedorainfracloud.org/coprs/ceph/grpc/builds/ has protobuf 3.14.0 now.

If Ceph links against libprotobuf.so and requires it at runtime, then we'll need to enable the ceph/grpc copr to get libprotobuf.so.25 inside the el8 container image.

Yep, I'm on it.

I've observed builds with the right library dependency now. There needs to be one more change to ceph-container to enable the copr during package install; I'm working on that now, but it should be the last piece.