Build packages containing RBD and CephFS kernel modules for installation on RHEL7-beta
RHEL7 beta is now available. We need to provide two packages - one for RBD and one for CephFS - which provide prebuilt kernel modules that can install into the RHEL7 kernel.
#4 Updated by Sage Weil over 9 years ago
Ken Dreyer wrote:
Do we have some form of kernel module packages available already? Or will these be developed from scratch?
All we have are the (kernel) .deb's produced by the gitbuilders. No module packages. Not really familiar with how those are normally put together (either in the deb or rpm world)
#5 Updated by Ken Dreyer over 9 years ago
RPM kmods are a special kind of pain. Red Hat has a kABI, and in theory if you rely on that, you can re-use your kernel module on different kernel versions. More info on kABI here:
In my experience Red Hat's list only covers a very limited set of kernel symbols. If CephFS uses symbols that are not covered, all bets are off, and the only truly safe thing is to recompile the kmod every time Red Hat ships a new kernel version.
kabi-style packages are a lot easier from a user's perspective, since the packages don't require a specific kernel version, and in fact there are community add-on repos (such as ELRepo) that just gamble and take their chances that RHEL's kernel will be "stable enough". TBH I can't personally stand behind such an approach unless the CephFS developers are completely satisfied that all their symbols are on Red Hat's whitelist.
Inktank can sign up to be a Red Hat partner and submit a list of symbols for consideration, too. It would be a good idea to give that a shot and see what they say. Who can do that?
The two alternatives to kABI are 1) we build enough automation to watch for Red Hat's new kernel package releases and quickly rebuild against each one, or 2) we ship dkms-style packages and let users dynamically rebuild the kmod as they boot into each new kernel. These options are not mutually exclusive from an package development perspective, but the user does have to choose which one they'll use.
#8 Updated by Ken Dreyer over 9 years ago
#10 Updated by Ken Dreyer over 9 years ago
- Status changed from 12 to In Progress
I'm building against kernel-3.10.0-54.0.1.el7.x86_64, and I get the following error when using df5f7b6abd0dacf6fa60acd570c978b8a3aa1cd7 (the tip of the for-stable-3.10.24 branch):
/home/kdreyer/rpmbuild/BUILD/ceph-3.6-df5f7b6/mds_client.c: In function 'encode_caps_cb':
/home/kdreyer/rpmbuild/BUILD/ceph-3.6-df5f7b6/mds_client.c:2491:3: error: implicit declaration of function 'lock_flocks' [-Werror=implicit-function-declaration]
/home/kdreyer/rpmbuild/BUILD/ceph-3.6-df5f7b6/mds_client.c:2493:3: error: implicit declaration of function 'unlock_flocks' [-Werror=implicit-function-declaration]
Looks like Red Hat has cherry-picked 1c8c601a8c0dc59fe64907dcd9d512a3d181ddc7, or something like it, onto their 3.10.0-54.0.1.el7. When I cherry-pick that commit to fs/ceph, ceph.ko builds.
#11 Updated by Ken Dreyer over 9 years ago
I've pushed the cherry-picked commit to a new branch "rhel7": https://github.com/ceph/ceph-client/tree/rhel7 . As the commit log states, there were conflicts with
include/linux/fs.h. I tried to resolve the conflicts according to what Red Hat ships in 3.10.0-54.0.1.el7. We will build the RPMs from this branch going forward.
#12 Updated by Ken Dreyer over 9 years ago
Work-in-progress kmod packaging here: https://github.com/kdreyer-inktank/ceph-kmod-rpm
Originally I'd planned to build each kmod as a separate process, but that doesn't look like it's going to work out, because we get missing symbol errors during the ceph build.As of 0982aa43503cf61e0f24c325a4412d485a1142b9, we need to rewrite the package to build all the kernel modules at once. Specific plan of attack:
- Re-do the tarball generation script so that we're tarring up the entire ceph-client directory.
- Adjust the RPM so that it builds libceph.ko, ceph.ko, and rbd.ko all during the %build step.
- Adjust the RPM so that it ships each of the binaries as individual sub-packages from the single ceph-kmod package.
#13 Updated by Ken Dreyer over 9 years ago
I've unified all three kmods into a single package, available on GitHub: https://github.com/kdreyer-inktank/ceph-kmod-rpm
Jenkins is set up to the ceph-client repository every hour and build the rhel7 branch if there are any new changes. Here's the first successful build, with RPMs available for download: http://jenkins.ceph.com/view/ceph-kmod/job/ceph-kmod-rpm/label=rhel7/
The next step is to determine how to automate the transfer of these RPMs onto Inktank's web server.
#14 Updated by Neil Levine over 9 years ago
Does the package install the module into the kernel or does the user have to do modprobe manually?
If it installs it by default (via postint or whatever), I think we need to have each module in its own package as users won't want to run by modules and inserting both increases the chances of a kernel panic, making it harder to diagnose.
#16 Updated by Ken Dreyer over 9 years ago
The packages only run depmod, and they leave modprobe to the user or to the tooling. So it sounds like we're good.
I've pushed the packages to the "rpm-testing" repository on ceph.com where we normally publish development builds. There is a new rhel7 directory, here: http://ceph.com/rpm-testing/rhel7/
The modules load successfully, at least:
[root@el7 0.1.20140224gitc9d1614]# modprobe rbd [root@el7 0.1.20140224gitc9d1614]# modprobe ceph [root@el7 0.1.20140224gitc9d1614]# lsmod | grep ceph ceph 285983 0 libceph 224993 2 rbd,ceph libcrc32c 12644 2 xfs,libceph
#18 Updated by Neil Levine over 9 years ago
Having them in one package is fine for now, but on reflection I do think we are going to need to have these in separate packages, ideally for April when we officially release them under ICE 1.2. At the very least, I think it will make it easier for us rev a specific module and republish it, which I expect we are going to do frequently as we bug squash.
#19 Updated by Ken Dreyer over 9 years ago
The kmods are built from a common source RPM because ceph.ko and rbd.ko require libceph.ko to be built and available at build time, but from the users' perspective each kmod lives in its own binary package, as seen here: http://ceph.com/rpm-testing/rhel7/x86_64/