Ceph » devops

Ken Dreyer wrote:

Do we have some form of kernel module packages available already? Or will these be developed from scratch?

All we have are the (kernel) .deb's produced by the gitbuilders. No module packages. Not really familiar with how those are normally put together (either in the deb or rpm world)

RPM kmods are a special kind of pain. Red Hat has a kABI, and in theory if you rely on that, you can re-use your kernel module on different kernel versions. More info on kABI here:
https://access.redhat.com/site/solutions/444773

In my experience Red Hat's list only covers a very limited set of kernel symbols. If CephFS uses symbols that are not covered, all bets are off, and the only truly safe thing is to recompile the kmod every time Red Hat ships a new kernel version.

kabi-style packages are a lot easier from a user's perspective, since the packages don't require a specific kernel version, and in fact there are community add-on repos (such as ELRepo) that just gamble and take their chances that RHEL's kernel will be "stable enough". TBH I can't personally stand behind such an approach unless the CephFS developers are completely satisfied that all their symbols are on Red Hat's whitelist.

Inktank can sign up to be a Red Hat partner and submit a list of symbols for consideration, too. It would be a good idea to give that a shot and see what they say. Who can do that?

The two alternatives to kABI are 1) we build enough automation to watch for Red Hat's new kernel package releases and quickly rebuild against each one, or 2) we ship dkms-style packages and let users dynamically rebuild the kmod as they boot into each new kernel. These options are not mutually exclusive from an package development perspective, but the user does have to choose which one they'll use.

Where can I find the code to build against kernel-devel-3.10.0-54.0.1.el7 ?

Try the for-stable-3.10.24 branch of https://github.com/ceph/ceph-client.git

Adding these links for reference.

http://people.redhat.com/jcm/el6/dup/docs/old/pre-release/whitepaper.pdf

http://people.redhat.com/jcm/el6/dup/docs/scripts/rhel6_kabi_check-2.0.py

I've pushed the cherry-picked commit to a new branch "rhel7": https://github.com/ceph/ceph-client/tree/rhel7 . As the commit log states, there were conflicts with fs/locks.c and include/linux/fs.h. I tried to resolve the conflicts according to what Red Hat ships in 3.10.0-54.0.1.el7. We will build the RPMs from this branch going forward.

Work-in-progress kmod packaging here: https://github.com/kdreyer-inktank/ceph-kmod-rpm

Originally I'd planned to build each kmod as a separate process, but that doesn't look like it's going to work out, because we get missing symbol errors during the ceph build.

1. Re-do the tarball generation script so that we're tarring up the entire ceph-client directory.
2. Adjust the RPM so that it builds libceph.ko, ceph.ko, and rbd.ko all during the %build step.
3. Adjust the RPM so that it ships each of the binaries as individual sub-packages from the single ceph-kmod package.

I've unified all three kmods into a single package, available on GitHub: https://github.com/kdreyer-inktank/ceph-kmod-rpm

Sandon has set up a RHEL 7 Beta gitbuilder, and I've configured the Jenkins bits on it. The steps to configure Jenkins are here, submitted for review into Alfredo's new Ansible repo.

Jenkins is set up to the ceph-client repository every hour and build the rhel7 branch if there are any new changes. Here's the first successful build, with RPMs available for download: http://jenkins.ceph.com/view/ceph-kmod/job/ceph-kmod-rpm/label=rhel7/

The next step is to determine how to automate the transfer of these RPMs onto Inktank's web server.

Does the package install the module into the kernel or does the user have to do modprobe manually?

If it installs it by default (via postint or whatever), I think we need to have each module in its own package as users won't want to run by modules and inserting both increases the chances of a kernel panic, making it harder to diagnose.

mount.ceph and 'rbd map' will do the modprobe for the user if it's not already loaded, so there's no need for the package to do it from a usability perspective.

The packages only run depmod, and they leave modprobe to the user or to the tooling. So it sounds like we're good.

I've pushed the packages to the "rpm-testing" repository on ceph.com where we normally publish development builds. There is a new rhel7 directory, here: http://ceph.com/rpm-testing/rhel7/

The modules load successfully, at least:

[root@el7 0.1.20140224gitc9d1614]# modprobe rbd
[root@el7 0.1.20140224gitc9d1614]# modprobe ceph
[root@el7 0.1.20140224gitc9d1614]# lsmod | grep ceph
ceph                  285983  0 
libceph               224993  2 rbd,ceph
libcrc32c              12644  2 xfs,libceph

Having them in one package is fine for now, but on reflection I do think we are going to need to have these in separate packages, ideally for April when we officially release them under ICE 1.2. At the very least, I think it will make it easier for us rev a specific module and republish it, which I expect we are going to do frequently as we bug squash.

The kmods are built from a common source RPM because ceph.ko and rbd.ko require libceph.ko to be built and available at build time, but from the users' perspective each kmod lives in its own binary package, as seen here: http://ceph.com/rpm-testing/rhel7/x86_64/

For the record, the Git repo has moved to the Ceph org: https://github.com/ceph/ceph-kmod-rpm

I've updated the Jenkins job to point at this new location, and I'm discussing with Sandon the steps needed to update Gitbuilder as well.