Ceph

We are so eager to test out the Firefly erasure coding features even before the RPM packages being released. Here we got some exceptions and hope to get your support.

I tried to download latest code from its master and firefly branch, build rpm package and then deploy cluster with rpm package. The MON will become unavailable after creating erasure code crush rule because by default there is no any erasure code crush rule. Neither master nor firefly branch can work.

-bash-4.1$ sudo ceph osd crush rule create-erasure erasure_ruleset
created rule erasure_ruleset at 1
-bash-4.1$ sudo ceph -s
2014-02-20 09:58:29.554241 7ff8b422b700 0 -- xxx.xxx.xxx.xxx:0/1024566 >> xxx.xxx.xxx.xxx:6789/0 pipe(0x7ff8b0025440 sd=4 :43241 s=1 pgs=0 cs=0 l=1 c=0x7ff8b00256a0).connect protocol feature mismatch, my 7ffffffff < peer 17ffffffff missing 1000000000
2014-02-20 09:58:29.554324 7ff8b422b700 0 -- xxx.xxx.xxx.xxx:0/1024566 >> xxx.xxx.xxx.xxx:6789/0 pipe(0x7ff8b0025440 sd=4 :43241 s=1 pgs=0 cs=0 l=1 c=0x7ff8b00256a0).fault
2014-02-20 09:58:29.554822 7ff8b422b700 0 -- xxx.xxx.xxx.xxx:0/1024566 >> xxx.xxx.xxx.xxx:6789/0 pipe(0x7ff8b0025440 sd=4 :43242 s=1 pgs=0 cs=0 l=1 c=0x7ff8b00256a0).connect protocol feature mismatch, my 7ffffffff < peer 17ffffffff missing 1000000000

Snippet of code:

} else if (prefix == "osd crush rule create-erasure") {
err = check_cluster_features(CEPH_FEATURE_CRUSH_V2, ss);
if (err)
goto reply;

Since the steps to build rpm package on ceph site can’t work, I have to make some workarounds.

1. download source code, update submodule, like libs3, etc, and install libfcgi on build env.
2. run './autogen.sh && ./configure --with-rest-bench --with-radosgw --with-debug --with-PACKAGE’ to generate ceph.spec for rpm.
3. compress source code directory to .tar.bz2.
4. run ‘sudo rpmbuild -tb rpmbuild/SOURCES/ceph-0.76.tar.bz2’. The final ceph.spec will be extracted to rpmbuild/SPECS. And this command will output an error because of the conflict after merging original ceph.spec and libs3.spec.
5. modify final ceph.spec to remove all the contents of libs3.spec. Libs3 will be compiled in original ceph.spec.
6. modify final ceph.spec to remove all the libcephfs settings because libcephfs_jni1 will have compile error.
7. run 'sudo rpmbuild -bb rpmbuild/SPECS/ceph.spec’. There will be an error for the first time to indicate following files installed but unpackaged.
/usr/bin/ceph_test_libcephfs
/usr/bin/ceph_test_rgw_manifest
/usr/bin/cephfs
/usr/include/cephfs/libcephfs.h
/usr/lib/python2.6/site-packages/cephfs.py
/usr/lib/python2.6/site-packages/cephfs.pyc
/usr/lib/python2.6/site-packages/cephfs.pyo
/usr/lib64/libcephfs.so
/usr/lib64/libcephfs.so.1
/usr/lib64/libcephfs.so.1.0.0
/usr/lib64/rados-classes/libcls_user.so
/usr/lib64/rados-classes/libcls_user.so.1
/usr/lib64/rados-classes/libcls_user.so.1.0.0
8. add above files under “%files” section in final ceph.spec.
9. rerun step 7 and then rpm package will be built out successfully.