Ceph » devops

There are some references to the python files that will need to be updated for the new paths. The first one is for building docs, which can be checked at http://ceph.com/docs/wip-5900/rbd/librbdpy/#api-reference compared to http://ceph.com/docs/master/rbd/librbdpy/#api-reference

admin/build-doc:PYTHONPATH=`pwd`/../src/pybind $vdir/bin/sphinx-build -a -b dirhtml -d doctrees ../doc output/html
qa/workunits/rados/test_python.sh:#wget -q https://raw.github.com/ceph/ceph/$CEPH_REF/src/test/pybind/test_rados.py
qa/workunits/rados/test_python.sh:wget -O test_rados.py "https://ceph.com/git/?p=ceph.git;a=blob_plain;hb=$CEPH_REF;f=src/test/pybind/test_rados.py" || \
qa/workunits/rados/test_python.sh:    wget -O test_rados.py "https://ceph.com/git/?p=ceph.git;a=blob_plain;hb=ref/heads/$CEPH_REF;f=src/test/pybind/test_rados.py" 
qa/workunits/rbd/test_librbd_python.sh:#wget -q https://raw.github.com/ceph/ceph/$CEPH_REF/src/test/pybind/test_rbd.py
qa/workunits/rbd/test_librbd_python.sh:wget -O test_rbd.py "https://ceph.com/git/?p=ceph.git;a=blob_plain;hb=$CEPH_REF;f=src/test/pybind/test_rbd.py" || \
qa/workunits/rbd/test_librbd_python.sh:    wget -O test_rbd.py "https://ceph.com/git/?p=ceph.git;a=blob_plain;hb=ref/heads/$CEPH_REF;f=src/test/pybind/test_rbd.py" 
src/Makefile.am:python_PYTHON = pybind/rados.py \
src/Makefile.am:                pybind/rbd.py \
src/Makefile.am:                pybind/cephfs.py \
src/Makefile.am:                pybind/ceph_argparse.py \
src/Makefile.am:                pybind/ceph_rest_api.py
src/ceph-rest-api:   os.path.exists(os.path.join(MYDIR, 'pybind')):
src/ceph-rest-api:    sys.path.insert(0, os.path.join(MYDIR, 'pybind'))
src/ceph.in:# If in src/, and .libs and pybind exist here, assume we're running
src/ceph.in:   os.path.exists(os.path.join(MYDIR, 'pybind')):
src/ceph.in:    sys.path.insert(0, os.path.join(MYDIR, 'pybind'))
src/test/run-rbd-tests:export PYTHONPATH="$CEPH_SRC/pybind:$CEPH_SRC/test/pybind" 
src/vstart.sh:export PYTHONPATH=./pybind
src/vstart.sh:echo "export PYTHONPATH=./pybind" 

Josh also pointed out about the licensing.

I am not sure how to proceed here but the Python files say GPLv2

https://github.com/ceph/ceph/blob/master/src/pybind/ceph_argparse.py#L11-12

I've made all the bindings individual packages and updates `ceph-rest-api` script to use the right imports.

Installing this packages need to be treated differently. We no longer can alter PYTHONPATH to point to `pybind` because that will not work as these sub-directories include non-package files.

https://github.com/ceph/ceph/compare/wip-5900

Discussed in IRC today with Alfredo and others: We're going to keep the pyceph modules as individual Python packages as wip-5900 currently does. During each ceph release, we'll "mirror" the python packages to PyPI. As far as RPM packaging goes, we're going to continue to ship the pyceph stuff as subpackages in the main ceph.spec file.

I will rebase the wip-5900 branch so that we're doing the right thing in ceph.spec.in.

This is the BZ ticket associated with the work on this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1116757

I've rebased wip-5900 today and pushed it to ceph.git. There's one last thing we need to resolve. It looks like the individual python modules do not install to /usr/lib/python2.7/site-packages any more.

I was expecting to see make install put the modules into /usr/lib/python2.7/site-packages/python-librados/, /usr/lib/python2.7/site-packages/python-cephfs/ , etc

Originally all the files were a part of python_PYTHON:

-# pybind
-
-python_PYTHON = pybind/rados.py \
-               pybind/rbd.py \
-               pybind/cephfs.py \
-               pybind/ceph_argparse.py \
-               pybind/ceph_rest_api.py
+# deprecated pybind modules

+python_PYTHON = pybind/ceph_argparse.py \
+               pybind/ceph_rest_api.py

Now it looks like they only get installed into a virtualenv? Was that done in order to build the docs?

+# Install Python bindings into a virtualenv
+install-pybind:
+       @cd pybind && ./bootstrap
+
+uninstall-pybind:
+       rm -rf pybind/virtualenv

I think we'll need the main Makefile to run python setup.py install in each of the modules' directories, right?

if you look into the bootstrapping script you will see that it does in fact call `python setup.py develop` on each pybind module.

The reason for this is because it will always be safer to use a virtualenv than using the system python environment. We should really try to keep this model
of working with the bindings.

I think we discussed this on IRC a couple weeks ago, but returning to it now:

We still need Ceph's make install to install the various Python modules to the site-wide installation location. A virtualenv covers some use-cases, but python setup.py install should still run be run during make install.

To summarize progress here: that "wip-5900" branch in GitHub will need to be rebased onto the current master branch. Since the Python packaging split that shipped in Hammer, there are probably several commits in wip-5900, particularly in the RPM/DEB packaging, that we can reduce or eliminate altogether.

Ken Dreyer wrote:

To summarize progress here: that "wip-5900" branch in GitHub will need to be rebased onto the current master branch. Since the Python packaging split that shipped in Hammer, there are probably several commits in wip-5900, particularly in the RPM/DEB packaging, that we can reduce or eliminate altogether.

Hi Ken! Are there any updates on this issue? We are also looking to be able to install python-rados, python-rbd, etc. from pypi into a python virtualenv. Any information appreciated! :)

Hey, I've manually created a pypi package for the ceph python bindings, I'll happily remove it once the official packages are created. https://pypi.python.org/pypi?name=python-cephlibs&version=0.94.5.post1&:action=display

I reworked the build system of rados bindings to make it suitable for PyPI. The pull request is still under review, but there is a test package on PyPI testing site in case someone wants to give it a try. https://github.com/ceph/ceph/pull/9332