Fix #5900
openCreate a Python package for ceph Python bindings
0%
Description
The `src/pybind` directory needs to be converted to a proper Python package structure so we can create and publish Python packages that can be used, installed and managed in Python projects (most notably OpenStack).
Updated by Alfredo Deza over 10 years ago
- Status changed from In Progress to Fix Under Review
Branch `wip-5900` was pushed and it needs a few things addressed:
- Versioning -> (it currently is defaulting to 0.0.1, which is obviously wrong)
- README.rst -> There is a stub for it, John has some work done here and will be getting some contents there
I've done some renaming for the modules (e.g. ceph_rest_api to rest_api), that also needs some reviewing/feedback.
Updated by Josh Durgin over 10 years ago
There are some references to the python files that will need to be updated for the new paths. The first one is for building docs, which can be checked at http://ceph.com/docs/wip-5900/rbd/librbdpy/#api-reference compared to http://ceph.com/docs/master/rbd/librbdpy/#api-reference
admin/build-doc:PYTHONPATH=`pwd`/../src/pybind $vdir/bin/sphinx-build -a -b dirhtml -d doctrees ../doc output/html qa/workunits/rados/test_python.sh:#wget -q https://raw.github.com/ceph/ceph/$CEPH_REF/src/test/pybind/test_rados.py qa/workunits/rados/test_python.sh:wget -O test_rados.py "https://ceph.com/git/?p=ceph.git;a=blob_plain;hb=$CEPH_REF;f=src/test/pybind/test_rados.py" || \ qa/workunits/rados/test_python.sh: wget -O test_rados.py "https://ceph.com/git/?p=ceph.git;a=blob_plain;hb=ref/heads/$CEPH_REF;f=src/test/pybind/test_rados.py" qa/workunits/rbd/test_librbd_python.sh:#wget -q https://raw.github.com/ceph/ceph/$CEPH_REF/src/test/pybind/test_rbd.py qa/workunits/rbd/test_librbd_python.sh:wget -O test_rbd.py "https://ceph.com/git/?p=ceph.git;a=blob_plain;hb=$CEPH_REF;f=src/test/pybind/test_rbd.py" || \ qa/workunits/rbd/test_librbd_python.sh: wget -O test_rbd.py "https://ceph.com/git/?p=ceph.git;a=blob_plain;hb=ref/heads/$CEPH_REF;f=src/test/pybind/test_rbd.py" src/Makefile.am:python_PYTHON = pybind/rados.py \ src/Makefile.am: pybind/rbd.py \ src/Makefile.am: pybind/cephfs.py \ src/Makefile.am: pybind/ceph_argparse.py \ src/Makefile.am: pybind/ceph_rest_api.py src/ceph-rest-api: os.path.exists(os.path.join(MYDIR, 'pybind')): src/ceph-rest-api: sys.path.insert(0, os.path.join(MYDIR, 'pybind')) src/ceph.in:# If in src/, and .libs and pybind exist here, assume we're running src/ceph.in: os.path.exists(os.path.join(MYDIR, 'pybind')): src/ceph.in: sys.path.insert(0, os.path.join(MYDIR, 'pybind')) src/test/run-rbd-tests:export PYTHONPATH="$CEPH_SRC/pybind:$CEPH_SRC/test/pybind" src/vstart.sh:export PYTHONPATH=./pybind src/vstart.sh:echo "export PYTHONPATH=./pybind"
Updated by Alfredo Deza over 10 years ago
Josh also pointed out about the licensing.
I am not sure how to proceed here but the Python files say GPLv2
https://github.com/ceph/ceph/blob/master/src/pybind/ceph_argparse.py#L11-12
Updated by Alfredo Deza over 10 years ago
I've made all the bindings individual packages and updates `ceph-rest-api` script to use the right imports.
Installing this packages need to be treated differently. We no longer can alter PYTHONPATH to point to `pybind` because that will not work as these sub-directories include non-package files.
Updated by Alfredo Deza over 9 years ago
- Status changed from Fix Under Review to In Progress
Updated by Loïc Dachary over 9 years ago
Updated by Ken Dreyer over 9 years ago
Discussed in IRC today with Alfredo and others: We're going to keep the pyceph modules as individual Python packages as wip-5900 currently does. During each ceph release, we'll "mirror" the python packages to PyPI. As far as RPM packaging goes, we're going to continue to ship the pyceph stuff as subpackages in the main ceph.spec file.
I will rebase the wip-5900 branch so that we're doing the right thing in ceph.spec.in.
Updated by Alfredo Deza over 9 years ago
This is the BZ ticket associated with the work on this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1116757
Updated by Alfredo Deza over 9 years ago
- Assignee changed from Alfredo Deza to Ken Dreyer
Updated by Ken Dreyer over 9 years ago
I've rebased wip-5900 today and pushed it to ceph.git. There's one last thing we need to resolve. It looks like the individual python modules do not install to /usr/lib/python2.7/site-packages
any more.
I was expecting to see make install
put the modules into /usr/lib/python2.7/site-packages/python-librados/
, /usr/lib/python2.7/site-packages/python-cephfs/
, etc
Originally all the files were a part of python_PYTHON
:
-# pybind - -python_PYTHON = pybind/rados.py \ - pybind/rbd.py \ - pybind/cephfs.py \ - pybind/ceph_argparse.py \ - pybind/ceph_rest_api.py +# deprecated pybind modules +python_PYTHON = pybind/ceph_argparse.py \ + pybind/ceph_rest_api.py
Now it looks like they only get installed into a virtualenv? Was that done in order to build the docs?
+# Install Python bindings into a virtualenv +install-pybind: + @cd pybind && ./bootstrap + +uninstall-pybind: + rm -rf pybind/virtualenv
I think we'll need the main Makefile to run python setup.py install
in each of the modules' directories, right?
Updated by Alfredo Deza over 9 years ago
if you look into the bootstrapping script you will see that it does in fact call `python setup.py develop` on each pybind module.
The reason for this is because it will always be safer to use a virtualenv than using the system python environment. We should really try to keep this model
of working with the bindings.
Updated by Ken Dreyer over 9 years ago
I think we discussed this on IRC a couple weeks ago, but returning to it now:
We still need Ceph's make install
to install the various Python modules to the site-wide installation location. A virtualenv covers some use-cases, but python setup.py install
should still run be run during make install
.
Updated by Ken Dreyer over 8 years ago
To summarize progress here: that "wip-5900" branch in GitHub will need to be rebased onto the current master branch. Since the Python packaging split that shipped in Hammer, there are probably several commits in wip-5900, particularly in the RPM/DEB packaging, that we can reduce or eliminate altogether.
Updated by Matt Thompson over 8 years ago
Ken Dreyer wrote:
To summarize progress here: that "wip-5900" branch in GitHub will need to be rebased onto the current master branch. Since the Python packaging split that shipped in Hammer, there are probably several commits in wip-5900, particularly in the RPM/DEB packaging, that we can reduce or eliminate altogether.
Hi Ken! Are there any updates on this issue? We are also looking to be able to install python-rados, python-rbd, etc. from pypi into a python virtualenv. Any information appreciated! :)
Updated by Hugh Saunders over 8 years ago
Hey, I've manually created a pypi package for the ceph python bindings, I'll happily remove it once the official packages are created. https://pypi.python.org/pypi?name=python-cephlibs&version=0.94.5.post1&:action=display
Updated by Anirudha Bose almost 8 years ago
I reworked the build system of rados bindings to make it suitable for PyPI. The pull request is still under review, but there is a test package on PyPI testing site in case someone wants to give it a try. https://github.com/ceph/ceph/pull/9332