Project

General

Profile

Cleanup #38088

Add dependencies for missing Python packages

Added by Dan Mick about 5 years ago. Updated about 5 years ago.

Status:
New
Priority:
High
Assignee:
Category:
common
Target version:
-
% Done:

0%

Tags:
mgr plugins
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

ceph-mgr-rook, ceph-mgr-diskprediction-local, and ceph-mgr-diskprediction-cloud packages will need to have dependencies added for their currently-unpackaged Python library dependencies once distro packages are available for them (into debian/control and ceph.spec.in).

-rook: python-kubernetes (apt), ? (yum)
-local: python-{numpy, scipy, sklearn} (apt), numpy, scipy, sklearn (yum)
-cloud: google-{api-python-client, auth, auth-httplib2}, googleapis-common-protos, grpc, grpc-google-{logging-v2, pubsub-v1}, grpcio

Related issues

Related to mgr - Bug #38262: New Bionic install fails qa/standalone/ceph-helpers.sh Resolved 02/11/2019
Related to Ceph - Bug #38810: ceph-mgr-diskprediction-local requires "numpy" and "scipy" on SUSE, but these packages do not exist on SUSE Resolved 03/19/2019
Related to mgr - Bug #44948: Module 'diskprediction_local' has failed: No module named 'sklearn' - ceph docker v15 Resolved

History

#1 Updated by Dan Mick about 5 years ago

  • Description updated (diff)

#2 Updated by James Page about 5 years ago

What are the minimum version requirements for these new dependencies?

I think most will be in Ubuntu/Debian development so we'll need backports for older Ubuntu and Debian releases.

#3 Updated by James Page about 5 years ago

"-cloud: google-{api-python-client, auth, auth-httplib2}, googleapis-common-protos, grpc, grpc-google-{logging-v2, pubsub-v1}, grpcio"

This set might be challenging - predominately not package to-date.

#4 Updated by Dan Mick about 5 years ago

The existing code was written with requirements.txt as such, so I guess that's at least a version that is known to work:

google-api-python-client==1.7.3
google-auth==1.5.0
google-auth-httplib2==0.0.3
googleapis-common-protos==1.5.3
grpc==0.3.post19
grpc-google-logging-v2==0.8.1
grpc-google-pubsub-v1==0.8.1
grpcio==1.14.1
numpy==1.15.1
scikit-learn==0.19.2
scipy==1.1.0

Not clear about python-kubernetes; whatever's current in pypi is probably the closest to an answer.

And, yes, I don't think anyone's tried to package the google stuff at all yet. I'm hacking at a Debian package for grpc.

#5 Updated by Dan Mick about 5 years ago

trying to build grpc into a .deb, I find that

1) it hasn't been released on pypi since June 2016, and
2) it uses a now-deprecated module in gevent, "coros", removed in Dec 2016, and
3) the latest source on https://bitbucket.org/seewind/grpc doesn't fix the issue

I...don't understand.

#6 Updated by James Page about 5 years ago

for -root and -local:

https://launchpad.net/~james-page/+archive/ubuntu/ceph-backport-testing

contains backports for bionic - do we need xenial as well?

#7 Updated by David Zafman about 5 years ago

Does this tracker apply to a new build machine? The pyopenssl is needed for a local build machine run-standalone.sh to pass.

I have this tracker: http://tracker.ceph.com/issues/38262

Something like this.

Install new machine
git clone ....ceph.git
cd ceph
sudo install-deps.sh
./do-cmake.sh
cd build
make
../qa/run-standalone.sh "test-ceph-helpers.sh test_wait_for_health_ok" 

Fails with:

HEALTH_WARN Module 'restful' has failed dependency: No module named OpenSSL
MGR_MODULE_DEPENDENCY Module 'restful' has failed dependency: No module named OpenSSL
   Module 'restful' has failed dependency: No module named OpenSSL

#8 Updated by David Zafman about 5 years ago

  • Related to Bug #38262: New Bionic install fails qa/standalone/ceph-helpers.sh added

#9 Updated by Noah Watkins about 5 years ago

Adding the SSH orchestrator manager module to this list.

The dependency is the remoto package version >= 0.35: https://github.com/ceph/remoto

The main dependencies are:

https://packages.debian.org/source/buster/apipkg
https://packages.debian.org/source/buster/execnet

It looks like Ken Dryer built some packages a while back (https://launchpad.net/~kdreyer-redhat/+archive/ubuntu/ceph-medic), but they are out of date.

#10 Updated by Dan Mick about 5 years ago

The confusion about grpc in https://tracker.ceph.com/issues/38088#note-5 is because grpc is nothing to do with this code; it's extraneous nonsense in the requirements.txt file. Glad the developers were careful with their dependencies and making sure they existed in the distros.

#11 Updated by James Page about 5 years ago

python-kubernetes also added to:

https://launchpad.net/~james-page/+archive/ubuntu/ceph-backport-testing

alongside the -local depends (which I also added a copy of smartmontools 7.0 to - I need to see if I can get that into current Ubuntu development).

#12 Updated by Brad Hubbard almost 5 years ago

  • Related to Bug #38810: ceph-mgr-diskprediction-local requires "numpy" and "scipy" on SUSE, but these packages do not exist on SUSE added

#13 Updated by Josh Durgin over 3 years ago

  • Related to Bug #44948: Module 'diskprediction_local' has failed: No module named 'sklearn' - ceph docker v15 added

#14 Updated by Deepika Upadhyay over 3 years ago

  • Related to deleted (Bug #44948: Module 'diskprediction_local' has failed: No module named 'sklearn' - ceph docker v15)

#15 Updated by Deepika Upadhyay over 3 years ago

  • Related to Bug #44948: Module 'diskprediction_local' has failed: No module named 'sklearn' - ceph docker v15 added

Also available in: Atom PDF