Ceph » mgr

I have both python 2.7 and python 3.6 installed, but python3.6 should be the default python interpreter used. Python 2 will be end-of-life on Jan 1st, 2020 : https://pythonclock.org/.

The error message is indicative of the python code for this feature only working with python2, as the "itervalues" attribute of the "dict" object was replaced with "items" in python3 : https://docs.python.org/3.1/whatsnew/3.0.html#views-and-iterators-instead-of-lists

mimir ~ # eselect python list
Available Python interpreters, in order of preference:
[1] python3.6
[2] python2.7

mimir ~ # ceph -s
cluster:
id: 07cb5105-68ea-4f1c-bace-a2be0baae5fa
health: HEALTH_ERR
Module 'pg_autoscaler' has failed: 'dict' object has no attribute 'itervalues'

services:
    mon: 3 daemons, quorum fenrir,mimir,hoenir (age 6h)
    mgr: hoenir(active, since 20h), standbys: mimir, fenrir
    mds: fileserver_backup:1 {0=hoenir=up:active} 2 up:standby
    osd: 16 osds: 16 up, 16 in

data:
    pools:   2 pools, 1500 pgs
    objects: 3.27M objects, 9.8 TiB
    usage:   9.8 TiB used, 38 TiB / 48 TiB avail
    pgs:     1497 active+clean
             3    active+clean+scrubbing

mimir ~ # emerge --info ceph
Portage 2.3.66 (python 3.6.5-final-0, default/linux/amd64/17.0, gcc-8.3.0, glibc-2.29-r2, 4.19.44-gentoo x86_64) =================================================================
System Settings =================================================================
System uname: Linux-4.19.44-gentoo-x86_64-AMD_E-350D_APU_with_Radeon-tm-_HD_Graphics-with-gentoo-2.6
KiB Mem: 16132484 total, 222260 free
KiB Swap: 0 total, 0 free
Timestamp of repository gentoo: Mon, 03 Jun 2019 16:24:41 +0000
Head commit of repository gentoo: 6463dc05c45b01ca9e81e27b0d3492fcfb026827

Head commit of repository jonesmz-public-overlay: 86e790188a0acbd99656428bf8ce1833f3a9fbe2

Head commit of repository steam-overlay: 5314904baf6dcb2e33d0b20a3222d28d8b7de7ba

sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.31.1 p7) 2.31.1
distcc 3.3.2 x86_64-pc-linux-gnu [disabled]
app-shells/bash: 4.4_p23-r1::gentoo
dev-lang/perl: 5.28.2-r1::gentoo
dev-lang/python: 2.7.15::gentoo, 3.6.5::gentoo
dev-util/cmake: 3.14.3::gentoo
dev-util/pkgconfig: 0.29.2::gentoo
sys-apps/baselayout: 2.6-r1::gentoo
sys-apps/sandbox: 2.13::gentoo
sys-devel/autoconf: 2.69-r4::gentoo
sys-devel/automake: 1.11.6-r3::gentoo, 1.16.1-r1::gentoo
sys-devel/binutils: 2.31.1-r6::gentoo
sys-devel/gcc: 8.3.0-r1::gentoo
sys-devel/gcc-config: 2.0::gentoo
sys-devel/libtool: 2.4.6-r3::gentoo
sys-devel/make: 4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.14-r1::gentoo (virtual/os-headers)
sys-libs/glibc: 2.29-r2::gentoo
Repositories:

gentoo
location: /usr/portage
sync-type: git
sync-uri: git://anongit.gentoo.org/repo/sync/gentoo.git
priority: -1000

jonesmz-public-overlay
location: /usr/portage-overlays/jonesmz-public-overlay
sync-type: git
sync-uri: https://github.com/jonesmz/gentoo-overlay.git
masters: gentoo

steam-overlay
location: /usr/portage-overlays/steam-overlay
sync-type: git
sync-uri: https://github.com/anyc/steam-overlay.git
masters: gentoo
priority: 50

Installed sets: @pc-base-system, @portage
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe"
DISTDIR="/usr/portage-distfiles"
EMERGE_DEFAULT_OPTS=" --jobs --keep-going --newuse --deep --tree --backtrack=3000 --complete-graph --with-bdeps=y --binpkg-respect-use=y --binpkg-changed-deps=y --usepkg=y --usepkg"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg clean-logs compress-build-logs compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles installsources ipc-sandbox merge-sync multilib-strict network-sandbox news nostrip parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned sandbox sfperms split-elog split-log strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en en_US"
MAKEOPTS="-j1"
PKGDIR="/usr/portage-packages"
PORTAGE_COMPRESS="xz"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl amd64 avahi btrfs bzip2 clang crypt cxx dbus gd gudev hardened iconv ipv6 libtirpc lm_sensors multilib ncurses nls nptl openmp pam pcre pie python readline samba seccomp ssl ssp systemd threads udev udisks unicode v4l xattr xtpax zeroconf zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core authz_host dir mime unixd socache_shmcb info log_config" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx sse sse2 mmxext" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="coreboot efi-64 emu qemu pc" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en en-US" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-1" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" QEMU_SOFTMMU_TARGETS="arm aarch64 x86_64" QEMU_USER_TARGETS="arm aarch64 x86_64" RUBY_TARGETS="ruby24" USERLAND="GNU" VIDEO_CARDS="r600 radeon radeonsi amdgpu vesa modesetting fbdev qxl" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset: CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

=================================================================
Package Settings =================================================================

sys-cluster/ceph-14.2.1::gentoo was built with the following:
USE="cephfs fuse mgr radosgw ssl systemd tcmalloc -babeltrace -dpdk -grafana -jemalloc -kerberos -ldap -libressl -lttng -numa -rabbitmq (-static-libs) (-system-boost) -test -xfs -zfs" ABI_X86="(64)" CPU_FLAGS_X86="sse sse2 -sse3 -sse4_1 -sse4_2 -ssse3" PYTHON_TARGETS="python2_7 python3_6 -python3_5 -python3_7"
FEATURES="distlocks sandbox xattr userpriv ipc-sandbox pid-sandbox usersync buildpkg binpkg-dostrip config-protect-if-modified compress-build-logs compressdebug binpkg-logs unmerge-logs split-elog userfetch parallel-install sfperms unmerge-orphans binpkg-multi-instance installsources clean-logs news usersandbox assume-digests split-log nostrip binpkg-docompress strict preserve-libs multilib-strict network-sandbox protect-owned ebuild-locks unknown-features-warn fixlafiles merge-sync parallel-fetch"

I wonder if Gentoo has /usr/bin/python pointing to Python 3? That might explain it.

(Note: RHEL/CentOS and SUSE have a policy that /usr/bin/python will continue to point to Python 2 even after the Python 2 deprecation date.)

Michael: I have no idea how Portage builds Ceph or where I would look if I wanted to find out. Do you?

If the resulting Ceph binaries are supposed to use Python 3, you have to provide (at least) the following options to cmake when building:

-DWITH_PYTHON2=OFF -DWITH_PYTHON3=ON -DMGR_PYTHON_VERSION=3

By default, cmake will set up a Python 2 build. If you combine a Python 2 ceph build with /usr/bin/python pointing to Python 3, I would expect problems (possibly like this one) to ensue.

You can see the complete packaging steps here: https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-cluster/ceph/ceph-14.2.1.ebuild

Note that Gentoo handles it packaging very differently than RedHat / Debian and so on. Packages are built, from sources, at the time of installation. So ceph is compiled using the libraries, and versions thereof, and settings that I have installed on my system at the time of package installation. (Binary packages are supported, but most people don't use them).

So that ebuild file (which is a variant of a bash script, i believe) is interpreted on my computer, not on a build server. Hopefully that helps you understand how it works.

With great flexibility like this comes greater possibility of issues, of course.

Could very well be that this patch is causing a negative outcome : https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-cluster/ceph/files/ceph-14.2.0-mgr-python-version.patch

The relevant configuration values set by the ebuild file are provided by this function: ceph_src_configure

Which (among other things) does this:

local mycmakeargs=(
        DWITH_BABELTRACE=$(usex babeltrace)
        -DWITH_CEPHFS=$(usex cephfs)
        -DWITH_CEPHFS_SHELL=$(if python_is_python3; then usex cephfs; else echo OFF; fi)
        -DWITH_DPDK=$(usex dpdk)
        -DWITH_FUSE=$(usex fuse)
        -DWITH_LTTNG=$(usex lttng)
        -DWITH_GSSAPI=$(usex kerberos)
        -DWITH_GRAFANA=$(usex grafana)
        -DWITH_MGR=$(usex mgr)
        -DWITH_MGR_DASHBOARD_FRONTEND=NO
        -DWITH_NUMA=$(usex numa)
        -DWITH_OPENLDAP=$(usex ldap)
        -DWITH_PYTHON3=$(if python_is_python3; then echo "ON"; else echo "OFF"; fi)
        -DWITH_PYTHON2=$(if python_is_python3; then echo "OFF"; else echo "ON"; fi)
        -DWITH_RADOSGW=$(usex radosgw)
        -DWITH_RADOSGW_AMQP_ENDPOINT=$(usex rabbitmq)
        -DWITH_SSL=$(usex ssl)
        -DWITH_SYSTEMD=$(usex systemd)
        -DWITH_TESTS=$(usex test)
        -DWITH_XFS=$(usex xfs)
        -DWITH_ZFS=$(usex zfs)
        -DENABLE_SHARED=$(usex static-libs '' 'ON' 'OFF')
        -DALLOCATOR=$(usex tcmalloc 'tcmalloc' "$(usex jemalloc 'jemalloc' 'libc')")
        -DWITH_SYSTEM_BOOST=$(usex system-boost)
        -DBOOST_J=$(makeopts_jobs)
        -DWITH_RDMA=no
        -DWITH_TBB=no
        -DSYSTEMD_UNITDIR=$(systemd_get_systemunitdir)
        -DEPYTHON_VERSION="${EPYTHON#python}" 
        -DCMAKE_INSTALL_DOCDIR="${EPREFIX}/usr/share/doc/${PN}${PVR}" 
        -DCMAKE_INSTALL_SYSCONFDIR="${EPREFIX}/etc" 
        #-Wno-dev

It's not immediately clear to me what:

-DWITH_PYTHON3=$(if python_is_python3; then echo "ON"; else echo "OFF"; fi)
        -DWITH_PYTHON2=$(if python_is_python3; then echo "OFF"; else echo "ON"; fi)

Does, in terms of whether python2 || python3 -> python_is_python3 == true, or not.

However, I do have ceph configured (as per my system defaults, the current distribution defaults) to build for both python2.7 and python3.6 (with 3.5 and 3.7 available, but considered not-fully-tested by the distribution maintainers)

Could be that the ebuild file has a bug in it where python2 is what ceph is told to build for, despite my runtime-default being python3.

That being said, I did change the runtime default python and restart all ceph-processes, and ended up with the same result, so it's unfortunately a bit more complicated than simply switching the python version.

On a gentoo system, /usr/bin/python points to

mimir ~ # ls -lah /usr/bin/python
lrwxrwxrwx 1 root root 13 May 23 23:37 /usr/bin/python -> python-exec2c
mimir ~ # /usr/bin/python-exec2c --help
Usage: /usr/bin/python-exec2c <script>

python-exec is a wrapper to run Python scripts in an environment
supporting parallel install of multiple Python implementations.
For more information, please see the included README file.

Additional options:
--help, -h print this help message
--list-implementations, -l list supported implementations
--version, -V print the package name and version

---

Does ceph change the python code that's deployed for this feature based on the python2 vs python3 configuration option?

If so, maybe I can just recompile to fix this on my end.

Is it possible for the associated code here to be improved so that it dynamically uses dict.items vs dict.itervalues? That would solve the issue for me, as well as reduce the number of customization points that ceph needs to account for.

What's going on with -DMGR_PYTHON_VERSION ? Does that default to ONLY python2? Or does it default to one of the other configuration values?

What's going on with -DMGR_PYTHON_VERSION ? Does that default to ONLY python2? Or does it default to one of the other configuration values?

If you have a Python 3 build (which it sounds like you're trying to have), you would need to set "-DMGR_PYTHON_VERSION=3" explicitly, which it looks like you're not.

Created bug report with Gentoo: https://bugs.gentoo.org/687946

This SCR can be closed, since it's not a ceph project specific bug.