Bug #40206

pybind/mgr: module 'pg_autoscaler' has failed: 'dict' object has no attribute 'itervalues'

Added by Michael Jones 12 months ago. Updated 12 months ago.

Target version:
% Done:


3 - minor
Affected Versions:
Pull request ID:
Crash signature:


Is this functionality only available with python 2?


#1 Updated by Patrick Donnelly 12 months ago

  • Project changed from Ceph to mgr
  • Subject changed from Module 'pg_autoscaler' has failed: 'dict' object has no attribute 'itervalues' to pybind/mgr: module 'pg_autoscaler' has failed: 'dict' object has no attribute 'itervalues'
  • Start date deleted (06/07/2019)

What version of Python are you using?

#2 Updated by Michael Jones 12 months ago

I have both python 2.7 and python 3.6 installed, but python3.6 should be the default python interpreter used. Python 2 will be end-of-life on Jan 1st, 2020 :

The error message is indicative of the python code for this feature only working with python2, as the "itervalues" attribute of the "dict" object was replaced with "items" in python3 :

mimir ~ # eselect python list
Available Python interpreters, in order of preference:
[1] python3.6
[2] python2.7

mimir ~ # ceph -s
id: 07cb5105-68ea-4f1c-bace-a2be0baae5fa
health: HEALTH_ERR
Module 'pg_autoscaler' has failed: 'dict' object has no attribute 'itervalues'

mon: 3 daemons, quorum fenrir,mimir,hoenir (age 6h)
mgr: hoenir(active, since 20h), standbys: mimir, fenrir
mds: fileserver_backup:1 {0=hoenir=up:active} 2 up:standby
osd: 16 osds: 16 up, 16 in
pools: 2 pools, 1500 pgs
objects: 3.27M objects, 9.8 TiB
usage: 9.8 TiB used, 38 TiB / 48 TiB avail
pgs: 1497 active+clean
3 active+clean+scrubbing

mimir ~ # emerge --info ceph
Portage 2.3.66 (python 3.6.5-final-0, default/linux/amd64/17.0, gcc-8.3.0, glibc-2.29-r2, 4.19.44-gentoo x86_64) =================================================================
System Settings =================================================================
System uname: Linux-4.19.44-gentoo-x86_64-AMD_E-350D_APU_with_Radeon-tm-_HD_Graphics-with-gentoo-2.6
KiB Mem: 16132484 total, 222260 free
KiB Swap: 0 total, 0 free
Timestamp of repository gentoo: Mon, 03 Jun 2019 16:24:41 +0000
Head commit of repository gentoo: 6463dc05c45b01ca9e81e27b0d3492fcfb026827

Head commit of repository jonesmz-public-overlay: 86e790188a0acbd99656428bf8ce1833f3a9fbe2

Head commit of repository steam-overlay: 5314904baf6dcb2e33d0b20a3222d28d8b7de7ba

sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.31.1 p7) 2.31.1
distcc 3.3.2 x86_64-pc-linux-gnu [disabled]
app-shells/bash: 4.4_p23-r1::gentoo
dev-lang/perl: 5.28.2-r1::gentoo
dev-lang/python: 2.7.15::gentoo, 3.6.5::gentoo
dev-util/cmake: 3.14.3::gentoo
dev-util/pkgconfig: 0.29.2::gentoo
sys-apps/baselayout: 2.6-r1::gentoo
sys-apps/sandbox: 2.13::gentoo
sys-devel/autoconf: 2.69-r4::gentoo
sys-devel/automake: 1.11.6-r3::gentoo, 1.16.1-r1::gentoo
sys-devel/binutils: 2.31.1-r6::gentoo
sys-devel/gcc: 8.3.0-r1::gentoo
sys-devel/gcc-config: 2.0::gentoo
sys-devel/libtool: 2.4.6-r3::gentoo
sys-devel/make: 4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.14-r1::gentoo (virtual/os-headers)
sys-libs/glibc: 2.29-r2::gentoo

location: /usr/portage
sync-type: git
sync-uri: git://
priority: -1000

location: /usr/portage-overlays/jonesmz-public-overlay
sync-type: git
masters: gentoo

location: /usr/portage-overlays/steam-overlay
sync-type: git
masters: gentoo
priority: 50

Installed sets: @pc-base-system, @portage
CFLAGS="-O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe"
EMERGE_DEFAULT_OPTS=" --jobs --keep-going --newuse --deep --tree --backtrack=3000 --complete-graph --with-bdeps=y --binpkg-respect-use=y --binpkg-changed-deps=y --usepkg=y --usepkg"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg clean-logs compress-build-logs compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles installsources ipc-sandbox merge-sync multilib-strict network-sandbox news nostrip parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned sandbox sfperms split-elog split-log strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en en_US"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
USE="acl amd64 avahi btrfs bzip2 clang crypt cxx dbus gd gudev hardened iconv ipv6 libtirpc lm_sensors multilib ncurses nls nptl openmp pam pcre pie python readline samba seccomp ssl ssp systemd threads udev udisks unicode v4l xattr xtpax zeroconf zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core authz_host dir mime unixd socache_shmcb info log_config" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx sse sse2 mmxext" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="coreboot efi-64 emu qemu pc" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en en-US" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-1" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" QEMU_SOFTMMU_TARGETS="arm aarch64 x86_64" QEMU_USER_TARGETS="arm aarch64 x86_64" RUBY_TARGETS="ruby24" USERLAND="GNU" VIDEO_CARDS="r600 radeon radeonsi amdgpu vesa modesetting fbdev qxl" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

Package Settings =================================================================

sys-cluster/ceph-14.2.1::gentoo was built with the following:
USE="cephfs fuse mgr radosgw ssl systemd tcmalloc -babeltrace -dpdk -grafana -jemalloc -kerberos -ldap -libressl -lttng -numa -rabbitmq (-static-libs) (-system-boost) -test -xfs -zfs" ABI_X86="(64)" CPU_FLAGS_X86="sse sse2 -sse3 -sse4_1 -sse4_2 -ssse3" PYTHON_TARGETS="python2_7 python3_6 -python3_5 -python3_7"
FEATURES="distlocks sandbox xattr userpriv ipc-sandbox pid-sandbox usersync buildpkg binpkg-dostrip config-protect-if-modified compress-build-logs compressdebug binpkg-logs unmerge-logs split-elog userfetch parallel-install sfperms unmerge-orphans binpkg-multi-instance installsources clean-logs news usersandbox assume-digests split-log nostrip binpkg-docompress strict preserve-libs multilib-strict network-sandbox protect-owned ebuild-locks unknown-features-warn fixlafiles merge-sync parallel-fetch"

#3 Updated by Nathan Cutler 12 months ago

I wonder if Gentoo has /usr/bin/python pointing to Python 3? That might explain it.

(Note: RHEL/CentOS and SUSE have a policy that /usr/bin/python will continue to point to Python 2 even after the Python 2 deprecation date.)

#4 Updated by Nathan Cutler 12 months ago

Michael: I have no idea how Portage builds Ceph or where I would look if I wanted to find out. Do you?

If the resulting Ceph binaries are supposed to use Python 3, you have to provide (at least) the following options to cmake when building:


By default, cmake will set up a Python 2 build. If you combine a Python 2 ceph build with /usr/bin/python pointing to Python 3, I would expect problems (possibly like this one) to ensue.

#5 Updated by Michael Jones 12 months ago

You can see the complete packaging steps here:

Note that Gentoo handles it packaging very differently than RedHat / Debian and so on. Packages are built, from sources, at the time of installation. So ceph is compiled using the libraries, and versions thereof, and settings that I have installed on my system at the time of package installation. (Binary packages are supported, but most people don't use them).

So that ebuild file (which is a variant of a bash script, i believe) is interpreted on my computer, not on a build server. Hopefully that helps you understand how it works.

With great flexibility like this comes greater possibility of issues, of course.

Could very well be that this patch is causing a negative outcome :

The relevant configuration values set by the ebuild file are provided by this function: ceph_src_configure

Which (among other things) does this:

local mycmakeargs=(
-DWITH_BABELTRACE=$(usex babeltrace)
-DWITH_CEPHFS=$(usex cephfs)
-DWITH_CEPHFS_SHELL=$(if python_is_python3; then usex cephfs; else echo OFF; fi)
-DWITH_DPDK=$(usex dpdk)
-DWITH_FUSE=$(usex fuse)
-DWITH_LTTNG=$(usex lttng)
-DWITH_GSSAPI=$(usex kerberos)
-DWITH_GRAFANA=$(usex grafana)
-DWITH_MGR=$(usex mgr)
-DWITH_NUMA=$(usex numa)
-DWITH_OPENLDAP=$(usex ldap)
-DWITH_PYTHON3=$(if python_is_python3; then echo "ON"; else echo "OFF"; fi)
-DWITH_PYTHON2=$(if python_is_python3; then echo "OFF"; else echo "ON"; fi)
-DWITH_RADOSGW=$(usex radosgw)
-DWITH_SSL=$(usex ssl)
-DWITH_SYSTEMD=$(usex systemd)
-DWITH_TESTS=$(usex test)
-DWITH_XFS=$(usex xfs)
-DWITH_ZFS=$(usex zfs)
-DENABLE_SHARED=$(usex static-libs '' 'ON' 'OFF')
-DALLOCATOR=$(usex tcmalloc 'tcmalloc' "$(usex jemalloc 'jemalloc' 'libc')")
-DWITH_SYSTEM_BOOST=$(usex system-boost)

It's not immediately clear to me what:

-DWITH_PYTHON3=$(if python_is_python3; then echo "ON"; else echo "OFF"; fi)
-DWITH_PYTHON2=$(if python_is_python3; then echo "OFF"; else echo "ON"; fi)

Does, in terms of whether python2 || python3 -> python_is_python3 == true, or not.

However, I do have ceph configured (as per my system defaults, the current distribution defaults) to build for both python2.7 and python3.6 (with 3.5 and 3.7 available, but considered not-fully-tested by the distribution maintainers)

Could be that the ebuild file has a bug in it where python2 is what ceph is told to build for, despite my runtime-default being python3.

That being said, I did change the runtime default python and restart all ceph-processes, and ended up with the same result, so it's unfortunately a bit more complicated than simply switching the python version.

On a gentoo system, /usr/bin/python points to

mimir ~ # ls -lah /usr/bin/python
lrwxrwxrwx 1 root root 13 May 23 23:37 /usr/bin/python -> python-exec2c
mimir ~ # /usr/bin/python-exec2c --help
Usage: /usr/bin/python-exec2c <script>

python-exec is a wrapper to run Python scripts in an environment
supporting parallel install of multiple Python implementations.
For more information, please see the included README file.

Additional options:
--help, -h print this help message
--list-implementations, -l list supported implementations
--version, -V print the package name and version

Does ceph change the python code that's deployed for this feature based on the python2 vs python3 configuration option?

If so, maybe I can just recompile to fix this on my end.

Is it possible for the associated code here to be improved so that it dynamically uses dict.items vs dict.itervalues? That would solve the issue for me, as well as reduce the number of customization points that ceph needs to account for.

What's going on with -DMGR_PYTHON_VERSION ? Does that default to ONLY python2? Or does it default to one of the other configuration values?

#6 Updated by Nathan Cutler 12 months ago

What's going on with -DMGR_PYTHON_VERSION ? Does that default to ONLY python2? Or does it default to one of the other configuration values?

If you have a Python 3 build (which it sounds like you're trying to have), you would need to set "-DMGR_PYTHON_VERSION=3" explicitly, which it looks like you're not.

#7 Updated by Michael Jones 12 months ago

Created bug report with Gentoo:

This SCR can be closed, since it's not a ceph project specific bug.

#8 Updated by Nathan Cutler 12 months ago

  • Status changed from New to Closed

Also available in: Atom PDF