Ceph » devops

@Boris Ranto ken said to ask you to merge this PR https://github.com/ceph/ceph/pull/6773
Pls let me know when I can test the fix.

(pls mark duplicate after it's done)

Also happens in infernalis see http://pulpito.ceph.com/loic-2016-02-04_00:33:20-rados-infernalis-backports---basic-multi/6132/

The core of the problem is that some packages from v0.80.7 is installed by ansible even before the test starts.

TASK: [testnode | Install packages] ******************************************* 

2016-02-04T00:49:15.306 INFO:teuthology.task.ansible.out:[0;33mchanged: [smithi006.front.sepia.ceph.com] => (item=@core,@base,yum-plugin-priorities,yum-plugin-fastestmirror,redhat-lsb,sysstat,gdb,git-all,python-configobj,libedit,openssl098e,boost-thread,xfsprogs,gdisk,parted,libgcrypt,fuse,fuse-libs,python-virtualenv,openssl,libuuid,btrfs-progs,attr,valgrind,python-nose,mpich,ant,iozone,libtool,automake,gettext,libuuid-devel,libacl-devel,bc,xfsdump,xfsprogs-devel,blktrace,numpy,python-matplotlib,qemu-kvm,usbredir,genisoimage,httpd,httpd-devel,httpd-tools,mod_ssl,mod_fastcgi-2.4.7-1.ceph.el7.centos,libevent-devel,perl-XML-Twig,java-1.6.0-openjdk-devel,junit4,smartmontools,nfs-utils) => {"changed": true, "item": "@core,@base,yum-plugin-priorities,yum-plugin-fastestmirror,redhat-lsb,sysstat,gdb,git-all,python-configobj,libedit,openssl098e,boost-thread,xfsprogs,gdisk,parted,libgcrypt,fuse,fuse-libs,python-virtualenv,openssl,libuuid,btrfs-progs,attr,valgrind,python-nose,mpich,ant,iozone,libtool,automake,gettext,libuuid-devel,libacl-devel,bc,xfsdump,xfsprogs-devel,blktrace,numpy,python-matplotlib,qemu-kvm,usbredir,genisoimage,httpd,httpd-devel,httpd-tools,mod_ssl,mod_fastcgi-2.4.7-1.ceph.el7.centos,libevent-devel,perl-XML-Twig,java-1.6.0-openjdk-devel,junit4,smartmontools,nfs-utils", "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-34.el7.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-34.el7.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.centos.1.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-7.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-80.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-6.el7.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7.centos.2.x86_64 providing openssl098e is 
2016-02-04T00:49:15.308 INFO:teuthology.task.ansible.out:already installed", "boost-thread-1.53.0-25.el7.x86_64 providing boost-thread is already installed", "xfsprogs-3.2.2-2.el7.x86_64 providing xfsprogs is already installed", "gdisk-0.8.6-5.el7.x86_64 providing gdisk is already installed", "parted-3.1-23.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-12.el7_1.1.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-6.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-6.el7.x86_64 providing fuse-libs is already installed", "python-virtualenv-1.10.1-2.el7.noarch providing python-virtualenv is already installed", "openssl-1.0.1e-42.el7.9.x86_64 providing openssl is already installed", "libuuid-2.23.2-26.el7.x86_64 providing libuuid is already installed", "btrfs-progs-3.19.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-12.el7.x86_64 providing attr is already installed", "valgrind-3.10.0-16.el7.x86_64 providing valgrind is already installed", "python-nose-1.3.0-3.el7.noarch providing python-nose is already installed", "mpich-3.0.4-8.el7.x86_64 providing mpich is already installed", "ant-1.9.2-9.el7.noarch providing ant is already installed", "iozone-3.424-2_ceph.el7.centos.x86_64 providing iozone is already installed", "libtool-2.4.2-20.el7.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.18.2.1-4.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-26.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-12.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.4-1.el7.x86_64 providing xfsdump is already installed", "xfsprogs-devel-3.2.2-2.el7.x86_64 providing xfsprogs-devel is already installed", "blktrace-1.0.5-6.el7.x86_64 providing blktrace is already installed", "numpy-1.7.1-11.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2
2016-02-04T00:49:15.310 INFO:teuthology.task.ansible.out:.0-15.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.6-7.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-23.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-40.el7.centos.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-40.el7.centos.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-40.el7.centos.x86_64 providing httpd-tools is already installed", "mod_ssl-2.4.6-40.el7.centos.x86_64 providing mod_ssl is already installed", "mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 providing mod_fastcgi-2.4.7-1.ceph.el7.centos is already installed", "libevent-devel-2.0.21-4.el7.x86_64 providing libevent-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "java-1.6.0-openjdk-devel-1.6.0.38-1.13.10.0.el7_2.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "smartmontools-6.2-4.el7.x86_64 providing smartmontools is already installed", "nfs-utils-1.3.0-0.21.el7.x86_64 providing nfs-utils is already installed", "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.vcu.edu\n * epel: fedora-epel.mirror.lstn.net\n * extras: mirror.cs.pitt.edu\n * updates: ftp.linux.ncsu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package qemu-kvm.x86_64 10:1.5.3-105.el7_2.3 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-105.el7_2.3 for package: 10:qemu-kvm-1.5.3-105.el7_2.3.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-105.el7_2.3 for package: 10:qemu-kvm-1.5.3-105.el7_2.3.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-105.el7_2.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-105.el7_2.3.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:0.80.7-3.el7
2016-02-04T00:49:15.312 INFO:teuthology.task.ansible.out: will be installed\n---> Package librbd1.x86_64 1:0.80.7-3.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-105.el7_2.3 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-105.el7_2.3 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package              Arch        Version                    Repository    Size\n================================================================================\nInstalling:\n qemu-kvm             x86_64      10:1.5.3-105.el7_2.3       updates      1.8 M\nInstalling for dependencies:\n librados2            x86_64      1:0.80.7-3.el7             base         1.5 M\n librbd1              x86_64      1:0.80.7-3.el7             base         350 k\n qemu-img             x86_64      10:1.5.3-105.el7_2.3       updates      657 k\n qemu-kvm-common      x86_64      10:1.5.3-105.el7_2.3       updates      391 k\n\nTransaction Summary\n================================================================================\nInstall  1 Package (+4 Dependent packages)\n\nTotal download size: 4.7 M\nInstalled size: 15 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal                                              4.5 MB/s | 4.7 MB  00:01     \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n  Installing : 1:librados2-0.80.7-3.el7.x86_64                              1/5 \n  Installing : 1:librbd1-0.80.7-3.el7.x86_64                                2/5 \n  Installing : 10:qemu-img-1.5.3-105.el7_2.3.x86_64                         3/5 \n  Installing : 10:qemu-kvm-common-1.5.3-105.el7_2.3.x86_64                  4/5 \n  Installing : 10:qemu-kvm-1.5.3-105.el7_2.3.x86_64                         5/5 \n  Verifying  : 10:qemu-img-1.5.3-105.el7_2.3.x86_64                         1/5 \n  Verifying  : 10:qemu-kvm-1.5.3-10
2016-02-04T00:49:15.314 INFO:teuthology.task.ansible.out:5.el7_2.3.x86_64                         2/5 \n  Verifying  : 1:librbd1-0.80.7-3.el7.x86_64                                3/5 \n  Verifying  : 1:librados2-0.80.7-3.el7.x86_64                              4/5 \n  Verifying  : 10:qemu-kvm-common-1.5.3-105.el7_2.3.x86_64                  5/5 \n\nInstalled:\n  qemu-kvm.x86_64 10:1.5.3-105.el7_2.3                                          \n\nDependency Installed:\n  librados2.x86_64 1:0.80.7-3.el7                                               \n  librbd1.x86_64 1:0.80.7-3.el7                                                 \n  qemu-img.x86_64 10:1.5.3-105.el7_2.3                                          \n  qemu-kvm-common.x86_64 10:1.5.3-105.el7_2.3                                   \n\nComplete!\n"]}[0m

And upgrading to v0.80.8 fails, presumably because of packaging problems.

2016-02-04T00:51:13.336 INFO:teuthology.orchestra.run.smithi006.stderr:Error: Package: ceph-common-0.80.8-0.el7.x86_64 (Ceph)
2016-02-04T00:51:13.337 INFO:teuthology.orchestra.run.smithi006.stderr:           Requires: librbd1 = 0.80.8-0.el7
2016-02-04T00:51:13.337 INFO:teuthology.orchestra.run.smithi006.stderr:           Installed: 1:librbd1-0.80.7-3.el7.x86_64 (@base)
2016-02-04T00:51:13.337 INFO:teuthology.orchestra.run.smithi006.stderr:               librbd1 = 1:0.80.7-3.el7
2016-02-04T00:51:13.337 INFO:teuthology.orchestra.run.smithi006.stderr:           Available: librbd1-0.80.8-0.el7.x86_64 (Ceph)
2016-02-04T00:51:13.337 INFO:teuthology.orchestra.run.smithi006.stderr:               librbd1 = 0.80.8-0.el7

@Loïc Dachary: librbd is a dependency of qemu so ansible probably installs qemu(-kvm) which in-turn installs librbd1 and librados2. However, we should be good 2 go in infernalis since infernalis packages do update these old firefly packages. Even the latest firefly packages should do that but the latest means .12 (which does not exist, yet so essentially the current firefly branch), not .8.

As for the other issue, that should not be related to this bug as it happened on a debian/ubuntu (apt-based) platform.

Marking Resolved as we have already fixed this in firefly (at least for rhel/epel).