Tools » ceph-cm-ansible

/a/samuelj-2016-03-24_10:43:33-rados-wip-sam-testing-distro-basic-smithi/84082

marking smithi003 down

~ » ssh ubuntu@smithi003.front.sepia.ceph.com
Warning: Permanently added 'smithi003.front.sepia.ceph.com,172.21.15.3' (ECDSA) to the list of known hosts.
Last login: Thu Mar 24 19:15:37 2016 from gw.rdu.sepia.ceph.com
[ubuntu@smithi003 ~]$ sudo yum d 2 -y remove boost-random boost-program-options leveldb xmlstarlet python-jinja2 python-ceph python-flask python-requests boost-random python-urllib3 python-babel hdparm python-markupsafe python-werkzeug python-itsdangerous
Loaded plugins: fastestmirror, langpacks, priorities
No Match for argument: boost-program-options
No Match for argument: leveldb
No Match for argument: xmlstarlet
No Match for argument: python-jinja2
No Match for argument: python-ceph
No Match for argument: python-flask
No Match for argument: python-babel
No Match for argument: hdparm
No Match for argument: python-markupsafe
Traceback (most recent call last):
File "/bin/yum", line 29, in <module>
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 365, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 174, in main
result, resultmsgs = base.doCommands()
File "/usr/share/yum-cli/cli.py", line 573, in doCommands
return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
File "/usr/share/yum-cli/yumcommands.py", line 878, in doCommand
ret = base.erasePkgs(extcmds, pos=pos, basecmd=basecmd)
File "/usr/share/yum-cli/cli.py", line 1198, in erasePkgs
rms = self.remove(pattern=arg)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 5399, in remove
(e,m,u) = self.rpmdb.matchPackageNames([kwargs['pattern']])
File "/usr/lib/python2.7/site-packages/yum/packageSack.py", line 304, in matchPackageNames
exactmatch.append(self.searchPkgTuple(pkgtup)[0])
IndexError: list index out of range
[ubuntu@smithi003 ~]$ sudo yum -y remove boost-random boost-program-options leveldb xmlstarlet python-jinja2 python-ceph python-flask python-requests boost-random python-urllib3 python-babel hdparm python-markupsafe python-werkzeug python-itsdangerous
Loaded plugins: fastestmirror, langpacks, priorities
No Match for argument: boost-program-options
No Match for argument: leveldb
No Match for argument: xmlstarlet
No Match for argument: python-jinja2
No Match for argument: python-ceph
No Match for argument: python-flask
No Match for argument: python-babel
No Match for argument: hdparm
No Match for argument: python-markupsafe
Traceback (most recent call last):
File "/bin/yum", line 29, in <module>
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 365, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 174, in main
result, resultmsgs = base.doCommands()
File "/usr/share/yum-cli/cli.py", line 573, in doCommands
return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
File "/usr/share/yum-cli/yumcommands.py", line 878, in doCommand
ret = base.erasePkgs(extcmds, pos=pos, basecmd=basecmd)
File "/usr/share/yum-cli/cli.py", line 1198, in erasePkgs
rms = self.remove(pattern=arg)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 5399, in remove
(e,m,u) = self.rpmdb.matchPackageNames([kwargs['pattern']])
File "/usr/lib/python2.7/site-packages/yum/packageSack.py", line 304, in matchPackageNames
exactmatch.append(self.searchPkgTuple(pkgtup)[0])
IndexError: list index out of range
[ubuntu@smithi003 ~]$ sudo yum remove boost-random boost-program-options leveldb xmlstarlet python-jinja2 python-ceph python-flask python-requests boost-random python-urllib3 python-babel hdparm python-markupsafe python-werkzeug python-itsdangerous
Loaded plugins: fastestmirror, langpacks, priorities
No Match for argument: boost-program-options
No Match for argument: leveldb
No Match for argument: xmlstarlet
No Match for argument: python-jinja2
No Match for argument: python-ceph
No Match for argument: python-flask
No Match for argument: python-babel
No Match for argument: hdparm
No Match for argument: python-markupsafe
Traceback (most recent call last):
File "/bin/yum", line 29, in <module>
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 365, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 174, in main
result, resultmsgs = base.doCommands()
File "/usr/share/yum-cli/cli.py", line 573, in doCommands
return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
File "/usr/share/yum-cli/yumcommands.py", line 878, in doCommand
ret = base.erasePkgs(extcmds, pos=pos, basecmd=basecmd)
File "/usr/share/yum-cli/cli.py", line 1198, in erasePkgs
rms = self.remove(pattern=arg)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 5399, in remove
(e,m,u) = self.rpmdb.matchPackageNames([kwargs['pattern']])
File "/usr/lib/python2.7/site-packages/yum/packageSack.py", line 304, in matchPackageNames
exactmatch.append(self.searchPkgTuple(pkgtup)[0])
IndexError: list index out of range
[ubuntu@smithi003 ~]$ sudo yum remove boost-random
Loaded plugins: fastestmirror, langpacks, priorities
Resolving Dependencies
-> Running transaction check
---> Package boost-random.x86_64 0:1.53.0-25.el7 will be erased
--> Finished Dependency Resolution
base/7/x86_64 | 3.6 kB 00:00:00
base/7/x86_64/group_gz | 155 kB 00:00:00
base/7/x86_64/primary_db | 5.3 MB 00:00:01
centos7-fcgi-ceph | 951 B 00:00:00
centos7-fcgi-ceph/primary | 1.5 kB 00:00:00
epel | 4.3 kB 00:00:00
epel/group_gz | 169 kB 00:00:00
epel/updateinfo | 518 kB 00:00:00
epel/primary_db | 3.9 MB 00:00:00
extras/7/x86_64 | 3.4 kB 00:00:00
extras/7/x86_64/primary_db | 101 kB 00:00:00
lab-extras | 951 B 00:00:00
lab-extras/primary | 1.4 kB 00:00:00
updates/7/x86_64 | 3.4 kB 00:00:00
updates/7/x86_64/primary_db | 3.2 MB 00:00:00

Dependencies Resolved

=======================================================================================================================================================================================================
Package Arch Version Repository Size =======================================================================================================================================================================================================
Removing:
boost-random x86_64 1.53.0-25.el7 installed 24 k

Transaction Summary =======================================================================================================================================================================================================
Remove 1 Package

Installed size: 24 k
Is this ok [y/N]: Exiting on user command
Your transaction was saved, rerun it with:
yum load-transaction /tmp/yum_save_tx.2016-03-24.19-43.mmlour.yumtx
[ubuntu@smithi003 ~]$ sudo yum remove boost-random boost-program-options leveldb xmlstarlet python-jinja2 python-ceph python-flask python-requests boost-random python-urllib3 python-babel hdparm python-markupsafe python-werkzeug python-itsdangerous^C

ubuntu@smithi003 ~]$ sudo yum remove python-werkzeug
Loaded plugins: fastestmirror, langpacks, priorities
Traceback (most recent call last):
File "/bin/yum", line 29, in <module>
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 365, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 174, in main
result, resultmsgs = base.doCommands()
File "/usr/share/yum-cli/cli.py", line 573, in doCommands
return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
File "/usr/share/yum-cli/yumcommands.py", line 878, in doCommand
ret = base.erasePkgs(extcmds, pos=pos, basecmd=basecmd)
File "/usr/share/yum-cli/cli.py", line 1198, in erasePkgs
rms = self.remove(pattern=arg)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 5399, in remove
(e,m,u) = self.rpmdb.matchPackageNames([kwargs['pattern']])
File "/usr/lib/python2.7/site-packages/yum/packageSack.py", line 304, in matchPackageNames
exactmatch.append(self.searchPkgTuple(pkgtup)[0])
IndexError: list index out of range

removing python-workzeug from the list doesn't trigger the crash:

[ubuntu@smithi003 ~]$ sudo yum remove boost-random boost-program-options leveldb xmlstarlet python-jinja2 python-ceph python-flask python-requests boost-random python-urllib3 python-babel hdparm python-markupsafe python-werkzeug python-itsdangerous^C
[ubuntu@smithi003 ~]$ sudo yum remove python-werkzeug
Loaded plugins: fastestmirror, langpacks, priorities
Traceback (most recent call last):
File "/bin/yum", line 29, in <module>
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 365, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 174, in main
result, resultmsgs = base.doCommands()
File "/usr/share/yum-cli/cli.py", line 573, in doCommands
return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
File "/usr/share/yum-cli/yumcommands.py", line 878, in doCommand
ret = base.erasePkgs(extcmds, pos=pos, basecmd=basecmd)
File "/usr/share/yum-cli/cli.py", line 1198, in erasePkgs
rms = self.remove(pattern=arg)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 5399, in remove
(e,m,u) = self.rpmdb.matchPackageNames([kwargs['pattern']])
File "/usr/lib/python2.7/site-packages/yum/packageSack.py", line 304, in matchPackageNames
exactmatch.append(self.searchPkgTuple(pkgtup)[0])
IndexError: list index out of range
[ubuntu@smithi003 ~]$ sudo yum remove boost-random boost-program-options leveldb xmlstarlet python-jinja2 python-ceph python-flask python-requests boost-random python-urllib3 python-babel hdparm python-markupsafe python-itsdangerous
Loaded plugins: fastestmirror, langpacks, priorities
No Match for argument: boost-program-options
No Match for argument: leveldb
No Match for argument: xmlstarlet
No Match for argument: python-jinja2
No Match for argument: python-ceph
No Match for argument: python-flask
No Match for argument: python-babel
No Match for argument: hdparm
No Match for argument: python-markupsafe
No Match for argument: python-itsdangerous
Resolving Dependencies
--> Running transaction check
---> Package boost-random.x86_64 0:1.53.0-25.el7 will be erased
---> Package python-requests.noarch 0:2.6.0-1.el7_1 will be erased
---> Package python-urllib3.noarch 0:1.10.2-2.el7_1 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================================================================================================
Package Arch Version Repository Size =======================================================================================================================================================================================================
Removing:
boost-random x86_64 1.53.0-25.el7 installed 24 k
python-requests noarch 2.6.0-1.el7_1 installed 343 k
python-urllib3 noarch 1.10.2-2.el7_1 installed 369 k

Transaction Summary =======================================================================================================================================================================================================
Remove 3 Packages

Installed size: 735 k
Is this ok [y/N]: ^C^C^C^C^Cn
Exiting on user command
Your transaction was saved, rerun it with:
yum load-transaction /tmp/yum_save_tx.2016-03-24.19-48.C1fQAI.yumtx

Predictably, yum clean all doesn't fix it.

Ok, let's start digging into yum. Recklessly changing packageSack.py

[ubuntu@smithi003 ~]$ diff /usr/lib/python2.7/site-packages/yum/packageSack.py /tmp/packageSack.py 
304c304,311
<                     exactmatch.append(self.searchPkgTuple(pkgtup)[0])
---
>                     x = self.searchPkgTuple(pkgtup)
>                     y = None
>                     try:
>                         y = x[0]
>                     except Exception:
>                         print y
>                         raise
>                     exactmatch.append(y)

gives us

[ubuntu@smithi003 ~]$ sudo yum remove python-werkzeug
Loaded plugins: fastestmirror, langpacks, priorities
None
Traceback (most recent call last):
  File "/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 365, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 174, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 573, in doCommands
    return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
  File "/usr/share/yum-cli/yumcommands.py", line 878, in doCommand
    ret = base.erasePkgs(extcmds, pos=pos, basecmd=basecmd)
  File "/usr/share/yum-cli/cli.py", line 1198, in erasePkgs
    rms = self.remove(pattern=arg)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 5399, in remove
    (e,m,u) = self.rpmdb.matchPackageNames([kwargs['pattern']])
  File "/usr/lib/python2.7/site-packages/yum/packageSack.py", line 307, in matchPackageNames
    y = x[0]
IndexError: list index out of range

so apparently searchPkgTuple is returning None when we'd rather it didn't.

Perhaps related:

[ubuntu@smithi003 ~]$ rpm -q python-werkzeug
package python-werkzeug is not installed

vs

[ubuntu@smithi003 ~]$ sudo yum info python-werkzeug
Loaded plugins: fastestmirror, langpacks, priorities
Loading mirror speeds from cached hostfile
 * base: mirror.netdepot.com
 * epel: fedora-epel.mirror.lstn.net
 * extras: mirror.symnds.com
 * updates: mirror.us.leaseweb.net
Installed Packages
Name        : python-werkzeug
Arch        : noarch
Version     : 0.9.1
Release     : 2.el7
Size        : 2.8 M
Repo        : installed
Summary     : The Swiss Army knife of Python web development
URL         : http://werkzeug.pocoo.org/
License     : BSD
Description : Werkzeug
            : ========
            : 
            : Werkzeug started as simple collection of various utilities for WSGI
            : applications and has become one of the most advanced WSGI utility
            : modules.  It includes a powerful debugger, full featured request and
            : response objects, HTTP utilities to handle entity tags, cache control
            : headers, HTTP dates, cookie handling, file uploads, a powerful URL
            : routing system and a bunch of community contributed addon modules.
            : 
            : Werkzeug is unicode aware and doesn't enforce a specific template
            : engine, database adapter or anything else.  It doesn't even enforce
            : a specific way of handling requests and leaves all that up to the
            : developer. It's most useful for end user applications which should work
            : on as many server environments as possible (such as blogs, wikis,
            : bulletin boards, etc.).

So rpmdb and yum disagree. Interesting.