Project

General

Profile

Actions
Copy link

Bug #12005

closed

yum crash: "ValueError: need more than 2 values to unpack"

Added by Yuri Weinstein almost 9 years ago. Updated about 8 years ago.

Status:
Won't Fix
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
v0.52
ceph-qa-suite:
krbd
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Lots of jobs failed, here is one example

Run: http://pulpito.ceph.redhat.com/teuthology-2015-06-12_18:06:01-krbd-hammer-testing-basic-magna/
Logs: http://magna002.ceph.redhat.com/teuthology-2015-06-12_18:06:01-krbd-hammer-testing-basic-magna/130456/

2015-06-13T11:59:08.195 INFO:teuthology.orchestra.run.magna062:Running: 'uname -r'
2015-06-13T11:59:08.201 DEBUG:teuthology.task.kernel:current kernel version is 3.13.0-54-generic
2015-06-13T11:59:08.202 DEBUG:teuthology.task.kernel:extracting sha1, 3.13.0-54-generic -> eneric
2015-06-13T11:59:08.206 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@magna023.ceph.redhat.com
2015-06-13T11:59:08.207 INFO:teuthology.orchestra.run.magna023:Running: 'sudo package-cleanup -y --oldkernels'
2015-06-13T11:59:08.423 INFO:teuthology.orchestra.run.magna023.stdout:Loaded plugins: priorities, product-id
2015-06-13T11:59:08.561 INFO:teuthology.orchestra.run.magna023.stdout:Not removing kernel 3.10.0-229.4.2.el7 because it is the running kernel
2015-06-13T11:59:08.562 INFO:teuthology.orchestra.run.magna023.stdout:No old kernels to remove
2015-06-13T11:59:08.575 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@magna117.ceph.redhat.com
2015-06-13T11:59:08.576 INFO:teuthology.orchestra.run.magna117:Running: 'sudo package-cleanup -y --oldkernels'
2015-06-13T11:59:08.805 INFO:teuthology.orchestra.run.magna117.stdout:Loaded plugins: priorities, product-id
2015-06-13T11:59:09.058 INFO:teuthology.orchestra.run.magna117.stdout:There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help).
2015-06-13T11:59:12.073 INFO:teuthology.orchestra.run.magna117.stdout:No old kernels to remove
2015-06-13T11:59:12.089 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@magna039.ceph.redhat.com
2015-06-13T11:59:12.121 INFO:teuthology.orchestra.run.magna039:Running: 'sudo package-cleanup -y --oldkernels'
2015-06-13T11:59:12.335 INFO:teuthology.orchestra.run.magna039.stdout:Loaded plugins: priorities, product-id
2015-06-13T11:59:12.586 INFO:teuthology.orchestra.run.magna039.stdout:Not removing kernel 3.10.0-229.4.2.el7 because it is the running kernel
2015-06-13T11:59:12.591 INFO:teuthology.orchestra.run.magna039.stdout:There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help).
2015-06-13T11:59:15.601 INFO:teuthology.orchestra.run.magna039.stdout:No old kernels to remove
2015-06-13T11:59:15.873 INFO:teuthology.orchestra.run.magna023:Running: 'sudo yum upgrade -y linux-firmware'
2015-06-13T11:59:16.108 INFO:teuthology.orchestra.run.magna023.stdout:Loaded plugins: priorities, product-id, subscription-manager
2015-06-13T11:59:17.974 INFO:teuthology.orchestra.run.magna023.stderr:Traceback (most recent call last):
2015-06-13T11:59:18.014 INFO:teuthology.orchestra.run.magna023.stderr:  File "/bin/yum", line 29, in <module>
2015-06-13T11:59:18.015 INFO:teuthology.orchestra.run.magna023.stderr:    yummain.user_main(sys.argv[1:], exit_code=True)
2015-06-13T11:59:18.016 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/share/yum-cli/yummain.py", line 367, in user_main
2015-06-13T11:59:18.017 INFO:teuthology.orchestra.run.magna023.stderr:    errcode = main(args)
2015-06-13T11:59:18.058 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/share/yum-cli/yummain.py", line 174, in main
2015-06-13T11:59:18.059 INFO:teuthology.orchestra.run.magna023.stderr:    result, resultmsgs = base.doCommands()
2015-06-13T11:59:18.100 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/share/yum-cli/cli.py", line 571, in doCommands
2015-06-13T11:59:18.101 INFO:teuthology.orchestra.run.magna023.stderr:    return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
2015-06-13T11:59:18.141 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/share/yum-cli/yumcommands.py", line 1794, in doCommand
2015-06-13T11:59:18.142 INFO:teuthology.orchestra.run.magna023.stderr:    ret = base.updatePkgs(extcmds, update_to=(basecmd == 'upgrade-to'))
2015-06-13T11:59:18.143 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/share/yum-cli/cli.py", line 1025, in updatePkgs
2015-06-13T11:59:18.184 INFO:teuthology.orchestra.run.magna023.stderr:    txmbrs = self.update(pattern=item, update_to=update_to)
2015-06-13T11:59:18.185 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 5162, in update
2015-06-13T11:59:18.186 INFO:teuthology.orchestra.run.magna023.stderr:    m = self.pkgSack.returnNewestByNameArch(patterns=pats)
2015-06-13T11:59:18.187 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1079, in <lambda>
2015-06-13T11:59:18.227 INFO:teuthology.orchestra.run.magna023.stderr:    pkgSack = property(fget=lambda self: self._getSacks(),
2015-06-13T11:59:18.228 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 784, in _getSacks
2015-06-13T11:59:18.269 INFO:teuthology.orchestra.run.magna023.stderr:    self.repos.populateSack(which=repos)
2015-06-13T11:59:18.270 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 344, in populateSack
2015-06-13T11:59:18.310 INFO:teuthology.orchestra.run.magna023.stderr:    self.doSetup()
2015-06-13T11:59:18.311 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
2015-06-13T11:59:18.312 INFO:teuthology.orchestra.run.magna023.stderr:    self.retrieveAllMD()
2015-06-13T11:59:18.313 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
2015-06-13T11:59:18.353 INFO:teuthology.orchestra.run.magna023.stderr:    dl = repo._async and repo._commonLoadRepoXML(repo)
2015-06-13T11:59:18.354 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1456, in _commonLoadRepoXML
2015-06-13T11:59:18.395 INFO:teuthology.orchestra.run.magna023.stderr:    result = self._getFileRepoXML(local, text)
2015-06-13T11:59:18.396 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1234, in _getFileRepoXML
2015-06-13T11:59:18.397 INFO:teuthology.orchestra.run.magna023.stderr:    size=102400) # setting max size as 100K
2015-06-13T11:59:18.398 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1014, in _getFile
2015-06-13T11:59:18.439 INFO:teuthology.orchestra.run.magna023.stderr:    result = self.grab.urlgrab(misc.to_utf8(relative), local,
2015-06-13T11:59:18.439 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 694, in <lambda>
2015-06-13T11:59:18.440 INFO:teuthology.orchestra.run.magna023.stderr:    grab = property(lambda self: self._getgrab())
2015-06-13T11:59:18.441 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 689, in _getgrab
2015-06-13T11:59:18.442 INFO:teuthology.orchestra.run.magna023.stderr:    self._setupGrab()
2015-06-13T11:59:18.443 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 655, in _setupGrab
2015-06-13T11:59:18.444 INFO:teuthology.orchestra.run.magna023.stderr:    failure_callback=mirror_failure)
2015-06-13T11:59:18.485 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 275, in __init__
2015-06-13T11:59:18.486 INFO:teuthology.orchestra.run.magna023.stderr:    self.mirrors.sort(key=estimate, reverse=True)
2015-06-13T11:59:18.526 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 269, in estimate
2015-06-13T11:59:18.527 INFO:teuthology.orchestra.run.magna023.stderr:    speed, fail = _TH.estimate(m['mirror'])
2015-06-13T11:59:18.528 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2417, in estimate
2015-06-13T11:59:18.569 INFO:teuthology.orchestra.run.magna023.stderr:    _TH.load()
2015-06-13T11:59:18.570 INFO:teuthology.orchestra.run.magna023.stderr:  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2366, in load
2015-06-13T11:59:18.610 INFO:teuthology.orchestra.run.magna023.stderr:    host, speed, fail, ts = line.rsplit(' ', 3)
2015-06-13T11:59:18.611 INFO:teuthology.orchestra.run.magna023.stderr:ValueError: need more than 2 values to unpack
2015-06-13T11:59:18.654 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 53, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 41, in run_one_task
    return fn(**kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/kernel.py", line 1208, in task
    install_firmware(ctx, need_install)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/kernel.py", line 199, in install_firmware
    'sudo', 'yum', 'upgrade', '-y', 'linux-firmware',
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 156, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on magna023 with status 1: 'sudo yum upgrade -y linux-firmware'
Actions
Copy link
 #1

Updated by Zack Cerza almost 9 years ago

We talked about this last week. It's a bug in yum that breaks it entirely. You can see in the log you pasted that it's a crash in yum, not just sudo yum upgrade -y linux-firmware being broken.

The workaround seems to be:
find /var/cache/yum -name 'timedhosts' -exec rm {} \;

I'm running it on all the magnas now.

Actions
Copy link
 #2

Updated by Zack Cerza almost 9 years ago

  • Subject changed from 'sudo yum upgrade -y linux-firmware' broken in octo lab to yum crash: "ValueError: need more than 2 values to unpack"
Actions
Copy link
 #4

Updated by Yuri Weinstein almost 9 years ago

Same lots of jobs failing with this, for example run:
http://pulpito.ceph.redhat.com/teuthology-2015-06-23_21:00:01-rados-hammer-distro-basic-magna/

It's odd that I did not see similar failures in Sepia lab.
And the reason for this - we don't have RHEL nodes in Sepia

Actions
Copy link
 #5

Updated by Sage Weil about 8 years ago

  • Status changed from New to Won't Fix
Actions
Copy link

Also available in: Atom PDF