Actions
Bug #10515
closedchef / apt-get compete during parallel runs
Status:
Won't Fix
Priority:
Normal
Assignee:
Sandon Van Ness
Category:
-
Target version:
-
% Done:
0%
Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
http://pulpito.ceph.com/loic-2015-01-10_11:54:25-rados-giant-backports-testing-basic-vps/694228/
Maybe because parallel execution gets two competing installation jobs to run at the same time ?
2015-01-10T22:25:03.062 INFO:teuthology.orchestra.run.vpm142.stdout:[0m 2015-01-10T22:25:03.062 INFO:teuthology.orchestra.run.vpm142.stdout:================================================================================[0m 2015-01-10T22:25:03.062 INFO:teuthology.orchestra.run.vpm142.stdout:[31mError executing action `upgrade` on resource 'package[libapache2-mod-fastcgi]'[0m 2015-01-10T22:25:03.063 INFO:teuthology.orchestra.run.vpm142.stdout:================================================================================[0m 2015-01-10T22:25:03.063 INFO:teuthology.orchestra.run.vpm142.stdout: 2015-01-10T22:25:03.063 INFO:teuthology.orchestra.run.vpm142.stdout:[0m 2015-01-10T22:25:03.065 INFO:teuthology.orchestra.run.vpm142.stdout:Mixlib::ShellOut::ShellCommandFailed[0m 2015-01-10T22:25:03.065 INFO:teuthology.orchestra.run.vpm142.stdout:------------------------------------[0m 2015-01-10T22:25:03.065 INFO:teuthology.orchestra.run.vpm142.stdout:Expected process to exit with [0], but received '100' 2015-01-10T22:25:03.065 INFO:teuthology.orchestra.run.vpm142.stdout:---- Begin output of apt-get -q -y install libapache2-mod-fastcgi=2.4.7~0910052141-2~bpo70+1.ceph ---- 2015-01-10T22:25:03.066 INFO:teuthology.orchestra.run.vpm142.stdout:STDOUT: 2015-01-10T22:25:03.066 INFO:teuthology.orchestra.run.vpm142.stdout:STDERR: E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable) 2015-01-10T22:25:03.066 INFO:teuthology.orchestra.run.vpm142.stdout:E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it? 2015-01-10T22:25:03.066 INFO:teuthology.orchestra.run.vpm142.stdout:---- End output of apt-get -q -y install libapache2-mod-fastcgi=2.4.7~0910052141-2~bpo70+1.ceph ---- 2015-01-10T22:25:03.066 INFO:teuthology.orchestra.run.vpm142.stdout:Ran apt-get -q -y install libapache2-mod-fastcgi=2.4.7~0910052141-2~bpo70+1.ceph returned 100[0m 2015-01-10T22:25:03.066 INFO:teuthology.orchestra.run.vpm142.stdout: 2015-01-10T22:25:03.067 INFO:teuthology.orchestra.run.vpm142.stdout:[0m 2015-01-10T22:25:03.067 INFO:teuthology.orchestra.run.vpm142.stdout:Resource Declaration:[0m 2015-01-10T22:25:03.067 INFO:teuthology.orchestra.run.vpm142.stdout:---------------------[0m 2015-01-10T22:25:03.067 INFO:teuthology.orchestra.run.vpm142.stdout:# In /tmp/solo-from-scratch.ncEKxkB0CC91/cookbooks/ceph-qa/recipes/radosgw.rb 2015-01-10T22:25:03.067 INFO:teuthology.orchestra.run.vpm142.stdout: 2015-01-10T22:25:03.067 INFO:teuthology.orchestra.run.vpm142.stdout: 61: package 'libapache2-mod-fastcgi' do 2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout: 62: action :upgrade 2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout: 63: end 2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout: 64: package 'libfcgi0ldbl' 2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout:[0m 2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout: 2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout:[0m 2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout:Compiled Resource:[0m 2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout:------------------[0m 2015-01-10T22:25:03.069 INFO:teuthology.orchestra.run.vpm142.stdout:# Declared in /tmp/solo-from-scratch.ncEKxkB0CC91/cookbooks/ceph-qa/recipes/radosgw.rb:61:in `from_file' 2015-01-10T22:25:03.069 INFO:teuthology.orchestra.run.vpm142.stdout: 2015-01-10T22:25:03.069 INFO:teuthology.orchestra.run.vpm142.stdout:package("libapache2-mod-fastcgi") do 2015-01-10T22:25:03.069 INFO:teuthology.orchestra.run.vpm142.stdout: action [:upgrade] 2015-01-10T22:25:03.069 INFO:teuthology.orchestra.run.vpm142.stdout: retries 0 2015-01-10T22:25:03.069 INFO:teuthology.orchestra.run.vpm142.stdout: retry_delay 2 2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout: package_name "libapache2-mod-fastcgi" 2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout: version "2.4.7~0910052141-2~bpo70+1.ceph" 2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout: cookbook_name :"ceph-qa" 2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout: recipe_name "radosgw" 2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout:end 2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout:[0m 2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout: 2015-01-10T22:25:03.071 INFO:teuthology.orchestra.run.vpm142.stdout:[0m 2015-01-10T22:25:03.071 INFO:teuthology.orchestra.run.vpm142.stdout:[2015-01-11T06:25:03+00:00] INFO: Running queued delayed notifications before re-raising exception 2015-01-10T22:25:03.071 INFO:teuthology.orchestra.run.vpm142.stdout:[2015-01-11T06:25:03+00:00] ERROR: Running exception handlers 2015-01-10T22:25:03.071 INFO:teuthology.orchestra.run.vpm142.stdout:[2015-01-11T06:25:03+00:00] ERROR: Exception handlers complete 2015-01-10T22:25:03.071 INFO:teuthology.orchestra.run.vpm142.stdout:[2015-01-11T06:25:03+00:00] FATAL: Stacktrace dumped to /tmp/solo-from-scratch.ncEKxkB0CC91/solo/chef-stacktrace.out 2015-01-10T22:25:03.094 INFO:teuthology.orchestra.run.vpm142.stdout:[2015-01-11T06:25:03+00:00] ERROR: package[libapache2-mod-fastcgi] (ceph-qa::radosgw line 61) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '100' 2015-01-10T22:25:03.094 INFO:teuthology.orchestra.run.vpm142.stdout:---- Begin output of apt-get -q -y install libapache2-mod-fastcgi=2.4.7~0910052141-2~bpo70+1.ceph ---- 2015-01-10T22:25:03.094 INFO:teuthology.orchestra.run.vpm142.stdout:STDOUT: 2015-01-10T22:25:03.094 INFO:teuthology.orchestra.run.vpm142.stdout:STDERR: E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable) 2015-01-10T22:25:03.095 INFO:teuthology.orchestra.run.vpm142.stdout:E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it? 2015-01-10T22:25:03.095 INFO:teuthology.orchestra.run.vpm142.stdout:---- End output of apt-get -q -y install libapache2-mod-fastcgi=2.4.7~0910052141-2~bpo70+1.ceph ---- 2015-01-10T22:25:03.095 INFO:teuthology.orchestra.run.vpm142.stdout:Ran apt-get -q -y install libapache2-mod-fastcgi=2.4.7~0910052141-2~bpo70+1.ceph returned 100 2015-01-10T22:25:03.183 INFO:teuthology.orchestra.run.vpm142.stdout:[2015-01-11T06:25:03+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1) 2015-01-10T22:25:03.259 ERROR:teuthology.parallel:Exception in parallel execution
Updated by Loïc Dachary over 9 years ago
- Project changed from devops to teuthology
- Subject changed from chef fails on Could not get lock /var/lib/dpkg/lock for libapache2-mod-fastcgi to chef / apt-get compete during parallel runs
- Category deleted (
chef)
http://pulpito.ceph.com/loic-2015-01-10_11:54:25-rados-giant-backports-testing-basic-vps/694240/
apt-get fails (meaning chef wins ?)
2015-01-10T22:24:53.020 INFO:teuthology.orchestra.run.vpm112.stdout:Get:28 http://apt-mirror.front.sepia.ceph.com precise-security/universe i386 Packages [111 kB] 2015-01-10T22:24:53.030 INFO:teuthology.orchestra.run.vpm112.stdout:Get:29 http://apt-mirror.front.sepia.ceph.com precise-security/main TranslationIndex [208 B] 2015-01-10T22:24:53.036 INFO:teuthology.orchestra.run.vpm112.stdout:Get:30 http://apt-mirror.front.sepia.ceph.com precise-security/universe TranslationIndex [205 B] 2015-01-10T22:24:53.039 INFO:teuthology.orchestra.run.vpm112.stdout:Get:31 http://apt-mirror.front.sepia.ceph.com precise/main Translation-en [726 kB] 2015-01-10T22:24:53.184 INFO:teuthology.orchestra.run.vpm112.stdout:Get:32 http://apt-mirror.front.sepia.ceph.com precise/universe Translation-en [3,341 kB] 2015-01-10T22:24:53.631 INFO:teuthology.orchestra.run.vpm112.stdout:Get:33 http://apt-mirror.front.sepia.ceph.com precise-updates/main Translation-en [376 kB] 2015-01-10T22:24:53.659 INFO:teuthology.orchestra.run.vpm112.stdout:Get:34 http://apt-mirror.front.sepia.ceph.com precise-updates/universe Translation-en [146 kB] 2015-01-10T22:24:53.712 INFO:teuthology.orchestra.run.vpm112.stdout:Get:35 http://apt-mirror.front.sepia.ceph.com precise-security/main Translation-en [208 kB] 2015-01-10T22:24:53.720 INFO:teuthology.orchestra.run.vpm112.stdout:Get:36 http://apt-mirror.front.sepia.ceph.com precise-security/universe Translation-en [63.2 kB] 2015-01-10T22:25:04.284 INFO:teuthology.orchestra.run.vpm093.stdout:Fetched 27.5 MB in 15s (1,818 kB/s) 2015-01-10T22:25:04.291 INFO:teuthology.orchestra.run.vpm093.stderr:E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable) 2015-01-10T22:25:04.292 INFO:teuthology.orchestra.run.vpm093.stderr:E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it? 2015-01-10T22:25:04.292 ERROR:teuthology.parallel:Exception in parallel execution Traceback (most recent call last): File "/home/teuthworker/src/teuthology_giant/teuthology/parallel.py", line 82, in __exit__ for result in self: File "/home/teuthworker/src/teuthology_giant/teuthology/parallel.py", line 101, in next resurrect_traceback(result) File "/home/teuthworker/src/teuthology_giant/teuthology/parallel.py", line 19, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/teuthology_giant/teuthology/task/internal.py", line 613, in _download_and_run_chef 'sh', File "/home/teuthworker/src/teuthology_giant/teuthology/orchestra/remote.py", line 128, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/teuthology_giant/teuthology/orchestra/run.py", line 368, in run r.wait() File "/home/teuthworker/src/teuthology_giant/teuthology/orchestra/run.py", line 106, in wait exitstatus=status, node=self.hostname) CommandFailedError: Command failed on vpm093 with status 100: "wget -q -O- 'http://ceph.com/git/?p=ceph-qa-chef.git;a=blob_plain;f=solo/solo-from-scratch;hb=HEAD' | sh" 2015-01-10T22:25:04.293 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/teuthology_giant/teuthology/run_tasks.py", line 53, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/teuthology_giant/teuthology/run_tasks.py", line 41, in run_one_task return fn(**kwargs) File "/home/teuthworker/src/teuthology_giant/teuthology/task/internal.py", line 603, in vm_setup p.spawn(_download_and_run_chef, rem) File "/home/teuthworker/src/teuthology_giant/teuthology/parallel.py", line 89, in __exit__ raise CommandFailedError: Command failed on vpm093 with status 100: "wget -q -O- 'http://ceph.com/git/?p=ceph-qa-chef.git;a=blob_plain;f=solo/solo-from-scratch;hb=HEAD' | sh" 2015-01-10T22:25:04.293 INFO:teuthology.nuke:Checking targets against current locks 2015-01-10T22:25:04.436 INFO:teuthology.lock:unlocked vpm093.front.sepia.ceph.com 2015-01-10T22:25:04.467 INFO:teuthology.lock:unlocked vpm112.front.sepia.ceph.com
Updated by Zack Cerza over 9 years ago
- Project changed from teuthology to devops
- Status changed from New to Need More Info
- Assignee set to Sandon Van Ness
In both cases, the error occurs when apt is being called by chef. The parallel processes are on different hosts. Sandon, any clue as to why we'd be hitting the dpkg lock considering these are VMs?
Updated by Sage Weil about 6 years ago
- Status changed from Need More Info to Won't Fix
Actions