Project

General

Profile

Actions
Copy link

Bug #10515

closed

chef / apt-get compete during parallel runs

Added by Loïc Dachary over 9 years ago. Updated about 6 years ago.

Status:
Won't Fix
Priority:
Normal
Assignee:
Sandon Van Ness
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://pulpito.ceph.com/loic-2015-01-10_11:54:25-rados-giant-backports-testing-basic-vps/694228/
Maybe because parallel execution gets two competing installation jobs to run at the same time ?

2015-01-10T22:25:03.062 INFO:teuthology.orchestra.run.vpm142.stdout:[0m
2015-01-10T22:25:03.062 INFO:teuthology.orchestra.run.vpm142.stdout:================================================================================[0m
2015-01-10T22:25:03.062 INFO:teuthology.orchestra.run.vpm142.stdout:[31mError executing action `upgrade` on resource 'package[libapache2-mod-fastcgi]'[0m
2015-01-10T22:25:03.063 INFO:teuthology.orchestra.run.vpm142.stdout:================================================================================[0m
2015-01-10T22:25:03.063 INFO:teuthology.orchestra.run.vpm142.stdout:
2015-01-10T22:25:03.063 INFO:teuthology.orchestra.run.vpm142.stdout:[0m
2015-01-10T22:25:03.065 INFO:teuthology.orchestra.run.vpm142.stdout:Mixlib::ShellOut::ShellCommandFailed[0m
2015-01-10T22:25:03.065 INFO:teuthology.orchestra.run.vpm142.stdout:------------------------------------[0m
2015-01-10T22:25:03.065 INFO:teuthology.orchestra.run.vpm142.stdout:Expected process to exit with [0], but received '100'
2015-01-10T22:25:03.065 INFO:teuthology.orchestra.run.vpm142.stdout:---- Begin output of apt-get -q -y install libapache2-mod-fastcgi=2.4.7~0910052141-2~bpo70+1.ceph ----
2015-01-10T22:25:03.066 INFO:teuthology.orchestra.run.vpm142.stdout:STDOUT:
2015-01-10T22:25:03.066 INFO:teuthology.orchestra.run.vpm142.stdout:STDERR: E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
2015-01-10T22:25:03.066 INFO:teuthology.orchestra.run.vpm142.stdout:E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
2015-01-10T22:25:03.066 INFO:teuthology.orchestra.run.vpm142.stdout:---- End output of apt-get -q -y install libapache2-mod-fastcgi=2.4.7~0910052141-2~bpo70+1.ceph ----
2015-01-10T22:25:03.066 INFO:teuthology.orchestra.run.vpm142.stdout:Ran apt-get -q -y install libapache2-mod-fastcgi=2.4.7~0910052141-2~bpo70+1.ceph returned 100[0m
2015-01-10T22:25:03.066 INFO:teuthology.orchestra.run.vpm142.stdout:
2015-01-10T22:25:03.067 INFO:teuthology.orchestra.run.vpm142.stdout:[0m
2015-01-10T22:25:03.067 INFO:teuthology.orchestra.run.vpm142.stdout:Resource Declaration:[0m
2015-01-10T22:25:03.067 INFO:teuthology.orchestra.run.vpm142.stdout:---------------------[0m
2015-01-10T22:25:03.067 INFO:teuthology.orchestra.run.vpm142.stdout:# In /tmp/solo-from-scratch.ncEKxkB0CC91/cookbooks/ceph-qa/recipes/radosgw.rb
2015-01-10T22:25:03.067 INFO:teuthology.orchestra.run.vpm142.stdout:
2015-01-10T22:25:03.067 INFO:teuthology.orchestra.run.vpm142.stdout: 61: package 'libapache2-mod-fastcgi' do
2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout: 62:   action :upgrade
2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout: 63: end
2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout: 64: package 'libfcgi0ldbl'
2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout:[0m
2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout:
2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout:[0m
2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout:Compiled Resource:[0m
2015-01-10T22:25:03.068 INFO:teuthology.orchestra.run.vpm142.stdout:------------------[0m
2015-01-10T22:25:03.069 INFO:teuthology.orchestra.run.vpm142.stdout:# Declared in /tmp/solo-from-scratch.ncEKxkB0CC91/cookbooks/ceph-qa/recipes/radosgw.rb:61:in `from_file'
2015-01-10T22:25:03.069 INFO:teuthology.orchestra.run.vpm142.stdout:
2015-01-10T22:25:03.069 INFO:teuthology.orchestra.run.vpm142.stdout:package("libapache2-mod-fastcgi") do
2015-01-10T22:25:03.069 INFO:teuthology.orchestra.run.vpm142.stdout:  action [:upgrade]
2015-01-10T22:25:03.069 INFO:teuthology.orchestra.run.vpm142.stdout:  retries 0
2015-01-10T22:25:03.069 INFO:teuthology.orchestra.run.vpm142.stdout:  retry_delay 2
2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout:  package_name "libapache2-mod-fastcgi" 
2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout:  version "2.4.7~0910052141-2~bpo70+1.ceph" 
2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout:  cookbook_name :"ceph-qa" 
2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout:  recipe_name "radosgw" 
2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout:end
2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout:[0m
2015-01-10T22:25:03.070 INFO:teuthology.orchestra.run.vpm142.stdout:
2015-01-10T22:25:03.071 INFO:teuthology.orchestra.run.vpm142.stdout:[0m
2015-01-10T22:25:03.071 INFO:teuthology.orchestra.run.vpm142.stdout:[2015-01-11T06:25:03+00:00] INFO: Running queued delayed notifications before re-raising exception
2015-01-10T22:25:03.071 INFO:teuthology.orchestra.run.vpm142.stdout:[2015-01-11T06:25:03+00:00] ERROR: Running exception handlers
2015-01-10T22:25:03.071 INFO:teuthology.orchestra.run.vpm142.stdout:[2015-01-11T06:25:03+00:00] ERROR: Exception handlers complete
2015-01-10T22:25:03.071 INFO:teuthology.orchestra.run.vpm142.stdout:[2015-01-11T06:25:03+00:00] FATAL: Stacktrace dumped to /tmp/solo-from-scratch.ncEKxkB0CC91/solo/chef-stacktrace.out
2015-01-10T22:25:03.094 INFO:teuthology.orchestra.run.vpm142.stdout:[2015-01-11T06:25:03+00:00] ERROR: package[libapache2-mod-fastcgi] (ceph-qa::radosgw line 61) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '100'
2015-01-10T22:25:03.094 INFO:teuthology.orchestra.run.vpm142.stdout:---- Begin output of apt-get -q -y install libapache2-mod-fastcgi=2.4.7~0910052141-2~bpo70+1.ceph ----
2015-01-10T22:25:03.094 INFO:teuthology.orchestra.run.vpm142.stdout:STDOUT:
2015-01-10T22:25:03.094 INFO:teuthology.orchestra.run.vpm142.stdout:STDERR: E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
2015-01-10T22:25:03.095 INFO:teuthology.orchestra.run.vpm142.stdout:E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
2015-01-10T22:25:03.095 INFO:teuthology.orchestra.run.vpm142.stdout:---- End output of apt-get -q -y install libapache2-mod-fastcgi=2.4.7~0910052141-2~bpo70+1.ceph ----
2015-01-10T22:25:03.095 INFO:teuthology.orchestra.run.vpm142.stdout:Ran apt-get -q -y install libapache2-mod-fastcgi=2.4.7~0910052141-2~bpo70+1.ceph returned 100
2015-01-10T22:25:03.183 INFO:teuthology.orchestra.run.vpm142.stdout:[2015-01-11T06:25:03+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
2015-01-10T22:25:03.259 ERROR:teuthology.parallel:Exception in parallel execution

Actions
Copy link
 #1

Updated by Loïc Dachary over 9 years ago

  • Project changed from devops to teuthology
  • Subject changed from chef fails on Could not get lock /var/lib/dpkg/lock for libapache2-mod-fastcgi to chef / apt-get compete during parallel runs
  • Category deleted (chef)

http://pulpito.ceph.com/loic-2015-01-10_11:54:25-rados-giant-backports-testing-basic-vps/694240/
apt-get fails (meaning chef wins ?)

2015-01-10T22:24:53.020 INFO:teuthology.orchestra.run.vpm112.stdout:Get:28 http://apt-mirror.front.sepia.ceph.com precise-security/universe i386 Packages [111 kB]
2015-01-10T22:24:53.030 INFO:teuthology.orchestra.run.vpm112.stdout:Get:29 http://apt-mirror.front.sepia.ceph.com precise-security/main TranslationIndex [208 B]
2015-01-10T22:24:53.036 INFO:teuthology.orchestra.run.vpm112.stdout:Get:30 http://apt-mirror.front.sepia.ceph.com precise-security/universe TranslationIndex [205 B]
2015-01-10T22:24:53.039 INFO:teuthology.orchestra.run.vpm112.stdout:Get:31 http://apt-mirror.front.sepia.ceph.com precise/main Translation-en [726 kB]
2015-01-10T22:24:53.184 INFO:teuthology.orchestra.run.vpm112.stdout:Get:32 http://apt-mirror.front.sepia.ceph.com precise/universe Translation-en [3,341 kB]
2015-01-10T22:24:53.631 INFO:teuthology.orchestra.run.vpm112.stdout:Get:33 http://apt-mirror.front.sepia.ceph.com precise-updates/main Translation-en [376 kB]
2015-01-10T22:24:53.659 INFO:teuthology.orchestra.run.vpm112.stdout:Get:34 http://apt-mirror.front.sepia.ceph.com precise-updates/universe Translation-en [146 kB]
2015-01-10T22:24:53.712 INFO:teuthology.orchestra.run.vpm112.stdout:Get:35 http://apt-mirror.front.sepia.ceph.com precise-security/main Translation-en [208 kB]
2015-01-10T22:24:53.720 INFO:teuthology.orchestra.run.vpm112.stdout:Get:36 http://apt-mirror.front.sepia.ceph.com precise-security/universe Translation-en [63.2 kB]
2015-01-10T22:25:04.284 INFO:teuthology.orchestra.run.vpm093.stdout:Fetched 27.5 MB in 15s (1,818 kB/s)
2015-01-10T22:25:04.291 INFO:teuthology.orchestra.run.vpm093.stderr:E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
2015-01-10T22:25:04.292 INFO:teuthology.orchestra.run.vpm093.stderr:E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
2015-01-10T22:25:04.292 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_giant/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_giant/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_giant/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/teuthology_giant/teuthology/task/internal.py", line 613, in _download_and_run_chef
    'sh',
  File "/home/teuthworker/src/teuthology_giant/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_giant/teuthology/orchestra/run.py", line 368, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_giant/teuthology/orchestra/run.py", line 106, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on vpm093 with status 100: "wget -q -O- 'http://ceph.com/git/?p=ceph-qa-chef.git;a=blob_plain;f=solo/solo-from-scratch;hb=HEAD' | sh" 
2015-01-10T22:25:04.293 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_giant/teuthology/run_tasks.py", line 53, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/teuthology_giant/teuthology/run_tasks.py", line 41, in run_one_task
    return fn(**kwargs)
  File "/home/teuthworker/src/teuthology_giant/teuthology/task/internal.py", line 603, in vm_setup
    p.spawn(_download_and_run_chef, rem)
  File "/home/teuthworker/src/teuthology_giant/teuthology/parallel.py", line 89, in __exit__
    raise
CommandFailedError: Command failed on vpm093 with status 100: "wget -q -O- 'http://ceph.com/git/?p=ceph-qa-chef.git;a=blob_plain;f=solo/solo-from-scratch;hb=HEAD' | sh" 
2015-01-10T22:25:04.293 INFO:teuthology.nuke:Checking targets against current locks
2015-01-10T22:25:04.436 INFO:teuthology.lock:unlocked vpm093.front.sepia.ceph.com
2015-01-10T22:25:04.467 INFO:teuthology.lock:unlocked vpm112.front.sepia.ceph.com

Actions
Copy link
 #2

Updated by Zack Cerza over 9 years ago

  • Project changed from teuthology to devops
  • Status changed from New to Need More Info
  • Assignee set to Sandon Van Ness

In both cases, the error occurs when apt is being called by chef. The parallel processes are on different hosts. Sandon, any clue as to why we'd be hitting the dpkg lock considering these are VMs?

Actions
Copy link
 #3

Updated by Sage Weil about 6 years ago

  • Status changed from Need More Info to Won't Fix
Actions
Copy link

Also available in: Atom PDF