Project

General

Profile

Actions
Copy link

Bug #13881

closed

Calamari-setup fails. I suspect something conflicts with one of the ansible installs.

Added by Anonymous over 8 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
ansible
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

Run teuthology using the following yaml file:

roles:
- [mon.a, mon.c, osd.0, osd.1, osd.2, client.0]
- [mon.b, osd.3, osd.4, osd.5, client.1]
machine_type: vps
tasks:
- ssh_keys:
- calamari_setup:
    test_image: /home/wusui/latestyamls/rhceph-1.2.3-ubuntu-14.04-x86_64.iso
    email: wusui@redhat.com
- interactive:
targets:
  vpm125.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCk//jo59RTTF4o1ramN7C7ZerQLVKRQ+gCJNyshMooICVVHU4k6UVzXhU3KcGHOIOD8dDCF5eTIUXsdVmnP9u78vEvF9Uf/zcIWhSmDQ+xaHn+dLKyjkV5mIbLyhdGfYPISnYpBfs0gG2MM1IBpVsCyy3UV+ZvG4doZEFcF1OlGqoPDVd5TKEUACuCA2wacTGJkMJE2qGS8NKitjlLbH7UNXpx+oDIcUWof7YAGBWXbulDqikiVnRxn1iMSv5HucDhDa5/TsTQI4A/n4ZgYsUu90OzYpwEKW2UH7Iw3z0SlrADATw3Gu2pX1VC138nC93Cp4GH958N6sT0ft9tnTB
  vpm156.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDf6ujRu/CitXihoZyyR/C12e/OOTD5dhKI4kE8mNJvleFuwzShK9rm09JCd9bxq1mEcH6gnwFZidrrw+iYUkohZh9FimpF5fsCfQP5hTy8pggOh7V+5oe4QhTch9oMc/3gtR3WvWq6uV49NmQtq87zfEntrZ4Si5djZ0sIKQKeoVDu/lUZdOi/qrZZ8Wol1qVSNqbEvwaOk8f8yQaqKgukWwewX2PwZJVL3Hfzpma1B1fo8P/xFcXjS9x7yS5IkXIP86NZHjRKtNQG62dEIFseJ0AkzWrwUkrUhBpTBhc3zqTxF9O7MJz5KCX+SamhRd/X4ybJqXA6EwOrL7U47A6P

This should normally install a cluster using the test_image iso. The run does not complete. Teuthology.log after this run shows:

2015-11-25T14:14:04.042 INFO:teuthology.orchestra.run.vpm156.stderr:[vpm125][DEBUG ] Reading state information...
2015-11-25T14:14:04.157 INFO:teuthology.orchestra.run.vpm156.stderr:[vpm125][DEBUG ] Some packages could not be installed. This may mean that you have
2015-11-25T14:14:04.159 INFO:teuthology.orchestra.run.vpm156.stderr:[vpm125][DEBUG ] requested an impossible situation or if you are using the unstable
2015-11-25T14:14:04.161 INFO:teuthology.orchestra.run.vpm156.stderr:[vpm125][DEBUG ] distribution that some required packages have not yet been created
2015-11-25T14:14:04.162 INFO:teuthology.orchestra.run.vpm156.stderr:[vpm125][DEBUG ] or been moved out of Incoming.
2015-11-25T14:14:04.162 INFO:teuthology.orchestra.run.vpm156.stderr:[vpm125][DEBUG ] The following information may help to resolve the situation:
2015-11-25T14:14:04.162 INFO:teuthology.orchestra.run.vpm156.stderr:[vpm125][DEBUG ]
2015-11-25T14:14:04.163 INFO:teuthology.orchestra.run.vpm156.stderr:[vpm125][DEBUG ] The following packages have unmet dependencies:
2015-11-25T14:14:04.163 INFO:teuthology.orchestra.run.vpm156.stderr:[vpm125][DEBUG ]  ceph-common : Depends: librbd1 (= 0.80.8.1-1trusty) but 0.80.10-0ubuntu1.14.04.3 is to be installed
2015-11-25T14:14:04.163 INFO:teuthology.orchestra.run.vpm156.stderr:[vpm125][WARNING] E: Unable to correct problems, you have held broken packages.
2015-11-25T14:14:04.164 INFO:teuthology.orchestra.run.vpm156.stderr:[vpm125][ERROR ] RuntimeError: command returned non-zero exit status: 100
2015-11-25T14:14:04.182 INFO:teuthology.orchestra.run.vpm156.stderr:[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env DEBIAN_FRONTEND=noninteractive apt-get install --assume-yes ceph ceph-mds ceph-common ceph-fs-common gdisk
2015-11-25T14:14:04.183 INFO:teuthology.orchestra.run.vpm156.stderr:
2015-11-25T14:14:04.193 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/wusui/src/teuthology/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/wusui/src/ceph-qa-suite/tasks/calamari_setup.py", line 311, in ceph_install
    ret = deploy_ceph(ctx, cal_svr)
  File "/home/wusui/src/ceph-qa-suite/tasks/calamari_setup.py", line 396, in deploy_ceph
    cal_svr.run(args=cmd).exitstatus
  File "/home/wusui/src/teuthology/teuthology/orchestra/remote.py", line 156, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/wusui/src/teuthology/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/wusui/src/teuthology/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on vpm156 with status 1: 'ceph-deploy install vpm125 vpm156'

This used to work on runs where ansible did not run. I suspect that one of the ansible tasks is installing something that conflicts with what the iso expects.

I don't know if there is a way to specify additional information in the yaml file to avoid this problem.

The complete log information is in ~/wusui/calamari-setup on teuthology.front.sepia.ceph.com

Actions
Copy link
 #1

Updated by Anonymous over 8 years ago

I got this result twice -- once on my branch (vpm125 and vpm156) and once on master (vpm048 and vpm127). I have locked those machines and left them up for the time being in case anyone wants to take a look.

Actions
Copy link
 #2

Updated by Anonymous over 8 years ago

Note: This problem does not occur when I manually install a cluster using ceph-deploy. So even though this looks like a packaging issue, something is exacerbated by whatever we are doing in calamari-setup.

Actions
Copy link
 #3

Updated by Anonymous over 8 years ago

Note: After manually running ice_setup, the following command worked without any errors:

sudo apt-get install --assume-yes ceph ceph-mds ceph-common ceph-fs-common gdisk

Actions
Copy link
 #4

Updated by Anonymous over 8 years ago

  • Assignee changed from Andrew Schoen to Anonymous
Actions
Copy link
 #5

Updated by Anonymous over 8 years ago 

interactive-on-error: true
roles:
- [mon.a, mon.c, osd.0, osd.1, osd.2, client.0]
- [mon.b, osd.3, osd.4, osd.5, client.1]
machine_type: vps
tasks:
- ssh_keys:
- ansible:
    playbook: /home/ubuntu/src/ceph-cm-ansible_master/cephlab.yml
    branch: firefly
- interactive:
- calamari_setup:
    test_image: /home/ubuntu/wusui/test_calamari_setup/rhceph-1.2.3-ubuntu-14.04-x86_64.iso
    email: wusui@redhat.com
- interactive:

Running teuthology using this yaml file stops at the first interactive prompt, and produces the ceph-common librbd1 conflict error before the second prompt.

At the first interactive prompt, if I manually run ice-setup and ceph-deploy install, I see the same problem.

The ceph-deploy install works if I manually run ice-setup and ceph deploy on a newly imaged vpm without running teuthology first.

I guessed that there was some sort of version incompatibility, so I tried the above with every permutation of branch: hammer, firefly, master and playbook: ceph-cm-ansible_master/cephlab.yml and ceph-cm-ansible_fail-pb/cephlab.yml (the only two cephlab.yml files that I found on ~ubuntu/src on teuthology.front.sepia.ceph.com). All behaved the same.

I am confused because after the ansible task runs, none of the ceph stuff is installed but whatever it does ends up finding a conflict when the the ceph stuff is installed later.

Actions
Copy link
 #6

Updated by Anonymous over 8 years ago

It appears that https://github.com/ceph/ceph-cm-ansible/pull/168 is probably what is needed to fix this problem.

Actions
Copy link
 #7

Updated by Anonymous over 8 years ago

  • Status changed from New to Resolved

This now appears to work fine on installs on both sepia and octo lab machines. I think that things.

Actions
Copy link

Also available in: Atom PDF