Ceph » devops

Run: http://pulpito.ceph.com/teuthology-2015-06-23_17:00:02-upgrade:firefly-firefly-distro-basic-vps/
Jobs: ~70

I have two questions after looking at this.

First question: v0.80.10 was pushed to the firefly branch and released on Jun 18th. Why would a job that's queued on Jun 21st be worried about an older version, v0.80.9?

Second question: The example run that Yuri linked above is http://qa-proxy.ceph.com/teuthology/teuthology-2015-06-21_17:18:01-upgrade:firefly-x-next-distro-basic-vps/943413/teuthology.log . This run's YAML has the following:

  os_type: centos
  os_version: '6.5'

... and then later in the log:

2015-06-21T19:03:08.737 DEBUG:teuthology.suite:Defaults for machine_type plana distro centos: arch=x86_64, release=centos7, pkg_type=rpm
2015-06-21T19:03:08.737 DEBUG:teuthology.suite:Looking for packages at http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/664123890c8679b456197f38040550f705423bb1/version

Why is it choosing CentOS 7 there, instead of CentOS 6.5?

Ken Dreyer wrote:

Why is it choosing CentOS 7 there, instead of CentOS 6.5?

It is actually provisioning the node correctly as CentOS 6.5. It's just that the check teuthology-suite is performing to check packages is not respecting ``os_version`` in the config. I suspect this is because teuthology.suite.get_distro_defaults does not seem to check for ctx.config.os_version at all when deciding what distro / version combo to check for. See this code: https://github.com/ceph/teuthology/blob/master/teuthology/suite.py#L374-L418

You can see the provisioning is correct here though:

2015-06-21T20:48:18.815 INFO:teuthology.provision:Provisioning a centos 6.5 vps
2015-06-21T20:48:23.163 INFO:teuthology.provision:Downburst created ubuntu@vpm043.front.sepia.ceph.com: Fetching default SSH key from

...

# chef prints this later
2015-06-21T20:50:55.498 INFO:teuthology.orchestra.run.vpm043.stdout:[2015-06-22T03:50:53-04:00] DEBUG: Platform is centos version 6.5

The curious thing that makes this log even stranger is this log output that say that vpm043 is ubuntu precise.

2015-06-21T20:50:22.316 DEBUG:teuthology.task.internal:machine status is {u'is_vm': True, u'locked': True, u'locked_since': u'2015-06-22 03:48:08.360914', u'locked_by': u'scheduled_teuthology@teuthology', u'up': True, u'mac_address': u'52:54:00:2a:bb:aa', 
u'name': u'vpm043.front.sepia.ceph.com', 
u'os_version': u'12.04', 
u'machine_type': u'vps', 
u'vm_host': {u'is_vm': False, u'locked': True, u'locked_since': u'2013-03-14 19:29:52', u'locked_by': u'VPSHOST@VPSHOST', u'up': True, u'mac_address': u'00:25:90:00:59:e6', u'name': u'mira008.front.sepia.ceph.com', u'os_version': None, u'machine_type': u'mira', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgHKHFNZkwD2QDF7JzCDxlgpDMLphzFALOCEXRENHTnxBWll8AK18ZAFJXMMfGcWpjbXEh8bc7ss2tB9Qzo+/8ePrgfWCl+7YgoXOrnSFPvn58IWpMrfkGGBH3eH2FnAwWNOr0oAHkEyA96iJ9lGpGAsl9t0F4OQglgSXiHzp4uIvOxGaRud13p2LRB789PglJ3lxcVxRS3RwndJ9tQNHFOF8msX9+aSu2+S3W5n4s4WNzrGyhhBYdaz/iQRuJdUNcCprfX8cdKZfTw0jXrvv4KNj2YZraKxwETHD7yDNA5IfjRmGdYJeXg8R+qDv1AeZwiU67ksyrrhL6eTay/LAl', u'description': u''}, 
u'os_type': u'ubuntu', 
u'arch': u'x86_64', 
u'ssh_pub_key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzCFAuSXBaTUm5TsAWSpZiu44e4l/jlXMt9qYi1XlVmthYRr/d86Q5BPSI77gCZqLJfi3s/U8RlKfEFyHzyOHI1fJcFK1aB0ZnjkYzUtHWNwZ8jHkX4kg81zU7Obe5Pkc1yxCLebcvbFAC6frSvjKG9jbbV43VLxkavydRKEhJMcbwN6mSiMoxxLruS91tYP403ffzscgql2WZ09/4S6fkGzMgz2a96qhi1fGpoo/8oiVVhIMShOqYgX562NDU+1EJ8b3Fq6xEfZGTwnnB8p4VwxRuQojGm8OLqFLEetrArLyOJNIN6zGZY1+POCOv9qGWns2qoOcEAAcPe66wzTPEQ==', u'description': u'/var/lib/teuthworker/archive/teuthology-2015-06-21_17:18:01-upgrade:firefly-x-next-distro-basic-vps/943413'}

I'm still not exactly sure what is causing this specific traceback yet though.

With with latest Andrew's additional logging - http://pulpito.ceph.com/teuthology-2015-06-29_16:59:48-upgrade:firefly-firefly---basic-vps/

Yuri Weinstein wrote:

With with latest Andrew's additional logging - http://pulpito.ceph.com/teuthology-2015-06-29_16:59:48-upgrade:firefly-firefly---basic-vps/

Yup, this is the issue we've seen before. This url returns the incorrect version, http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/v0.80.10/version

Same issue as http://tracker.ceph.com/issues/11222.

I've triggered a rebuild with this url: http://gitbuilder.sepia.ceph.com/ceph-rpm-centos6-x86_64-basic/rebuild.cgi?log=1623e6c6517ff6958b343493fce6a8e50a52a042

Andrew Schoen wrote:

I've triggered a rebuild with this url: http://gitbuilder.sepia.ceph.com/ceph-rpm-centos6-x86_64-basic/rebuild.cgi?log=1623e6c6517ff6958b343493fce6a8e50a52a042

That url was incorrect, I believe this one to be the correct one.

http://gitbuilder.sepia.ceph.com/gitbuilder-ceph-rpm-centos6-5-amd64-basic/rebuild.cgi?log=1623e6c6517ff6958b343493fce6a8e50a52a042

An explanation as to why this happens from #sepia:

[13:01:56] <dmick> branch is pushed, gitbuilder builds the sha1, links it to branch name
[13:02:06] <dmick> then, tag is added and pushed; gitbuilder looks, sha1 is already built, nothing to do

I kicked off a rebuild yesterday, but gitbuilder seems stuck here:

http://gitbuilder.sepia.ceph.com/gitbuilder-ceph-rpm-centos6-5-amd64-basic/log.cgi?log=ea6c958c38df1216bf95c927f143d8b13c4a9e70