Ceph » rgw

most probably a DNS configuration error in the openstack cluster but quite difficult to figure out whithout an indication of what fqdn failed to resolve. It also happens when running the rgw suite (see http://www.spinics.net/lists/ceph-devel/msg25217.html for more information).

It fails all rgw_snaps.yaml workloads on infernalis in rados/thrash

http://167.114.252.97:8081/ubuntu-2016-03-29_23:49:52-rados-jewel---basic-openstack/89/

2016-03-30T02:45:55.918 INFO:tasks.thrashosds.thrasher:in_osds:  [0, 2, 3, 4, 5] out_osds:  [1] dead_osds:  [2] live_osds:  [1, 0, 3, 5, 4]
2016-03-30T02:45:55.918 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0
2016-03-30T02:45:55.918 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000
2016-03-30T02:45:55.919 INFO:teuthology.orchestra.run.target167114230177:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 4 1'
2016-03-30T02:45:56.594 INFO:teuthology.orchestra.run.target167114230177.stderr:set osd.4 primary-affinity to 1 (8655362)
2016-03-30T02:46:01.281 INFO:teuthology.orchestra.run.target167114230177.stderr:Traceback (most recent call last):
2016-03-30T02:46:01.282 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
2016-03-30T02:46:01.282 INFO:teuthology.orchestra.run.target167114230177.stderr:    result = self._run(*self.args, **self.kwargs)
2016-03-30T02:46:01.283 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/readwrite.py", line 126, in write_file
2016-03-30T02:46:01.283 INFO:teuthology.orchestra.run.target167114230177.stderr:    key.set_contents_from_file(fp)
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:    chunked_transfer=chunked_transfer, size=size)
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:    chunked_transfer=chunked_transfer, size=size)
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:    query_args=query_args
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 665, in make_request
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:    retry_handler=retry_handler
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:    retry_handler=retry_handler)
2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:    raise ex
2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:DNSError: [Errno 3] name does not exist
2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:<Greenlet at 0x7f82142c4a50: write_file(fp=<tempfile.SpooledTemporaryFile instance at 0x7f821, file_name='jguxmwrrjyebsir', bucket=<Bucket: rwtest>)> failed with DNSError
2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:

• fail http://pulpito.ovh.sepia.ceph.com:8081/loic-2016-03-30_06:45:03-rados-jewel---basic-openstack/

2016-03-30T07:06:16.721 INFO:teuthology.orchestra.run.target092153.stderr:warning: line 36: 'osd_pool_default_size' in section 'global' redefined
2016-03-30T07:06:16.820 INFO:teuthology.orchestra.run.target092153.stdout:Importing pgid 1.5
2016-03-30T07:06:16.829 INFO:teuthology.orchestra.run.target092153.stdout:Import successful
2016-03-30T07:06:16.879 INFO:teuthology.orchestra.run.target092153:Running: 'rm -f /home/ubuntu/cephtest/data/exp.1.5.1'
2016-03-30T07:06:17.153 INFO:teuthology.orchestra.run.target092153.stderr:Traceback (most recent call last):
2016-03-30T07:06:17.154 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
2016-03-30T07:06:17.154 INFO:teuthology.orchestra.run.target092153.stderr:    result = self._run(*self.args, **self.kwargs)
2016-03-30T07:06:17.154 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/readwrite.py", line 126, in write_file
2016-03-30T07:06:17.155 INFO:teuthology.orchestra.run.target092153.stderr:    key.set_contents_from_file(fp)
2016-03-30T07:06:17.155 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-03-30T07:06:17.155 INFO:teuthology.orchestra.run.target092153.stderr:    chunked_transfer=chunked_transfer, size=size)
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:    chunked_transfer=chunked_transfer, size=size)
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:    query_args=query_args
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 665, in make_request
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:    retry_handler=retry_handler
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr:    retry_handler=retry_handler)
2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr:    raise ex
2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr:DNSError: [Errno 3] name does not exist
2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr:<Greenlet at 0x7ff6a9f0c910: write_file(fp=<tempfile.SpooledTemporaryFile instance at 0x7ff6a, file_name='xnkkblacjgzuanm', bucket=<Bucket: rwtest>)> failed with DNSError
2016-03-30T07:06:17.158 INFO:teuthology.orchestra.run.target092153.stderr:

See also http://tracker.ceph.com/issues/12471#note-4 for identical errors back when testing infernalis.

See for debug purposes https://github.com/ceph/ceph-qa-suite/pull/925

Running with https://github.com/ceph/s3-tests/pull/104 applied seems to fix the problem. Upgrading gevent from 0.13 to > 1.0 + changing the API calls that are not backward compatible.

<loicd> owasserm: how does master become ceph-master ? 
<loicd> owasserm: the idea is to fix the tests that people use, not the tests that noone uses ;-)
<owasserm> loicd,  ceph-master is the stable branch were the s3tests pass without failure. We will also have ceph-jewel for jewel
<owasserm> s3tests are used with other s3 providers
<loicd> owasserm: what is the difference between ceph-master and master ? 
<owasserm> loicd, at the moment one commit
<owasserm> sage rebased ceph-master last week
<loicd> https://github.com/ceph/s3-tests/commit/41c8be0584530d189c1c245a88073e771ac05e53
<owasserm> loicd, it is like the difference between master and jewel
<loicd> ok
<loicd> owasserm: so, since you don't see an issue with 104 it could be merged in master. But it won't be used for testing until it is merged in ceph-master. Is that right ? 
<owasserm> loicd, yes 
<loicd> owasserm: and http://pulpito.ceph.com/?suite=rgw&branch=master jobs involving s3-test use the ceph-master branch ? 
<loicd> hum, or not
<owasserm> loicd, yes it does
<loicd> owasserm: reading https://github.com/ceph/ceph-qa-suite/blob/master/tasks/s3tests.py#L127 suggest that if it does it does not come from this function. Is it explicitly set somewhere ? 
<owasserm> loicd, probably see in logs 2016-04-10T13:37:50.513 INFO:tasks.s3tests:Using branch 'ceph-master' for s3tests
<owasserm> loicd, here branch = cconf.get('branch', 'ceph-' + suite_branch)
<owasserm> line 136
<loicd> ok
<loicd> good :-)
<loicd> owasserm: I get how commits should travel and when they are supposed to end, thanks :-)