Bug #13885
closedrgw: DNSError on OpenStack
0%
Description
description: rados/thrash/{hobj-sort.yaml 0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr/simple.yaml msgr-failures/osd-delay.yaml thrashers/pggrow.yaml workloads/rgw_snaps.yaml} ... 2015-11-26T12:39:37.758 INFO:teuthology.orchestra.run.target172075.stderr:created pool .users snap 2 2015-11-26T12:39:40.289 INFO:teuthology.orchestra.run.target172075.stderr:Traceback (most recent call last): 2015-11-26T12:39:40.289 INFO:teuthology.orchestra.run.target172075.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run 2015-11-26T12:39:40.290 INFO:teuthology.orchestra.run.target172075.stderr: result = self._run(*self.args, **self.kwargs) 2015-11-26T12:39:40.290 INFO:teuthology.orchestra.run.target172075.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests/readwrite.py", line 126, in write_file 2015-11-26T12:39:40.290 INFO:teuthology.orchestra.run.target172075.stderr: key.set_contents_from_file(fp) 2015-11-26T12:39:40.290 INFO:teuthology.orchestra.run.target172075.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2015-11-26T12:39:40.290 INFO:teuthology.orchestra.run.target172075.stderr: chunked_transfer=chunked_transfer, size=size) 2015-11-26T12:39:40.290 INFO:teuthology.orchestra.run.target172075.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file 2015-11-26T12:39:40.290 INFO:teuthology.orchestra.run.target172075.stderr: chunked_transfer=chunked_transfer, size=size) 2015-11-26T12:39:40.290 INFO:teuthology.orchestra.run.target172075.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2015-11-26T12:39:40.290 INFO:teuthology.orchestra.run.target172075.stderr: query_args=query_args 2015-11-26T12:39:40.291 INFO:teuthology.orchestra.run.target172075.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 664, in make_request 2015-11-26T12:39:40.291 INFO:teuthology.orchestra.run.target172075.stderr: retry_handler=retry_handler 2015-11-26T12:39:40.291 INFO:teuthology.orchestra.run.target172075.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request 2015-11-26T12:39:40.291 INFO:teuthology.orchestra.run.target172075.stderr: retry_handler=retry_handler) 2015-11-26T12:39:40.291 INFO:teuthology.orchestra.run.target172075.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe 2015-11-26T12:39:40.292 INFO:teuthology.orchestra.run.target172075.stderr: raise ex 2015-11-26T12:39:40.292 INFO:teuthology.orchestra.run.target172075.stderr:DNSError: [Errno 3] name does not exist
Updated by Loïc Dachary over 8 years ago
most probably a DNS configuration error in the openstack cluster but quite difficult to figure out whithout an indication of what fqdn failed to resolve. It also happens when running the rgw suite (see http://www.spinics.net/lists/ceph-devel/msg25217.html for more information).
It fails all rgw_snaps.yaml workloads on infernalis in rados/thrash
Updated by Loïc Dachary about 8 years ago
- Project changed from ovh to rgw
Moving it back to rgw because, IMHO, it takes a rgw developer to figure this one out :-)
Updated by Loïc Dachary about 8 years ago
ceph-workbench --verbose ceph-qa-suite --suite rados --ceph-qa-suite-git-url http://github.com/dachary/ceph-qa-suite --suite-branch wip-rados-openstack --ceph-git-url http://github.com/ceph/ceph --ceph jewel --filter='rados/thrash/{hobj-sort.yaml rados.yaml 0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr/simple.yaml msgr-failures/osd-delay.yaml thrashers/pggrow.yaml workloads/rgw_snaps.yaml}' --upload
http://167.114.252.97:8081/ubuntu-2016-03-29_23:49:52-rados-jewel---basic-openstack/89/
2016-03-30T02:45:55.918 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [2] live_osds: [1, 0, 3, 5, 4] 2016-03-30T02:45:55.918 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2016-03-30T02:45:55.918 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2016-03-30T02:45:55.919 INFO:teuthology.orchestra.run.target167114230177:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 4 1' 2016-03-30T02:45:56.594 INFO:teuthology.orchestra.run.target167114230177.stderr:set osd.4 primary-affinity to 1 (8655362) 2016-03-30T02:46:01.281 INFO:teuthology.orchestra.run.target167114230177.stderr:Traceback (most recent call last): 2016-03-30T02:46:01.282 INFO:teuthology.orchestra.run.target167114230177.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run 2016-03-30T02:46:01.282 INFO:teuthology.orchestra.run.target167114230177.stderr: result = self._run(*self.args, **self.kwargs) 2016-03-30T02:46:01.283 INFO:teuthology.orchestra.run.target167114230177.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests/readwrite.py", line 126, in write_file 2016-03-30T02:46:01.283 INFO:teuthology.orchestra.run.target167114230177.stderr: key.set_contents_from_file(fp) 2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr: chunked_transfer=chunked_transfer, size=size) 2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file 2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr: chunked_transfer=chunked_transfer, size=size) 2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr: query_args=query_args 2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 665, in make_request 2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr: retry_handler=retry_handler 2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request 2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr: retry_handler=retry_handler) 2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe 2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr: raise ex 2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:DNSError: [Errno 3] name does not exist 2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:<Greenlet at 0x7f82142c4a50: write_file(fp=<tempfile.SpooledTemporaryFile instance at 0x7f821, file_name='jguxmwrrjyebsir', bucket=<Bucket: rwtest>)> failed with DNSError 2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:
Updated by Loïc Dachary about 8 years ago
teuthology-suite --ceph jewel --filter='rados/thrash/{hobj-sort.yaml rados.yaml 0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr/simple.yaml msgr-failures/osd-delay.yaml thrashers/pggrow.yaml workloads/rgw_snaps.yaml}' --suite rados
2016-03-30T07:06:16.721 INFO:teuthology.orchestra.run.target092153.stderr:warning: line 36: 'osd_pool_default_size' in section 'global' redefined 2016-03-30T07:06:16.820 INFO:teuthology.orchestra.run.target092153.stdout:Importing pgid 1.5 2016-03-30T07:06:16.829 INFO:teuthology.orchestra.run.target092153.stdout:Import successful 2016-03-30T07:06:16.879 INFO:teuthology.orchestra.run.target092153:Running: 'rm -f /home/ubuntu/cephtest/data/exp.1.5.1' 2016-03-30T07:06:17.153 INFO:teuthology.orchestra.run.target092153.stderr:Traceback (most recent call last): 2016-03-30T07:06:17.154 INFO:teuthology.orchestra.run.target092153.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run 2016-03-30T07:06:17.154 INFO:teuthology.orchestra.run.target092153.stderr: result = self._run(*self.args, **self.kwargs) 2016-03-30T07:06:17.154 INFO:teuthology.orchestra.run.target092153.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests/readwrite.py", line 126, in write_file 2016-03-30T07:06:17.155 INFO:teuthology.orchestra.run.target092153.stderr: key.set_contents_from_file(fp) 2016-03-30T07:06:17.155 INFO:teuthology.orchestra.run.target092153.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2016-03-30T07:06:17.155 INFO:teuthology.orchestra.run.target092153.stderr: chunked_transfer=chunked_transfer, size=size) 2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file 2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr: chunked_transfer=chunked_transfer, size=size) 2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr: query_args=query_args 2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 665, in make_request 2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr: retry_handler=retry_handler 2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request 2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr: retry_handler=retry_handler) 2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe 2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr: raise ex 2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr:DNSError: [Errno 3] name does not exist 2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr:<Greenlet at 0x7ff6a9f0c910: write_file(fp=<tempfile.SpooledTemporaryFile instance at 0x7ff6a, file_name='xnkkblacjgzuanm', bucket=<Bucket: rwtest>)> failed with DNSError 2016-03-30T07:06:17.158 INFO:teuthology.orchestra.run.target092153.stderr:
Updated by Loïc Dachary about 8 years ago
See also http://tracker.ceph.com/issues/12471#note-4 for identical errors back when testing infernalis.
Updated by Loïc Dachary about 8 years ago
See for debug purposes https://github.com/ceph/ceph-qa-suite/pull/925
Updated by Loïc Dachary about 8 years ago
Running with https://github.com/ceph/s3-tests/pull/104 applied seems to fix the problem. Upgrading gevent from 0.13 to > 1.0 + changing the API calls that are not backward compatible.
Updated by Abhishek Lekshmanan about 8 years ago
- Status changed from 12 to In Progress
- Assignee set to Abhishek Lekshmanan
Updated by Abhishek Lekshmanan about 8 years ago
- Assignee deleted (
Abhishek Lekshmanan)
Updated by Loïc Dachary about 8 years ago
<loicd> owasserm: how does master become ceph-master ? <loicd> owasserm: the idea is to fix the tests that people use, not the tests that noone uses ;-) <owasserm> loicd, ceph-master is the stable branch were the s3tests pass without failure. We will also have ceph-jewel for jewel <owasserm> s3tests are used with other s3 providers <loicd> owasserm: what is the difference between ceph-master and master ? <owasserm> loicd, at the moment one commit <owasserm> sage rebased ceph-master last week <loicd> https://github.com/ceph/s3-tests/commit/41c8be0584530d189c1c245a88073e771ac05e53 <owasserm> loicd, it is like the difference between master and jewel <loicd> ok <loicd> owasserm: so, since you don't see an issue with 104 it could be merged in master. But it won't be used for testing until it is merged in ceph-master. Is that right ? <owasserm> loicd, yes <loicd> owasserm: and http://pulpito.ceph.com/?suite=rgw&branch=master jobs involving s3-test use the ceph-master branch ? <loicd> hum, or not <owasserm> loicd, yes it does <loicd> owasserm: reading https://github.com/ceph/ceph-qa-suite/blob/master/tasks/s3tests.py#L127 suggest that if it does it does not come from this function. Is it explicitly set somewhere ? <owasserm> loicd, probably see in logs 2016-04-10T13:37:50.513 INFO:tasks.s3tests:Using branch 'ceph-master' for s3tests <owasserm> loicd, here branch = cconf.get('branch', 'ceph-' + suite_branch) <owasserm> line 136 <loicd> ok <loicd> good :-) <loicd> owasserm: I get how commits should travel and when they are supposed to end, thanks :-)
Updated by Loïc Dachary about 8 years ago
- Status changed from In Progress to 17
now that the PR is merged, there is a need to schedule a rgw on the sepia lab with the master branch of s3test to assert that https://github.com/ceph/s3-tests/pull/104 does not disrupt anything