Bug #13885: rgw: DNSError on OpenStack - rgw - Ceph

Actions

#1

Updated by Loïc Dachary over 8 years ago

most probably a DNS configuration error in the openstack cluster but quite difficult to figure out whithout an indication of what fqdn failed to resolve. It also happens when running the rgw suite (see http://www.spinics.net/lists/ceph-devel/msg25217.html for more information).

It fails all rgw_snaps.yaml workloads on infernalis in rados/thrash

Actions

Copy link

#2

Updated by Yuri Weinstein about 8 years ago

Project changed from 23 to ovh

Actions

Copy link

#3

Updated by Loïc Dachary about 8 years ago

Project changed from ovh to rgw

Moving it back to rgw because, IMHO, it takes a rgw developer to figure this one out :-)

Actions

Copy link

#4

Updated by Loïc Dachary about 8 years ago

ceph-workbench --verbose ceph-qa-suite --suite rados --ceph-qa-suite-git-url http://github.com/dachary/ceph-qa-suite --suite-branch wip-rados-openstack --ceph-git-url http://github.com/ceph/ceph --ceph jewel --filter='rados/thrash/{hobj-sort.yaml rados.yaml 0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr/simple.yaml msgr-failures/osd-delay.yaml thrashers/pggrow.yaml workloads/rgw_snaps.yaml}' --upload

http://167.114.252.97:8081/ubuntu-2016-03-29_23:49:52-rados-jewel---basic-openstack/89/

2016-03-30T02:45:55.918 INFO:tasks.thrashosds.thrasher:in_osds:  [0, 2, 3, 4, 5] out_osds:  [1] dead_osds:  [2] live_osds:  [1, 0, 3, 5, 4]
2016-03-30T02:45:55.918 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0
2016-03-30T02:45:55.918 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000
2016-03-30T02:45:55.919 INFO:teuthology.orchestra.run.target167114230177:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 4 1'
2016-03-30T02:45:56.594 INFO:teuthology.orchestra.run.target167114230177.stderr:set osd.4 primary-affinity to 1 (8655362)
2016-03-30T02:46:01.281 INFO:teuthology.orchestra.run.target167114230177.stderr:Traceback (most recent call last):
2016-03-30T02:46:01.282 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
2016-03-30T02:46:01.282 INFO:teuthology.orchestra.run.target167114230177.stderr:    result = self._run(*self.args, **self.kwargs)
2016-03-30T02:46:01.283 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/readwrite.py", line 126, in write_file
2016-03-30T02:46:01.283 INFO:teuthology.orchestra.run.target167114230177.stderr:    key.set_contents_from_file(fp)
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:    chunked_transfer=chunked_transfer, size=size)
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:    chunked_transfer=chunked_transfer, size=size)
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:    query_args=query_args
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 665, in make_request
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:    retry_handler=retry_handler
2016-03-30T02:46:01.284 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:    retry_handler=retry_handler)
2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:    raise ex
2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:DNSError: [Errno 3] name does not exist
2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:<Greenlet at 0x7f82142c4a50: write_file(fp=<tempfile.SpooledTemporaryFile instance at 0x7f821, file_name='jguxmwrrjyebsir', bucket=<Bucket: rwtest>)> failed with DNSError
2016-03-30T02:46:01.285 INFO:teuthology.orchestra.run.target167114230177.stderr:

Actions

Copy link

#5

Updated by Loïc Dachary about 8 years ago

teuthology-suite --ceph jewel --filter='rados/thrash/{hobj-sort.yaml rados.yaml 0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr/simple.yaml msgr-failures/osd-delay.yaml thrashers/pggrow.yaml workloads/rgw_snaps.yaml}'  --suite rados

fail http://pulpito.ovh.sepia.ceph.com:8081/loic-2016-03-30_06:45:03-rados-jewel---basic-openstack/

2016-03-30T07:06:16.721 INFO:teuthology.orchestra.run.target092153.stderr:warning: line 36: 'osd_pool_default_size' in section 'global' redefined
2016-03-30T07:06:16.820 INFO:teuthology.orchestra.run.target092153.stdout:Importing pgid 1.5
2016-03-30T07:06:16.829 INFO:teuthology.orchestra.run.target092153.stdout:Import successful
2016-03-30T07:06:16.879 INFO:teuthology.orchestra.run.target092153:Running: 'rm -f /home/ubuntu/cephtest/data/exp.1.5.1'
2016-03-30T07:06:17.153 INFO:teuthology.orchestra.run.target092153.stderr:Traceback (most recent call last):
2016-03-30T07:06:17.154 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
2016-03-30T07:06:17.154 INFO:teuthology.orchestra.run.target092153.stderr:    result = self._run(*self.args, **self.kwargs)
2016-03-30T07:06:17.154 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/readwrite.py", line 126, in write_file
2016-03-30T07:06:17.155 INFO:teuthology.orchestra.run.target092153.stderr:    key.set_contents_from_file(fp)
2016-03-30T07:06:17.155 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2016-03-30T07:06:17.155 INFO:teuthology.orchestra.run.target092153.stderr:    chunked_transfer=chunked_transfer, size=size)
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:    chunked_transfer=chunked_transfer, size=size)
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:    query_args=query_args
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 665, in make_request
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:    retry_handler=retry_handler
2016-03-30T07:06:17.156 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr:    retry_handler=retry_handler)
2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr:    raise ex
2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr:DNSError: [Errno 3] name does not exist
2016-03-30T07:06:17.157 INFO:teuthology.orchestra.run.target092153.stderr:<Greenlet at 0x7ff6a9f0c910: write_file(fp=<tempfile.SpooledTemporaryFile instance at 0x7ff6a, file_name='xnkkblacjgzuanm', bucket=<Bucket: rwtest>)> failed with DNSError
2016-03-30T07:06:17.158 INFO:teuthology.orchestra.run.target092153.stderr:

Actions

Copy link

#6

Updated by Loïc Dachary about 8 years ago

See also http://tracker.ceph.com/issues/12471#note-4 for identical errors back when testing infernalis.

Actions

Copy link

#7

Updated by Loïc Dachary about 8 years ago

See for debug purposes https://github.com/ceph/ceph-qa-suite/pull/925

Actions

Copy link

#8

Updated by Loïc Dachary about 8 years ago

Running with https://github.com/ceph/s3-tests/pull/104 applied seems to fix the problem. Upgrading gevent from 0.13 to > 1.0 + changing the API calls that are not backward compatible.

Actions

Copy link

#9

Updated by Abhishek Lekshmanan about 8 years ago

Status changed from 12 to In Progress
Assignee set to Abhishek Lekshmanan

Actions

Copy link

#10

Updated by Abhishek Lekshmanan about 8 years ago

Assignee deleted (~~Abhishek Lekshmanan~~)

Actions

Copy link

#11

Updated by Loïc Dachary about 8 years ago

<loicd> owasserm: how does master become ceph-master ? 
<loicd> owasserm: the idea is to fix the tests that people use, not the tests that noone uses ;-)
<owasserm> loicd,  ceph-master is the stable branch were the s3tests pass without failure. We will also have ceph-jewel for jewel
<owasserm> s3tests are used with other s3 providers
<loicd> owasserm: what is the difference between ceph-master and master ? 
<owasserm> loicd, at the moment one commit
<owasserm> sage rebased ceph-master last week
<loicd> https://github.com/ceph/s3-tests/commit/41c8be0584530d189c1c245a88073e771ac05e53
<owasserm> loicd, it is like the difference between master and jewel
<loicd> ok
<loicd> owasserm: so, since you don't see an issue with 104 it could be merged in master. But it won't be used for testing until it is merged in ceph-master. Is that right ? 
<owasserm> loicd, yes 
<loicd> owasserm: and http://pulpito.ceph.com/?suite=rgw&branch=master jobs involving s3-test use the ceph-master branch ? 
<loicd> hum, or not
<owasserm> loicd, yes it does
<loicd> owasserm: reading https://github.com/ceph/ceph-qa-suite/blob/master/tasks/s3tests.py#L127 suggest that if it does it does not come from this function. Is it explicitly set somewhere ? 
<owasserm> loicd, probably see in logs 2016-04-10T13:37:50.513 INFO:tasks.s3tests:Using branch 'ceph-master' for s3tests
<owasserm> loicd, here branch = cconf.get('branch', 'ceph-' + suite_branch)
<owasserm> line 136
<loicd> ok
<loicd> good :-)
<loicd> owasserm: I get how commits should travel and when they are supposed to end, thanks :-)

Actions

Copy link

#12

Updated by Loïc Dachary about 8 years ago

Status changed from In Progress to 17

now that the PR is merged, there is a need to schedule a rgw on the sepia lab with the master branch of s3test to assert that https://github.com/ceph/s3-tests/pull/104 does not disrupt anything

Actions

Copy link

#13

Updated by Yehuda Sadeh over 7 years ago

Status changed from 17 to Resolved

Project

General

Profile

Ceph » rgw

Custom queries

Bug #13885

rgw: DNSError on OpenStack

Updated by Loïc Dachary over 8 years ago

Updated by Yuri Weinstein about 8 years ago

Updated by Loïc Dachary about 8 years ago

Updated by Loïc Dachary about 8 years ago

Updated by Loïc Dachary about 8 years ago

Updated by Loïc Dachary about 8 years ago

Updated by Loïc Dachary about 8 years ago

Updated by Loïc Dachary about 8 years ago

Updated by Abhishek Lekshmanan about 8 years ago

Updated by Abhishek Lekshmanan about 8 years ago

Updated by Loïc Dachary about 8 years ago

Updated by Loïc Dachary about 8 years ago

Updated by Yehuda Sadeh over 7 years ago