Bug #43720: qa/workunits/rest/test-restful.sh fails - mgr - Ceph

Actions

Copy link

Bug #43720

closed

qa/workunits/rest/test-restful.sh fails

Added by Sage Weil over 4 years ago. Updated about 3 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Category:

Target version:

Ceph - v15.0.0

% Done:

Source:

Tags:

Backport:

octopus, nautilus, mimic, luminous

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

34310

Crash signature (v1):

Crash signature (v2):

Description

2020-01-20T15:44:48.673 INFO:tasks.workunit.client.a.smithi017.stdout:URL = https://smithi017:8003/perf?daemon=.*
2020-01-20T15:44:48.685 INFO:tasks.workunit.client.a.smithi017.stderr:Traceback (most recent call last):
2020-01-20T15:44:48.685 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 302, in _error_catcher
2020-01-20T15:44:48.685 INFO:tasks.workunit.client.a.smithi017.stderr:    yield
2020-01-20T15:44:48.686 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 384, in read
2020-01-20T15:44:48.686 INFO:tasks.workunit.client.a.smithi017.stderr:    data = self._fp.read(amt)
2020-01-20T15:44:48.686 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3.6/http/client.py", line 459, in read
2020-01-20T15:44:48.686 INFO:tasks.workunit.client.a.smithi017.stderr:    n = self.readinto(b)
2020-01-20T15:44:48.686 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3.6/http/client.py", line 503, in readinto
2020-01-20T15:44:48.686 INFO:tasks.workunit.client.a.smithi017.stderr:    n = self.fp.readinto(b)
2020-01-20T15:44:48.686 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3.6/socket.py", line 586, in readinto
2020-01-20T15:44:48.687 INFO:tasks.workunit.client.a.smithi017.stderr:    return self._sock.recv_into(b)
2020-01-20T15:44:48.687 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
2020-01-20T15:44:48.687 INFO:tasks.workunit.client.a.smithi017.stderr:    return self.read(nbytes, buffer)
2020-01-20T15:44:48.687 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3.6/ssl.py", line 874, in read
2020-01-20T15:44:48.687 INFO:tasks.workunit.client.a.smithi017.stderr:    return self._sslobj.read(len, buffer)
2020-01-20T15:44:48.687 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3.6/ssl.py", line 631, in read
2020-01-20T15:44:48.687 INFO:tasks.workunit.client.a.smithi017.stderr:    v = self._sslobj.read(len, buffer)
2020-01-20T15:44:48.688 INFO:tasks.workunit.client.a.smithi017.stderr:ConnectionResetError: [Errno 104] Connection reset by peer
2020-01-20T15:44:48.688 INFO:tasks.workunit.client.a.smithi017.stderr:
2020-01-20T15:44:48.688 INFO:tasks.workunit.client.a.smithi017.stderr:During handling of the above exception, another exception occurred:
2020-01-20T15:44:48.688 INFO:tasks.workunit.client.a.smithi017.stderr:
2020-01-20T15:44:48.688 INFO:tasks.workunit.client.a.smithi017.stderr:Traceback (most recent call last):
2020-01-20T15:44:48.688 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3/dist-packages/requests/models.py", line 745, in generate
2020-01-20T15:44:48.688 INFO:tasks.workunit.client.a.smithi017.stderr:    for chunk in self.raw.stream(chunk_size, decode_content=True):
2020-01-20T15:44:48.688 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 436, in stream
2020-01-20T15:44:48.689 INFO:tasks.workunit.client.a.smithi017.stderr:    data = self.read(amt=amt, decode_content=decode_content)
2020-01-20T15:44:48.689 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 401, in read
2020-01-20T15:44:48.689 INFO:tasks.workunit.client.a.smithi017.stderr:    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
2020-01-20T15:44:48.689 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
2020-01-20T15:44:48.689 INFO:tasks.workunit.client.a.smithi017.stderr:    self.gen.throw(type, value, traceback)
2020-01-20T15:44:48.689 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 320, in _error_catcher
2020-01-20T15:44:48.689 INFO:tasks.workunit.client.a.smithi017.stderr:    raise ProtocolError('Connection broken: %r' % e, e)
2020-01-20T15:44:48.689 INFO:tasks.workunit.client.a.smithi017.stderr:urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
2020-01-20T15:44:48.690 INFO:tasks.workunit.client.a.smithi017.stderr:
2020-01-20T15:44:48.690 INFO:tasks.workunit.client.a.smithi017.stderr:During handling of the above exception, another exception occurred:
2020-01-20T15:44:48.690 INFO:tasks.workunit.client.a.smithi017.stderr:
2020-01-20T15:44:48.690 INFO:tasks.workunit.client.a.smithi017.stderr:Traceback (most recent call last):
2020-01-20T15:44:48.690 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/home/ubuntu/cephtest/clone.client.a/qa/workunits/rest/test_mgr_rest_api.py", line 91, in <module>
2020-01-20T15:44:48.690 INFO:tasks.workunit.client.a.smithi017.stderr:    auth=auth)
2020-01-20T15:44:48.690 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3/dist-packages/requests/api.py", line 72, in get
2020-01-20T15:44:48.691 INFO:tasks.workunit.client.a.smithi017.stderr:    return request('get', url, params=params, **kwargs)
2020-01-20T15:44:48.691 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3/dist-packages/requests/api.py", line 58, in request
2020-01-20T15:44:48.691 INFO:tasks.workunit.client.a.smithi017.stderr:    return session.request(method=method, url=url, **kwargs)
2020-01-20T15:44:48.691 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request
2020-01-20T15:44:48.691 INFO:tasks.workunit.client.a.smithi017.stderr:    resp = self.send(prep, **send_kwargs)
2020-01-20T15:44:48.691 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 670, in send
2020-01-20T15:44:48.691 INFO:tasks.workunit.client.a.smithi017.stderr:    r.content
2020-01-20T15:44:48.691 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3/dist-packages/requests/models.py", line 823, in content
2020-01-20T15:44:48.692 INFO:tasks.workunit.client.a.smithi017.stderr:    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
2020-01-20T15:44:48.692 INFO:tasks.workunit.client.a.smithi017.stderr:  File "/usr/lib/python3/dist-packages/requests/models.py", line 748, in generate
2020-01-20T15:44:48.692 INFO:tasks.workunit.client.a.smithi017.stderr:    raise ChunkedEncodingError(e)
2020-01-20T15:44:48.692 INFO:tasks.workunit.client.a.smithi017.stderr:requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

/a/sage-2020-01-20_14:55:07-rados-wip-sage-testing-2020-01-19-1713-distro-basic-smithi/4688294
description: rados/rest/{mgr-restful.yaml supported-random-distro$/{ubuntu_latest.yaml}}

Related issues 4 (0 open — 4 closed)

Actions

Copy link

Updated by Sage Weil about 4 years ago

//a/sage-2020-01-24_23:29:53-rados-wip-sage2-testing-2020-01-24-1408-distro-basic-smithi/4702997

Actions

Copy link

Updated by Sage Weil about 4 years ago

Target version set to v15.0.0

Actions

Copy link

Updated by Sage Weil about 4 years ago

1. This appears to only happen on 18.04, but not on el 8:

el8: http://pulpito.ceph.com/sage-2020-01-27_15:35:35-rados-wip-sage2-testing-2020-01-24-1408-distro-basic-smithi
18.04: http://pulpito.ceph.com/sage-2020-01-27_15:35:27-rados-wip-sage2-testing-2020-01-24-1408-distro-basic-smithi

2. It looks like a client-side error message? In the log, the request returns 200.

Actions

Copy link

Updated by Sage Weil about 4 years ago

Pull request ID set to 32920

1. This appears to only happen on 18.04, but not on el 8:

2. It looks like a client-side error message? In the log, the request returns 200.

Actions

Copy link

Updated by Sage Weil about 4 years ago

Status changed from New to Fix Under Review

Actions

Copy link

Updated by Sage Weil about 4 years ago

Status changed from Fix Under Review to Resolved

Actions

Copy link

Updated by Brad Hubbard about 4 years ago

Status changed from Resolved to In Progress

Re-opening since this is still broken during ceph-ansible tests which run on centos and ubuntu latest.

Actions

Copy link

Updated by Brad Hubbard about 4 years ago

Not sure why this is only on Ubuntu but when we do a get request the "data" argument appears to be being sent (despite being an empty json object) and this seems to confuse the transaction.

I have a patch.

Actions

Copy link

Updated by Josh Durgin about 4 years ago

Status changed from In Progress to Pending Backport
Backport set to octopus, nautilus, mimic

Actions

Copy link

#10

Updated by Brad Hubbard about 4 years ago

Pull request ID changed from 32920 to 34310

Actions

Copy link

#11

Updated by Brad Hubbard about 4 years ago

Status changed from Pending Backport to In Progress

Actions

Copy link

#12

Updated by Brad Hubbard about 4 years ago

Status changed from In Progress to Pending Backport
Backport changed from octopus, nautilus, mimic to octopus, nautilus, mimic, luminous