Project

General

Profile

Actions

Bug #58671

closed

Frontend socket leak that leads to OOM when connections are reset

Added by Yixin Jin about 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
% Done:

100%

Source:
Tags:
ssl beast backport_processed
Backport:
pacific quincy
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This issue is related to the following type of traffic:
  1. Client uses non-persistent connection (Connection: close).
  2. Client uses SO_LINGER option, which causes it to send RST when the socket is closed. The server side gets ECONNRESET.
  3. Client does HTTPS requests.

As the result, those sockets that got ECONNRESET would still go ahead to call async_shutdown() in the request coroutine because that error isn't propagated all the way up to the coroutine. After handle_connection() call, it thinks everything is OK so it calls async_shutdown() to shutdown SSL stream. However, the call to async_shutdown() never returns since epoll no longer delivers events on this socket after it delivered the one for ECONNRESET or when ECONNRESET is returned upon sendmsg(). So, the call is stuck and we have a socket/coroutine leak. Each coroutine consumes 512KB stack and eventually it will OOM and crash.


Related issues 2 (0 open2 closed)

Copied to rgw - Backport #58769: pacific: Frontend socket leak that leads to OOM when connections are resetResolvedCasey BodleyActions
Copied to rgw - Backport #58770: quincy: Frontend socket leak that leads to OOM when connections are resetResolvedCasey BodleyActions
Actions

Also available in: Atom PDF