Bug #52067
closedRadosGW requests to Open Policy Agent add 40ms latency
100%
Description
I was testing the Open Policy Agent integration in radosgw, and I noticed in OPA logs that its responses were taking about 40 ms (`resp_duration` below):
```
{
"client_addr": "127.0.0.1:51344",
"level": "info",
"msg": "Sent response.",
"req_id": 3,
"req_method": "POST",
"req_path": "/v1/data/ceph/allow",
"resp_body": "{\"result\":true}",
"resp_bytes": 15,
"resp_duration": 40.226543,
"resp_status": 200,
"time": "2021-08-05T11:16:35Z"
}
```
This was surprising to me because I'm using a very simple policy which I expected would take less than 1 ms to process. And indeed, POSTing the same document with curl gave me much shorter response times:
```{
"client_addr": "127.0.0.1:59590",
"level": "info",
"msg": "Sent response.",
"req_id": 7,
"req_method": "POST",
"req_path": "/v1/data/ceph/allow",
"resp_bytes": 15,
"resp_duration": 0.911273,
"resp_status": 200,
"time": "2021-08-05T13:40:40Z"
}
```
Capturing radosgw and curl requests with tcpdump showed that curl passes the `Expect: 100-continue` header in its POST request, while radosgw doesn't. As a result, when radosgw makes a request, it sends the HTTP headers, waits for a TCP ACK from OPA, and then sends the body; and due to delayed ACK [1], the ACK usually comes 40 ms after sending the headers. By contrast, with `Expect: 100-continue`, OPA sends `HTTP/1.1 100 Continue` immediately after receiving the headers, so curl can proceed right away with sending the body of the request.
[1]: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment
Updated by Casey Bodley almost 3 years ago
- Status changed from New to Pending Backport
- Tags set to opa
- Backport set to octopus pacific
Updated by Backport Bot almost 3 years ago
- Copied to Backport #52241: pacific: RadosGW requests to Open Policy Agent add 40ms latency added
Updated by Backport Bot almost 3 years ago
- Copied to Backport #52242: octopus: RadosGW requests to Open Policy Agent add 40ms latency added
Updated by Backport Bot almost 2 years ago
- Tags changed from opa to opa backport_processed
Updated by Konstantin Shalygin 6 months ago
- Status changed from Pending Backport to Resolved
- % Done changed from 0 to 100