Ceph » rgw

I was testing the Open Policy Agent integration in radosgw, and I noticed in OPA logs that its responses were taking about 40 ms (`resp_duration` below):

``` {
"client_addr": "127.0.0.1:51344",
"level": "info",
"msg": "Sent response.",
"req_id": 3,
"req_method": "POST",
"req_path": "/v1/data/ceph/allow",
"resp_body": "{\"result\":true}",
"resp_bytes": 15,
"resp_duration": 40.226543,
"resp_status": 200,
"time": "2021-08-05T11:16:35Z"
}
```

This was surprising to me because I'm using a very simple policy which I expected would take less than 1 ms to process. And indeed, POSTing the same document with curl gave me much shorter response times:

```{
"client_addr": "127.0.0.1:59590",
"level": "info",
"msg": "Sent response.",
"req_id": 7,
"req_method": "POST",
"req_path": "/v1/data/ceph/allow",
"resp_bytes": 15,
"resp_duration": 0.911273,
"resp_status": 200,
"time": "2021-08-05T13:40:40Z"
}
```

Capturing radosgw and curl requests with tcpdump showed that curl passes the `Expect: 100-continue` header in its POST request, while radosgw doesn't. As a result, when radosgw makes a request, it sends the HTTP headers, waits for a TCP ACK from OPA, and then sends the body; and due to delayed ACK [1], the ACK usually comes 40 ms after sending the headers. By contrast, with `Expect: 100-continue`, OPA sends `HTTP/1.1 100 Continue` immediately after receiving the headers, so curl can proceed right away with sending the body of the request.

[1]: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment