Ceph » rgw

Some Background:

- This issue has been reproduced in Firefly and Hammer
- We have seen this issue on a variety of cluster sizes including clusters as large as 100 osds and clusters as small as 3 nodes on openstack.
- We cannot reproduce this issue using the S3 API. It appears to be a Swift specific problem. The S3 api will HEAD the same file split into the same amount of chunks in 0.0078 seconds.
- We are using the mod_fastcgi (2.4.7) module with httpd version 2.4.6.
- Our fastcgi timeout is set to 1600 seconds.
- We disable the fastcgi wrapper in the fastcgi conf like so: "FastCgiWrapper Off"

Steps to reproduce our exact problem:

1. Create a Swift user with full permissions.

2. Create an 80GB file by doing the following command:

xfs_mkfile 85774958592 80GB.txt

3. Upload the file to Swift by breaking into 200MB object segments. We use python-swiftclient's cli interface like so:

swift --insecure upload test_head --segment-size 200000000 --segment-threads=10 80GB.txt

4. Attempt to head the file, here is how we are doing it: [[https://gist.github.com/Atothendrew/2c0e904f339d0e8d02fa]]. The request should timeout.

We can loop through HEAD requests against the ~400 parts manually by doing the following: [[https://gist.github.com/Atothendrew/6012e8a751cd136a1607]]. This usually takes around 15 seconds. We are very confused as to why it is taking so long to do a HEAD request against the base object.