Project

General

Profile

Bug #4754

GET/HEAD on account is extremely slow, times out

Added by Faidon Liambotis almost 11 years ago. Updated almost 11 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Doing a GET or a HEAD on /swift/v1 times out, even after increasing the timeout to 5 minutes. It's hard to know the exact number now, but I'd guess we have about ~30k containers, containing about 200 million files. Swift in comparison responds to this instanteously. To be fair, the hardware is different as we have SSDs for account/container dbs (see #4613) but I don't think it accounts for such a big difference. Note that the radosgw cluster has a tiny proportion of the traffic the Swift cluster has.

I currently can't list all the containers on my cluster nor have a count of files for monitoring. The latter is important but the former is critical for operations, hence raising this to a high priority.

History

#1 Updated by Yehuda Sadeh almost 11 years ago

Right. There are two issues that play together here. One is that we don't paginate the request, the second one is that in swift, the response needs to also contain the container stats. The latter is probably what makes it slow. The fix for it would be to do internal pagination. The overall will still be slow, just that it'll stream the response better.
We can also extend the api to make it so that in swift it won't dump the container stats, which will make it much faster, and will probably be ok for most needs.

#2 Updated by Faidon Liambotis almost 11 years ago

Swift doesn't seem to paginate this. I haven't looked at Swift's internals for this but I doubt it lists all of my containers to get X-Account-Container-Count/X-Account-Bytes-Used/X-Account-Bytes-Used when I HEAD/GET the account…

#3 Updated by Faidon Liambotis almost 11 years ago

Corrected by a colleague of mine: Swift does paginate, at 10.000 items. It would help but not that much in my case as it all 4 pages and radosgw is currently orders of magnitude slower.

#4 Updated by Yehuda Sadeh almost 11 years ago

I opened 3 issues for this problem:: #4759, #4760, #4761. These will make it so that it'll be possible to list containers without the need for stats, will make it so that the result will be more streamlined (and avoiding the timeouts), and will make it overall faster to list the containers.

#5 Updated by Faidon Liambotis almost 11 years ago

So, I tried master, 3cc106453f79a0a0c332b164e282a35234a85659 with
curl -D - -H "X-Auth-Token: ..." 'http://localhost/swift/v1'
curl -D - -H "X-Auth-Token: ..." 'http://localhost/swift/v1?format=json'
curl -D - -H "X-Auth-Token: ..." 'http://localhost/swift/v1?format=json&stats=false'

All of plain, JSON and XML outputs print 9800 containers +/- a few dozen, then it hangs for a while, then Apache FastCGI timeouts and appends(!) an HTML message to the output:

plain:

[...]wikipedia-fi-local-deleted.u7wikipedia-fi-loca<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>200 OK</title>
</head><body>
<h1>OK</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>

or with JSON:

,{"name":"wikipedia-fi-local-deleted.yb"},{"name":"wikipedia-fi-loca<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>200 OK</title>
</head><body>

Even when I do request format=json with stats, the output comes normally in batches every 2-3 seconds, until it gets to a similar point (on the wikipedia-fi-local-deleted-* range) and timeouts.

So maybe there's a bug after all and it was just hidden by timing out the single output.

#6 Updated by Faidon Liambotis almost 11 years ago

Oh, I actually see a couple of fixes that match this description in subsequent commits that haven't reached gitbuilder yet. Apologies, I'll have a look first and let you know.

#7 Updated by Faidon Liambotis almost 11 years ago

OK, I actually ran a version that has all that fixed this time :)

Both the text/plain view and stats=false return instantaneously, for all 4 pages of the list. The first page of stats=true returns in a reasonable timeframe (but staggered/streamed, as intended). The second page (?marker=foo) just times out after 30s, which I guess is due to some container down the list being big enough to make the stats gathering slow.

I checked the code to make sure that stats gathering happens only for the items in the current page and not all items before that and from my little understanding of the code, that seems to be the case. No bug there that I could find :)

So, the situation is definitely improved, although I'm afraid not up to the point where 'swift -A ... list' just works.

#8 Updated by Yehuda Sadeh almost 11 years ago

Is your cluster completely healthy? Gathering a single container's stats is not related to the container's size, so if that operation fails / takes too long it means that there's some other issue there. Can you identify whether it's a specific container, and if so which one?

#9 Updated by Faidon Liambotis almost 11 years ago

I have no indications that the cluster is unhealthy (including but not limited to "ceph health"). I tried stating the list of containers one by one and while it took hours, it did finish, so I'm not sure why it'd be slow. Any ideas on what else I could check specifically?

#10 Updated by Yehuda Sadeh almost 11 years ago

Try setting 'rgw list buckets max chunk' to a smaller value (e.g., 100 or even 10). My guess is that certain buckets are slow, but not slow enough to time out a single bucket request.

#11 Updated by Sage Weil almost 11 years ago

  • Status changed from New to Resolved

see #5136 for larger architectural overhaul

Also available in: Atom PDF