Feature #11326
openrgw: make MultiDelete faster/parallelized
0%
Description
I'm using DreamObejcts (with is using Ceph as they fs) and it's very slow MultiDelete compared to AWS. Ceph is doing 10 deletes/second using multidelete on 500 objects.
So I decided to make the MultiDelete run parallel, but, each time I do more than one request at the same time, it return 500 (after some time, is not fast fail)
I don't have Ceph on my machine, so I can't test, but I guess someone here is capable of testing it.
Updated by Sage Weil about 9 years ago
- Project changed from Ceph to rgw
- Priority changed from High to Urgent
Updated by Yehuda Sadeh about 9 years ago
the 500 is a symptom of apache timing out. The delete operation takes too long, so apache is timing out.
Updated by renato cron about 9 years ago
Hmm, good!
I'll report it to dreamcompute.
Somethings to do so:
1: Change 500 to 408 Request Timeout Error.
2: Find a way to make delete less slow.
In my opinion, Ceph is design to scale horizontally, so, delete should scale as good as write.
I think I will need distribute my system to use a bucket for each hour, eg: "foo:00", "foo:01", ..., "foo:23" so I can do parallel requests and maybe the servers are different.
But, to know that for sure, I need to known: How 'buckets' are distributed on Ceph?
Thank you for your time.
Updated by Sage Weil almost 9 years ago
- Tracker changed from Bug to Feature
- Subject changed from Concurrent delete using MultiDelete s3 API causes 500 to rgw: make MultiDelete faster/parallelized
- Source changed from other to Community (user)