Project

General

Profile

Bug #24401

RadosGW credential locks itself during operation

Added by David González Ruiz almost 6 years ago. Updated over 5 years ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I have been seeing a behavior ever since I use Ceph, which at first I believed to be due to a misuse on my side, but now I clearly see as a flaw in RadosGW (or, a feature, which has this unforeseen consequence).

I have a java service that uses the S3 java client library to connect to RadosGW. I use SSL to encrypt connections to the gateways. This java service uses multi-threading, and essentially, it connects to RabbitMQ to retrieve storage requests asynchronously (one consumer per thread). Once this request is read from the queue, the the thread uploads the document to RGW with a provided key.

I came to the realization that, if all these multiple threads (can be up to 100) are uploading to RGW at the same time, the specific key which is performing an operation remains locked, and then any other thread trying to use the same key to store anything is unable to finish the operation in RGW until the previous one, happening in a different thread, is finished. The performance of the consumers because of this was really bad. After I created a huge subset of keys for this same user (one per thread), the performance of the service increased greatly, as there were no inter-locking issues happening, and all could store/retrieve documents at the same time without problems. Note that this is not user based (as the user remains the same), but authentication based. In case it helps, the operations don't even necessarily happen in the same buckets. This user owns exactly 4097 buckets (per design of the data storage system), and the operations happen randomly at any of them.

It would seem to me that RGW locks the key until the operation is finished, probably to ensure consistency of the data among users of this authentication method (key). I would be interested in knowing if this is expected, and in that case, if there's any other better mechanism to address the locking.

I am using ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)

Thank you in advance, to anyone who offers his/her support in this.

History

#1 Updated by Yehuda Sadeh almost 6 years ago

RGW doesn't lock requests by the access key. Maybe it's your java client that does it? Try running separate clients, see how it behaves. Or maybe you're running out of rgw processing threads, maybe try to increase the number?

#2 Updated by Abhishek Lekshmanan over 5 years ago

  • Status changed from New to Need More Info

Also available in: Atom PDF