Project

General

Profile

Actions

Feature #3494

closed

ceph S3 upload slowly

Added by lollipop king over 11 years ago. Updated over 11 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

Hi, there!
Nowadays I have tried some tests on ceph S3 python API, But I have found that it is very slow to update a big file through the S3 API
Here is my upload program:
*_import boto
import boto.s3.connection
access_key = '70XK3XKRPNMCDOHAWUJY'
secret_key = 'epJ1UXT8hYdwxDnKeznxkCy+UL83TVvhhwhVp5uz'

conn = boto.connect_s3(
aws_access_key_id = access_key,
aws_secret_access_key = secret_key,
host = '192.168.0.18',
is_secure = False,
calling_format=boto.s3.connection.OrdinaryCallingFormat(), )
for bucket in conn.get_all_buckets():
key = bucket.new_key('movie.1')
key.set_contents_from_filename('/root/movie')
for key in bucket.list():
print "{name}\t{size}\t{modified}".format(
name=key.name,
size=key.size,
modified=key.last_modified,
)*
_
And the size of the movie.1 is 1GB, it takes 4 minutes to complete the upload…
It takes only 1m23s to complete the upload by ceph POSIX API after I have mounted the 192.168.0.18:/ to /mnt/ceph
I am wandering why?

Another question is that how the file is stored on the osds when I upload it through the S3 API?
After uploading an 1GB file, using du command in the /ceph_osd0/osd0/current directory, I can get this :
1082864 ./7.7_head
Is the file stored on the osd as a hole?
Can I configure some parameters to strip the file to make it stored like the file uploaded by the POSIX API?
Looking for your reply…
Thank you very much!

Actions #1

Updated by Yehuda Sadeh over 11 years ago

When using the ceph filesystem, the objects are not necessarily being written to the backend immediately, so any performance comparison should take this into account.

Actions #2

Updated by Yehuda Sadeh over 11 years ago

  • Status changed from New to Closed

Closing, need more info about the specific user problem.

Actions

Also available in: Atom PDF