Project

General

Profile

Bug #4902

Issuess handling very large files

Added by Jiri Brunclik almost 11 years ago. Updated almost 11 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi,

I am new to Ceph. I am running version 0.56 on Debian Squeeze. I would like to use it to store very large files (over 100G).

I have been able to upload a file using S3 multipart uploads.

However now I am getting error 500 when I try to access the uploaded file:

b

<Bucket: bucket>

a = b.get_key('foo')

boto.exception.BotoServerError: BotoServerError: 500

In the web server logs I see the following:

89.233.152.242 - - [03/May/2013:12:10:33 +0200] "HEAD /bucket/foo HTTP/1.1" 500 0 "-" "Boto/2.8.0 (darwin)"

This is what gets reported to RADOS GW logs:

2013-05-03 11:53:09.338587 7f0a263f1700 20 rgw_get_bucket_info: bucket=bucket(.rgw.buckets[4313.2]) owner foo
2013-05-03 11:53:09.338598 7f0a263f1700 20 get_obj_state: rctx=0x7f0a2802a200 obj=ingest: state=0x7f0a28032e38 s->prefetch_data=0
2013-05-03 11:53:09.338607 7f0a263f1700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>foo</ID><DisplayName>foo</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>foo</ID><DisplayName>foo</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2013-05-03 11:53:09.338620 7f0a263f1700 20 get_obj_state: rctx=0x7f0a2802a200 obj=bucket:foo state=0x7f0a28035cf8 s->prefetch_data=1
2013-05-03 11:53:09.342230 7f0a263f1700 10 read_permissions on bucket(
.rgw.buckets[4313.2]):foo only_bucket=0 ret=-7
2013-05-03 11:53:09.342237 7f0a263f1700 0 WARNING: set_req_state_err err_no=7 resorting to 500

The error comes from osd.24, which is reporting the following:

2013-05-03 12:21:24.528923 7f2063280700 1 -- x:6808/3478 --> y:0/1002260 -- osd_op_reply(152 4313.2_foo [getxattrs,stat,read 0~524288] = 7 (Argument
list too long)) v4 -
?+0 0x2986c00 con 0x26fd6e0

Please let me know if you need more information to investigate the issue.

History

#1 Updated by Anonymous almost 11 years ago

  • Priority changed from Normal to Urgent

#2 Updated by Ian Colle almost 11 years ago

  • Assignee set to Yehuda Sadeh

#3 Updated by Ian Colle almost 11 years ago

  • Status changed from New to In Progress

#4 Updated by Yehuda Sadeh almost 11 years ago

Can you try running:

$ rados -p .rgw.buckets listxattr 4313.2_foo 

and

$ rados -p .rgw.buckets getxattr 4313.2_foo user.rgw.manifest | wc

#5 Updated by Jiri Brunclik almost 11 years ago

# rados -p .rgw.buckets listxattr 4313.2_foo
error getting xattr set .rgw.buckets/4313.2_foo: Argument list too long
# rados -p .rgw.buckets getxattr 4313.2_foo user.rgw.manifest | wc
   1137   34935 5705332

#6 Updated by Yehuda Sadeh almost 11 years ago

Are you using xfs? What happens is that we split the extended attribute data over multiple xattrs, as it became too big. Apparently there's some limit in the file system on the amount of data it can return on listxattr. Usually we don't hit this problem as we switched to putting all the xattrs on a different structure, but it may not be the case for xfs.You can try adding 'filestore xattr use omap = true' to your ceph.conf under your osd section. Then try to do the following:

$ rados -p .rgw.buckets cp 4313.2_foo _4313.2_foo
$ rados -p .rgw.buckets cp _4313.2_foo 4313.2_foo

This should rewrite the object with the attributes over omap.

#7 Updated by Yehuda Sadeh almost 11 years ago

On a second thought, the 'rados cp' stuff isn't going to work as it won't be able to list the xattrs.

#8 Updated by Jiri Brunclik almost 11 years ago

Yes, the underlying filesystem is XFS.

I enabled filestore xattr use omap and re-uploaded the file and now it seems to be working just fine. But still, it is weird. I thought that XFS does not have any limits on xattr size...

Now the open question is how to get rid of the previously uploaded file. Obviously it cannot be removed through the S3 API. I ran the following:

# for i in $(rados -p .rgw.buckets ls | grep foo); do rados -p .rgw.buckets rm $i; done

What else needs to be done to clean it up?

#9 Updated by Ian Colle almost 11 years ago

  • Priority changed from Urgent to Normal

Added ticket 4914 to add ability to read xattr from file / stdin

#10 Updated by Sage Weil almost 11 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF