Bug #4902
Issuess handling very large files
0%
Description
Hi,
I am new to Ceph. I am running version 0.56 on Debian Squeeze. I would like to use it to store very large files (over 100G).
I have been able to upload a file using S3 multipart uploads.
However now I am getting error 500 when I try to access the uploaded file:
b
<Bucket: bucket>
a = b.get_key('foo')
boto.exception.BotoServerError: BotoServerError: 500
In the web server logs I see the following:
89.233.152.242 - - [03/May/2013:12:10:33 +0200] "HEAD /bucket/foo HTTP/1.1" 500 0 "-" "Boto/2.8.0 (darwin)"
This is what gets reported to RADOS GW logs:
2013-05-03 11:53:09.338587 7f0a263f1700 20 rgw_get_bucket_info: bucket=bucket(.rgw.buckets[4313.2]) owner foo
.rgw.buckets[4313.2]):foo only_bucket=0 ret=-7
2013-05-03 11:53:09.338598 7f0a263f1700 20 get_obj_state: rctx=0x7f0a2802a200 obj=ingest: state=0x7f0a28032e38 s->prefetch_data=0
2013-05-03 11:53:09.338607 7f0a263f1700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>foo</ID><DisplayName>foo</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>foo</ID><DisplayName>foo</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2013-05-03 11:53:09.338620 7f0a263f1700 20 get_obj_state: rctx=0x7f0a2802a200 obj=bucket:foo state=0x7f0a28035cf8 s->prefetch_data=1
2013-05-03 11:53:09.342230 7f0a263f1700 10 read_permissions on bucket(
2013-05-03 11:53:09.342237 7f0a263f1700 0 WARNING: set_req_state_err err_no=7 resorting to 500
The error comes from osd.24, which is reporting the following:
2013-05-03 12:21:24.528923 7f2063280700 1 -- x:6808/3478 --> y:0/1002260 -- osd_op_reply(152 4313.2_foo [getxattrs,stat,read 0~524288] = 7 (Argument ?+0 0x2986c00 con 0x26fd6e0
list too long)) v4 -
Please let me know if you need more information to investigate the issue.
History
#1 Updated by Anonymous almost 11 years ago
- Priority changed from Normal to Urgent
#2 Updated by Ian Colle almost 11 years ago
- Assignee set to Yehuda Sadeh
#3 Updated by Ian Colle almost 11 years ago
- Status changed from New to In Progress
#4 Updated by Yehuda Sadeh almost 11 years ago
Can you try running:
$ rados -p .rgw.buckets listxattr 4313.2_foo
and
$ rados -p .rgw.buckets getxattr 4313.2_foo user.rgw.manifest | wc
#5 Updated by Jiri Brunclik almost 11 years ago
# rados -p .rgw.buckets listxattr 4313.2_foo error getting xattr set .rgw.buckets/4313.2_foo: Argument list too long # rados -p .rgw.buckets getxattr 4313.2_foo user.rgw.manifest | wc 1137 34935 5705332
#6 Updated by Yehuda Sadeh almost 11 years ago
Are you using xfs? What happens is that we split the extended attribute data over multiple xattrs, as it became too big. Apparently there's some limit in the file system on the amount of data it can return on listxattr. Usually we don't hit this problem as we switched to putting all the xattrs on a different structure, but it may not be the case for xfs.You can try adding 'filestore xattr use omap = true' to your ceph.conf under your osd section. Then try to do the following:
$ rados -p .rgw.buckets cp 4313.2_foo _4313.2_foo $ rados -p .rgw.buckets cp _4313.2_foo 4313.2_foo
This should rewrite the object with the attributes over omap.
#7 Updated by Yehuda Sadeh almost 11 years ago
On a second thought, the 'rados cp' stuff isn't going to work as it won't be able to list the xattrs.
#8 Updated by Jiri Brunclik almost 11 years ago
Yes, the underlying filesystem is XFS.
I enabled filestore xattr use omap
and re-uploaded the file and now it seems to be working just fine. But still, it is weird. I thought that XFS does not have any limits on xattr size...
Now the open question is how to get rid of the previously uploaded file. Obviously it cannot be removed through the S3 API. I ran the following:
# for i in $(rados -p .rgw.buckets ls | grep foo); do rados -p .rgw.buckets rm $i; done
What else needs to be done to clean it up?
#9 Updated by Ian Colle almost 11 years ago
- Priority changed from Urgent to Normal
Added ticket 4914 to add ability to read xattr from file / stdin
#10 Updated by Sage Weil almost 11 years ago
- Status changed from In Progress to Resolved