Actions
Bug #1638
closedCan't create object with large xattrs in a single operation (on extN)
Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
A single compound operation that does:
- create
- setxattr (small enough to fit but large enough to fill in the ext3/ext4 space for xattrs)
the operation can have multiple setxattrs, so enough xattrs on the operation will eventually fill up the space.
we'll end up with either bad object, or a bad one. The operation itself fill return success, but next operations will return -ENOENT.
2011-10-19 16:25:05.874133 7fc34230a710 filestore(dev/osd0) ENOSPC on setxattr on 10.1_head/255_2766b97ec2b249cfaa5e8c02e399c972/head name _user.rgw.shadow_name size 65 2011-10-19 16:25:05.874157 7fc34230a710 filestore(dev/osd0) WARNING: ignoring setattr ENOSPC failure, until we implement a workaround for extN xattr limitations [the next xattrs failing is a real problem] 2011-10-19 16:25:05.915935 7fc34230a710 filestore(dev/osd0) ENOSPC on setxattr on 10.1_head/255_2766b97ec2b249cfaa5e8c02e399c972/head name _ size 196 2011-10-19 16:25:05.915959 7fc34230a710 filestore(dev/osd0) WARNING: ignoring setattr ENOSPC failure, until we implement a workaround for extN xattr limitations 2011-10-19 16:25:05.966122 7fc34230a710 filestore(dev/osd0) ENOSPC on setxattr on 10.1_head/255_2766b97ec2b249cfaa5e8c02e399c972/head name snapset size 26 2011-10-19 16:25:05.966145 7fc34230a710 filestore(dev/osd0) WARNING: ignoring setattr ENOSPC failure, until we implement a workaround for extN xattr limitations 2011-10-19 16:25:05.966452 7fc340b07710 -- 10.0.1.203:6808/3605 --> 10.0.1.245:0/1007286 -- osd_op_reply(17731 255_2766b97ec2b249cfaa5e8c02e399c972 [delete,create 0~0,setxattr user.rgw.acl (102), setxattr user.rgw.content_type (25),setxattr user.rgw.etag (33),setxattr user.rgw.x-amz-meta-0298e4d400d74a3ebc7b727e1 (26),setxattr user.rgw.x-amz-meta-04db42bb62024aaaa7c13712f (26),setxattr user.rgw.x-amz-meta-08e6c6c6827b4ffda941e87ac (26),setxattr user.rgw.x-amz-meta-09dd3319c8bf4db1a32b441d0 (26),setxattr user.rgw.x-amz-meta-0b2310bc14a945aabc8b24cf0 (26), ... 2011-10-19 16:25:05.966122 7fc34230a710 filestore(dev/osd0) ENOSPC on setxattr on 10.1_head/255_2766b97ec2b249cfaa5e8c02e399c972/head name snapset size 26 2011-10-19 16:25:05.966145 7fc34230a710 filestore(dev/osd0) WARNING: ignoring setattr ENOSPC failure, until we implement a workaround for extN xattr limitations 2011-10-19 16:25:05.966452 7fc340b07710 -- 10.0.1.203:6808/3605 --> 10.0.1.245:0/1007286 -- osd_op_reply(17731 255_2766b97ec2b249cfaa5e8c02e399c972 [delete,create 0~0,setxattr user.rgw.acl (102), setxattr user.rgw.content_type (25),setxattr user.rgw.etag (33),setxattr user.rgw.x-amz-meta-0298e4d400d74a3ebc7b727e1 (26),setxattr user.rgw.x-amz-meta-04db42bb62024aaaa7c13712f (26),setxattr user.rgw.x-amz-meta-08e6c6c6827b4ffda941e87ac (26),setxattr user.rgw.x-amz-meta-09dd3319c8bf4db1a32b441d0 (26),setxattr user.rgw.x-amz-meta-0b2310bc14a945aabc8b24cf0 (26),setxattr user.rgw.x-amz- meta-0cfdad44837e4053b97652af1 (26),setxattr user.rgw.x-amz-meta-18cbc449b653437e82f466f58 (26),setxattr user.rgw.x-amz-meta-204d06dbb7a846c59f2873bf1 (26),setxattr user.rgw.x-amz-meta- ... edebfde15dee468cb86b98c62 (26),setxattr user.rgw.x-amz-meta-fa0c1cd891d54e3fba63e541e (26),setxattr user.rgw.x-amz-meta-ff51ea2b24fb4abd96c591c49 (26),setxattr user.rgw.idtag (32),setxattr user. rgw.shadow_name (65)] ack = 0) v1 -- ?+0 0x32028c0 con 0x31e6500 <-- here we return success ... [next operation on the same object returns ENOENT] 011-10-19 16:25:05.977834 7fc33f304710 -- 10.0.1.203:6808/3605 <== client.4149 10.0.1.245:0/1007286 17737 ==== osd_op(client.4149.0:17737 255_2766b97ec2b249cfaa5e8c02e399c972 [getxattrs,stat 0~0] 10.1e4619f1) v3 ==== 204+0+0 (2198824556 0 0) 0x2d5ab40 con 0x31e6500 2011-10-19 16:25:05.977908 7fc33b9fc710 -- 10.0.1.203:6808/3605 --> 10.0.1.245:0/1007286 -- osd_op_reply(17737 255_2766b97ec2b249cfaa5e8c02e399c972 [getxattrs,stat 0~0] = -2 (No such file or directory)) v1 -- ?+0 0x32fb540 con 0x31e6500
Updated by Anonymous over 12 years ago
So now there's an assert on the ENOSPC. I triggered t by running s3tests under teuthology. Adding this here so we know this actually happens:
2012-01-17 09:02:41.957418 7f52bf6ec700 osd.0 18 got the last reply from pending watchers, can send response now 2012-01-17 09:02:42.038498 7f52bf6ec700 <cls> cls_rgw.cc:149: rgw_bucket_prepare_op(): request: op=0 name=foo tag=_P49yVOrihlB1LlBLiytZ7xhk0ErO5_g 2012-01-17 09:02:42.185144 7f52bf6ec700 <cls> cls_rgw.cc:192: rgw_bucket_complete_op(): request: op=0 name=foo epoch=37 tag=_P49yVOrihlB1LlBLiytZ7xhk0ErO5_g 2012-01-17 09:02:42.185267 7f52bf6ec700 <cls> cls_rgw.cc:219: rgw_bucket_complete_op(): existing entry: epoch=0 2012-01-17 09:02:42.503259 7f52c57f9700 filestore(/tmp/cephtest/data/osd.0.data) ENOSPC on setxattr on 9.6_head/2012-01-17-09-0-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/head/bda37b7e name _ size 3177 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t)', in thread '7f52c57f9700' os/FileStore.cc: 2421: FAILED assert(0 == "ENOSPC handling not implemented") ceph version 0.40-11-g47db4d0 (commit:47db4d044cff20e76005d6280a561d02c3e95ae2) 1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long)+0x3275) [0x6eed15] 2: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x78) [0x6ef178] 3: (FileStore::_do_op(FileStore::OpSequencer*)+0x176) [0x6c8e16] 4: (ThreadPool::worker()+0xa28) [0x597918] 5: (ThreadPool::WorkThread::entry()+0xd) [0x564b8d] 6: (()+0x7971) [0x7f52cec7d971] 7: (clone()+0x6d) [0x7f52cd30892d] ceph version 0.40-11-g47db4d0 (commit:47db4d044cff20e76005d6280a561d02c3e95ae2) 1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long)+0x3275) [0x6eed15] 2: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x78) [0x6ef178] 3: (FileStore::_do_op(FileStore::OpSequencer*)+0x176) [0x6c8e16] 4: (ThreadPool::worker()+0xa28) [0x597918] 5: (ThreadPool::WorkThread::entry()+0xd) [0x564b8d] 6: (()+0x7971) [0x7f52cec7d971] 7: (clone()+0x6d) [0x7f52cd30892d] *** Caught signal (Aborted) ** in thread 7f52c57f9700 ceph version 0.40-11-g47db4d0 (commit:47db4d044cff20e76005d6280a561d02c3e95ae2) 1: /tmp/cephtest/binary/usr/local/bin/ceph-osd() [0x583b34] 2: (()+0xfb40) [0x7f52cec85b40] 3: (gsignal()+0x35) [0x7f52cd255ba5] 4: (abort()+0x180) [0x7f52cd2596b0] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f52cdaf96bd] 6: (()+0xb9906) [0x7f52cdaf7906] 7: (()+0xb9933) [0x7f52cdaf7933] 8: (()+0xb9a3e) [0x7f52cdaf7a3e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x39f) [0x59f27f] 10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long)+0x3275) [0x6eed15] 11: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x78) [0x6ef178] 12: (FileStore::_do_op(FileStore::OpSequencer*)+0x176) [0x6c8e16] 13: (ThreadPool::worker()+0xa28) [0x597918] 14: (ThreadPool::WorkThread::entry()+0xd) [0x564b8d] 15: (()+0x7971) [0x7f52cec7d971] 16: (clone()+0x6d) [0x7f52cd30892d]
And it really is just setxattr, not the disk filling:
ubuntu@sepia1:/tmp/cephtest$ df -h . Filesystem Size Used Avail Use% Mounted on /dev/sda1 920G 29G 846G 4% /
Actions