Ceph » Linux kernel client

On the osd, it looks to me like CEPH_MDS_OP_SETLAYOUT uses
ceph_file_layout_is_valid() to verify the layout supplied
is valid. The constraints defined for the stripe unit
there are:
- it must be non-zero; and
- it must be a multiple of CEPH_MIN_STRIPE_UNIT=65536

Since the stripe unit is represented in 32 bits, that
means the maximum valid stripe unit is 65535 or 0xffff.

I think an easy fix for now is just to allocate the pagevec_pool
to have objects sufficient to hold pages that would cover a
maximally-sized stripe unit.

Note that this also seems to show that the "wsize" mount
option is ignored if the stripe unit is set larger than
that via ioctl(). (Maybe that's a new bug to fix.)

I implemented a fix for this, and got all the way to
the end of describing it, when I realized the math
makes this impractical. Basically, the largest stripe
is about 4GB or about a million 4K pages. That's 8MB
per pagevec set aside for this out-of-memory case.

I don't think that's practical, so maybe a memory
failure in the midst of writeback should put the
file system in a failure mode and give up.

I can post the patch for review, but at this point I'd
like to have someone argue in favor of it before
doing so.

I'm going to post my explanation below.

From: Alex Elder <elder@inktank.com>
Subject: ceph: size writeback pagevec pool appropriately

There is a memory pool set aside for pagevecs to be used in the
writeback path when a kmalloc() fails, in ceph_writepages_start().
The size of the objects in the pool is based on the "wsize" mount
option. But that is not adequate, and this patch fixes that.

The pool is used if a malloc for a page vector fails deep inside
ceph_writepages_start(). In that case, the number of pages needed
in the vector depends on the inode block size (1 << inode->i_blkbits)
or if specified at mount time, the file system's "wsize" parameter
instead (but at least a page).

The inode block size is defined as the stripe unit value in the
inode's layout, and that can be set to a fairly arbitrary value
using a CEPH_IOC_SET_LAYOUT ioctl() call.

If the "wsize" mount option is supplied, the existing code is OK,
because ceph_writepages_start() limits the amount written to that
amount if it is smaller than the inode's block size.

But the default "wsize" value is 0, and for that case, the pool will
be created with pagevec objects sufficient to hold just one page.
An earlier fix made sure we used a non-zero size here, but it turns
out that's not enough either. If no "wsize" mount option was
supplied, ceph_writepages_start() will choose to write a whole
stripe unit's worth of data. And that is likely to be more than
just one page.

Because an inode's layout can be set with a stripe unit that's not
explicitly limited, to be safe we need to make the pagevecs in
this memory pool be big enough for the biggest stripe unit possible.
It must be represented in a 32-bit unsigned, and it must be a
non-zero multiple of CEPH_MIN_STRIPE_UNIT (which is 65536). So
the size is 2^32 - 65536 or 0xffff0000.

Change the size of the objects allocated in the file system's
writeback pagevec pool to be big enough to hold pages covering that
maximal size.

And now that that's all laid out...

One last (deadly) note: For 4KB pages, this means about a million
page slots per page vector, or about 8MB. These pools are sized to
have 10 objects in them, which doesn't seem like a number that
was determined scientifically. Scale that back to just 2,
to reduce the memory used. (This still multiples the memory used
in the default case by a factor of half a million or so.)

This resolves:
http://tracker.ceph.com/issues/4646

Signed-off-by: Alex Elder <elder@inktank.com>