Ceph

Updating wiki also.

Ceph requirements for running over alternative backend filesystems (non btrfs):

1. support for xattrs

Filesystem needs to support xattrs.

2. Large xattrs

Certain file systems provide xattrs, however, they only support small sized
xattrs. E.g., ext3/4 support only 4K xattrs.

We do hit the limit with radosgw pretty easily, though, and may
also hit it in exceptional cases where the OSD cluster is very unhealthy.

XFS does not have an xattr size limit and thus does have this problem.

btfs limits individual xattrs, however, we can work around that by chaining
multiple xattrs as it doesn't limit the total size of xattrs per file (unlike
ext3/ext4).

(note: it may be that we could modify radosgw to use less of xattrs at the cost of
reduced functionality, but that is yet to be proven)

So the order of perference:

(a) not xattr size limit (total or individual)
(b) limit on individual xattr (e.g., for btrfs it just fit into a btree
node), but no limit on total xattrs.
(c) large limit on total xattrs.

3. Keeping consistent state

There are a few options here (in order of preference):

(a) the ability to create and remove snapshots programatically over a
file system volume. Snapshot creation needs to be fast.
(b) fsync on a just-written file flushes the unerlying fs's journal
such that all previous operations are also comitted (such as
ext3's data=journal)
(c) ability to sync a single mounted file system
(d) operational sync operation (should be with any POSIX compliant
file system).