Ceph

> From:  <asomers@gmail.com>
> Date: Fri, Oct 25, 2013 at 10:35 AM
> Subject: Ceph's networking protocol is operating-system dependent
> To: Noah Watkins <noah.watkins@inktank.com>
> 
> 
> For several weeks, I've been vexed by a networking problem with Ceph.
> If I start a 1-node Linux cluster, the /usr/local/bin/rados client on
> FreeBSD can connect just fine.  However, if I start a 1-node FreeBSD
> cluster, then the /usr/bin/rados client on Linux will hit an
> assertion.
> 
> I finally figured it out.  The problem is that the encode function at
> msg/msg_types.h:164 casts a struct sockaddr_storage to a char*, which
> is subsequently sent over the wire.  However, struct sockaddr_storage
> is an operating system specific data structure.  It's different in
> FreeBSD than in Linux.  It doesn't even have the packed attribute,
> which means that it could technically differ between compilers,
> although it probably doesn't.  Somebody, probably you, tried to
> account for the difference by flipping the endianness of the ss_family
> field on Linux but not FreeBSD or OSX.  But there's another
> difference: FreeBSD has a ss_len field where Linux has the high byte
> of ss_family.  When a serialized struct sockaddr_storage gets sent
> from FreeBSD to Linux, Linux things that ss_family is 0x1002 instead
> of 0x0002, leading to the crash.
> 
> I haven't fully audited the code, but there are probably other such
> Linux/amd64 assumptions.  One such place is include/uuid.h:39, which
> encodes a uuid_t.  uuid_t is also different between FreeBSD and Linux.

I think we need to create ceph_sockaddr and ceph_uuid to define the wire format (as what Linux does) so that the translate to the platform-specific structures is clean and explicit (and easily pluggable).