Ceph

This also continues to happen on octopus, I just tested on 15.2.0.

I have attached the build instructions I used to compile on armv7 to #39039. I can also provide the alpine package or a docker image I use for rook.

I tested this on a lab cluster after disabling cephx per https://docs.ceph.com/docs/octopus/rados/configuration/auth-config-ref/#disabling-cephx

My armv7 instance was successfully able to get a response from the test cluser using

ceph -s

Re-enabling cephx on the same lab cluster caused the hang to reappear, so I guess it narrows the issue to cephx.

Sorry to avoid confusion I will explain my setup.

I am running rook (1.3.1) ceph (15.2.0) on kubernetes using multiple architectures.

• aarch64 (Rock Pi 4, RPi3b)
• x86_64 (vmware vm)
• armv7l (Olimex Lime2) - these have a sata port so I plan for them to be my osds

All are running near identical versions of Alpine Linux (3.11.5), kernels vary slightly but systems are in sync

• mon-a and mon-c are on aarch64, mon-b is x86_64
• mgr is on aarch64
• Working osd is on x86_64
• Rook operator is on aarch64

ceph -s

• aarch64 returns status
• x86_64 returns status
• armv7 hangs

I setup a seperate non kubernetes instance of mon on a seperate x86_64 vm running Alpine 3.11.5 and disabled cephx as per https://docs.ceph.com/docs/octopus/rados/configuration/auth-config-ref/#disabling-cephx and modified /etc/ceph/ceph.conf on the armv7 toolbox instance to use the vm mon and armv7 executing

ceph -s

If I then enable cephx on this standalone vm armv7 will start to hang when

ceph -s

All the ceph packages for alpine and the docker images used were created by me so I am happy to troubleshoot from that side.

So it sounds like cephx doesn't work when the client's 32-bit and the server's 64-bit?

We do try and maintain things like that but at this point it wouldn't surprise me.

Just to confirm, the monitors are in quorum together and the armv7 node without cephx can connect to every monitor? Can it do simple IO to the OSD?

The 3 monitors are in a quorum together when it has been failing.

I have not disabled cephx in that environment, I ran up a test env and disabled cephx on that to test so IO to the OSD has not been tested.

It is possible to disable cephx only for a single node? If so I would be happy to try.

Okay, I got a chance to really dive into this.

I set up just a monitor on x86_64 and created a container instance of the container you provided on an aarch64 machine and was able to reproduce a hang. I then set up a centos7 container and installed the packages from https://download.ceph.com/rpm-octopus/el7/noarch/. This container operates normally. I then compared debug output for the two.

Working: https://download.ceph.com/rpm-octopus/el7

2020-06-16T01:00:27.005-0400 7f4704a10700 20 --2- [v2:10.19.176.26:3300/0,v1:10.19.176.26:6789/0] >>  conn(0x5608c251a000 0x5608c251c000 unknown :-1 s=HELLO_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main r=0
2020-06-16T01:00:27.005-0400 7f4704a10700 30 --2- [v2:10.19.176.26:3300/0,v1:10.19.176.26:6789/0] >>  conn(0x5608c251a000 0x5608c251c000 unknown :-1 s=HELLO_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main preamble
00000000  01 01 24 00 00 00 08 00  00 00 00 00 00 00 00 00  |..$.............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 3f bd 6b 06  |............?.k.|
00000020

2020-06-16T01:00:27.005-0400 7f4704a10700 10 --2- [v2:10.19.176.26:3300/0,v1:10.19.176.26:6789/0] >>  conn(0x5608c251a000 0x5608c251c000 unknown :-1 s=HELLO_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main got new segment: len=36 align=8

Not working: Provided container image

2020-06-16T00:19:16.745-0400 7fa98cc21700 30 --2- [v2:10.19.176.26:3300/0,v1:10.19.176.26:6789/0] >>  conn(0x558d6c86c000 0x558d6c872000 unknown :-1 s=HELLO_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main preamble
00000000  01 01 24 00 00 00 04 00  00 00 00 00 00 00 00 00  |..$.............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 5c e9 a7 98  |............\...|
00000020

2020-06-16T00:19:16.745-0400 7fa98cc21700 10 --2- [v2:10.19.176.26:3300/0,v1:10.19.176.26:6789/0] >>  conn(0x558d6c86c000 0x558d6c872000 unknown :-1 s=HELLO_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main got new segment: len=36 align=4

Notice the differences in the value for 'align'.

Alignment appears to be a 'ceph_le16' which ultimately depends on the value of 'CEPH_BIG_ENDIAN' so maybe this is set in the failure case (since these machines are all little endian)?

src/include/byteorder.h:

// mswab == maybe swab (if not LE)
#ifdef CEPH_BIG_ENDIAN
template<typename T>
inline T mswab(T val) {
  return swab(val);
}
#else
template<typename T>
inline T mswab(T val) {
  return val;
}
#endif

template<typename T>
struct ceph_le {                                                                                                                                                                                                                             
private:
  T v;
public:
  ceph_le<T>& operator=(T nv) {
    v = mswab(nv);
    return *this;
  }
  operator T() const { return mswab(v); }
  friend inline bool operator==(ceph_le a, ceph_le b) {
    return a.v == b.v;
  }
} __attribute__ ((packed));

using ceph_le64 = ceph_le<__u64>;
using ceph_le32 = ceph_le<__u32>;
using ceph_le16 = ceph_le<__u16>;

Tony, can you check the cmake logs for the build you did for that container image to see how CEPH_BIG_ENDIAN and CEPH_LITTLE_ENDIAN are being used?

Brad Hubbard wrote:

Okay, I got a chance to really dive into this.

I set up just a monitor on x86_64 and created a container instance of the container you provided on an aarch64 machine and was able to reproduce a hang. I then set up a centos7 container and installed the packages from https://download.ceph.com/rpm-octopus/el7/noarch/. This container operates normally. I then compared debug output for the two.

[...]

Tony, can you check the cmake logs for the build you did for that container image to see how CEPH_BIG_ENDIAN and CEPH_LITTLE_ENDIAN are being used?

Hey Brad, I dont have the build logs for that particular image any more as it was built on cloud.drone.io. I just kicked off another build for 15.2.3 and get the following:

-- Check if the system is big endian
-- Searching 16 bit integer
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Searching 16 bit integer - Using unsigned short
-- Check if the system is big endian - little endian

I get the same on my x86_64 and aarch64 builds. Hit me up on IRC and we can chat about any more specific tests you want to run.

So what I said previously was wrong because all of these are little endian systems but the client that fails is 32 bit which probably makes the alignment value of '4' correct. Looking into this further.

Issue was likely use of 32 bit words on 32 bit systems to store 64 bit config option values. Config pushed by 64 bit mon to 32 bit osd was corrupted on arrival as 64 bit values were just trimmed. This was fixed in https://github.com/ceph/ceph/pull/38837 and I can run mixed 32/64 bit cluster with this patch.