cephx has embedded byte-order dependency
Calculation of the original session key is byte-order-dependent; cephx_calc_client_server_challenge gets a message digest of the random server and client challenges, and then steps through that digest 64 bits at a time and XORs each chunk to come up with a 64-bit key. But it does that by casting to uint64_t, which means the answer is different (byteswapped) if client and server have different byte orders.
Adding an mswab64() solves the problem:
--- a/src/auth/cephx/CephxProtocol.cc +++ b/src/auth/cephx/CephxProtocol.cc @@ -45,7 +45,7 @@ void cephx_calc_client_server_challenge(CephContext *cct, CryptoKey& secret, uin uint64_t k = 0; const uint64_t *p = (const uint64_t *)enc.c_str(); for (int pos = 0; pos + sizeof(k) <= enc.length(); pos+=sizeof(k), p++) - k ^= *p; + k ^= mswab64(*p); *key = k; }