Project

General

Profile

Backport #37972

Updated by Nathan Cutler about 5 years ago

https://github.com/ceph/ceph/pull/26042 We are using a ceph cluster in a mixed FreeBSD/Linux environment. The ceph cluster is based on FreeBSD. Linux clients cannot connect to the cluster. For example ceph -s results in:

NetHandler create_socket couldn't create socket (97) Address family not supported by protocol

Printing the monmap on a Linux client shows:

monmaptool: monmap file monmap_freebsd
epoch 0
fsid 7e15ef20-73c2-11e8-98ed-95470d800f64
last_changed 2018-06-19 18:21:16.099386
created 2018-06-19 18:21:16.099386
0: :/0 mon.freebsd03
1: :/0 mon.freebsd01
2: :/0 mon.freebsd02

Printing the same map on FreeBSD:

epoch 0
fsid 7e15ef20-73c2-11e8-98ed-95470d800f64
last_changed 2018-06-19 18:21:16.099386
created 2018-06-19 18:21:16.099386
0: 10.135.28.158:6789/0 mon.freebsd03
1: 10.135.69.231:6789/0 mon.freebsd01
2: 10.135.93.250:6789/0 mon.freebsd02

I used gdb on monmaptool on Linux to show me the mon_info map. It shows 512 for the sa_family.

$1 = std::map with 3 elements = {["freebsd01"] = {name = "freebsd01", public_addr = {static TYPE_DEFAULT = entity_addr_t::TYPE_LEGACY, type = 1, nonce = 0, u = {sa = {sa_family = 512,
sa_data = "\032\205\n\207E\347\000\000\000\000\000\000\000"}, sin = {sin_family = 512, sin_port = 34074, sin_addr = {s_addr = 3880093450}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {
sin6_family = 512, sin6_port = 34074, sin6_flowinfo = 3880093450, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
sin6_scope_id = 0}}}, priority = 0}, ["freebsd02"] = {name = "freebsd02", public_addr = {static TYPE_DEFAULT = entity_addr_t::TYPE_LEGACY, type = 1, nonce = 0, u = {sa = {sa_family = 512,
sa_data = "\032\205\n\207]\372\000\000\000\000\000\000\000"}, sin = {sin_family = 512, sin_port = 34074, sin_addr = {s_addr = 4200433418}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {
sin6_family = 512, sin6_port = 34074, sin6_flowinfo = 4200433418, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
sin6_scope_id = 0}}}, priority = 0}, ["freebsd03"] = {name = "freebsd03", public_addr = {static TYPE_DEFAULT = entity_addr_t::TYPE_LEGACY, type = 1, nonce = 0, u = {sa = {sa_family = 512,
sa_data = "\032\205\n\207\034\236\000\000\000\000\000\000\000"}, sin = {sin_family = 512, sin_port = 34074, sin_addr = {s_addr = 2652669706}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {
sin6_family = 512, sin6_port = 34074, sin6_flowinfo = 2652669706, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
sin6_scope_id = 0}}}, priority = 0}}

On a working monmap sa_family is equal 2.

I setup a ceph test cluster with Linux and FreeBSD as clients and the problem is symmetrical showing

NetHandler create_socket couldn't create socket (47) Address family not supported by protocol family

Could this be an encoding/decoding problem?

Versions used:

FreeBSD 11.1: ceph version 12.2.4 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
Debian 9.4: ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
Ubuntu 16.04: ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)

Back