Bug #3881
closedWrong ip network to exchange data between kernel ceph and MDS
0%
Description
I'm using ceph installation with three networks:
1st is Infiniband network for OSD exchange and replication
2nd is 10G ethernet network for remote OSD in another DC
3rd is 1G public ethernet network
My OSDs are sitting in 10.252.0.0 and 10.251.0.0
and my MONs are sitting 10.252.0.0 (3 of them) and I have MDSs on the same nodes as MONs (3 of them).
Every mon/mds node have two addresses - one in public network and one in private network (10.252.0.0) and in my clients I'm connecting to Ceph FS like:
10.252.0.3:6789,10.252.0.2:6789,10.252.0.4:6789:/shares/bitworks/castor-2/data1 on /mnt/ceph/data1 type ceph (name=admin,key=client.admin)
but actual data exchange with MDS goes across public network which is wrong way (because, for example, some client hosts are restricted to 100Mbit/sec for public interaction), so filesystem is slow and I'm unable to utilize 10G network.
Unfortunately I wasn't able to find any configuration options to bind MDS to specific address and as a result I see next listen:
tcp 0 0 0.0.0.0:6800 0.0.0.0:* LISTEN 21626/ceph-mds
The question is: how to bind MDS to specific network.
Updated by Ivan Kudryavtsev about 11 years ago
Hm. It seems that I'm wrong about the way it works. It connects to OSDs via OSD-defined public network. It seems that ceph fs works over public network and is "External" service for OSD. I think the bug can be closed. Just haven't found it in docs.
Updated by Sage Weil about 11 years ago
- Status changed from New to Rejected
Ivan Kudryavtsev wrote:
Hm. It seems that I'm wrong about the way it works. It connects to OSDs via OSD-defined public network. It seems that ceph fs works over public network and is "External" service for OSD. I think the bug can be closed. Just haven't found it in docs.
Right. The client-side connections go out whatever interface the kernel routes through to reach the published ("public") osd ip.
We do need to add a cluster network for the MDS's at some point for mds to mds communication, but that's another issue. :)