Project

General

Profile

Bug #3881

Wrong ip network to exchange data between kernel ceph and MDS

Added by Ivan Kudryavtsev over 7 years ago. Updated almost 4 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
Crash signature:

Description

I'm using ceph installation with three networks:

1st is Infiniband network for OSD exchange and replication
2nd is 10G ethernet network for remote OSD in another DC
3rd is 1G public ethernet network

My OSDs are sitting in 10.252.0.0 and 10.251.0.0
and my MONs are sitting 10.252.0.0 (3 of them) and I have MDSs on the same nodes as MONs (3 of them).

Every mon/mds node have two addresses - one in public network and one in private network (10.252.0.0) and in my clients I'm connecting to Ceph FS like:

10.252.0.3:6789,10.252.0.2:6789,10.252.0.4:6789:/shares/bitworks/castor-2/data1 on /mnt/ceph/data1 type ceph (name=admin,key=client.admin)

but actual data exchange with MDS goes across public network which is wrong way (because, for example, some client hosts are restricted to 100Mbit/sec for public interaction), so filesystem is slow and I'm unable to utilize 10G network.

Unfortunately I wasn't able to find any configuration options to bind MDS to specific address and as a result I see next listen:
tcp 0 0 0.0.0.0:6800 0.0.0.0:* LISTEN 21626/ceph-mds

The question is: how to bind MDS to specific network.

History

#1 Updated by Ivan Kudryavtsev over 7 years ago

Hm. It seems that I'm wrong about the way it works. It connects to OSDs via OSD-defined public network. It seems that ceph fs works over public network and is "External" service for OSD. I think the bug can be closed. Just haven't found it in docs.

#2 Updated by Sage Weil over 7 years ago

  • Status changed from New to Rejected

Ivan Kudryavtsev wrote:

Hm. It seems that I'm wrong about the way it works. It connects to OSDs via OSD-defined public network. It seems that ceph fs works over public network and is "External" service for OSD. I think the bug can be closed. Just haven't found it in docs.

Right. The client-side connections go out whatever interface the kernel routes through to reach the published ("public") osd ip.

We do need to add a cluster network for the MDS's at some point for mds to mds communication, but that's another issue. :)

#3 Updated by Greg Farnum almost 4 years ago

  • Component(FS) MDS added

Also available in: Atom PDF