Project

General

Profile

Actions

Bug #19792

closed

monitor hostname and dns cache

Added by Bertrand Gouny almost 7 years ago. Updated almost 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
libceph
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

Hello,

libceph seems to cache dns responses and the cache seems to never expire or the TTL is really big.
I'm not sure this comes from libceph or how the kernel dns resolver module handles requests.

Here the problem:

My monitors are behind a round robin dns:

dig ceph-mon.cluster.local
ceph-mon.cluster.local. 30 IN A 10.244.10.11
ceph-mon.cluster.local. 30 IN A 10.244.20.6
ceph-mon.cluster.local. 30 IN A 10.244.30.13

but libceph still use an old ip previously returned by the dns:
libceph: resolve 'ceph-mon.cluster.local' (ret=0): 10.244.10.6:0

the monitor with the ip 10.244.10.6 is now down so i can't mount any volume :(

is there a way to clear the dns cache ? Does it have something to do with libceph ?

Thanks.

Actions #1

Updated by Ilya Dryomov almost 7 years ago

CONFIG_CEPH_LIB_USE_DNS_RESOLVER is disabled by default. Did you build your own libceph module?

I think the userspace component (request-key -> key.dns_resolver in your case?) is responsible for setting the TTL. Try enabling the in-kernel dns resolver logging with "echo 1 >/sys/module/dns_resolver/parameters/debug".

There should be a way to inspect and invalidate the in-kernel cache with keyctl.

Actions #2

Updated by Bertrand Gouny almost 7 years ago

i use CoreOS Container Linux and indeed ceph module is compiled with dns resolver module.

the userspace component is key.dns_resolver as you guessed, and it seems to no support ttl for a or aaaa :(
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/keyutils.git/tree/key.dns_resolver.c

if this can be useful to someone else here the command i use to clear the kernel dns resolver cache:
sudo keyctl clear $((16#$(sudo cat /proc/keys | grep .dns_resolver | awk '{print $1;}')))

Thanks

Actions #3

Updated by Ilya Dryomov almost 7 years ago

Looks right... If you have three mons, you can specify three ips on the mount command line and avoid the in-kernel dns resolver.

Actions #4

Updated by Ilya Dryomov almost 7 years ago

  • Category set to libceph
  • Status changed from New to Closed
  • Assignee set to Ilya Dryomov
Actions

Also available in: Atom PDF