Bug #19792
closed
monitor hostname and dns cache
Added by Bertrand Gouny about 7 years ago.
Updated about 7 years ago.
Description
Hello,
libceph seems to cache dns responses and the cache seems to never expire or the TTL is really big.
I'm not sure this comes from libceph or how the kernel dns resolver module handles requests.
Here the problem:
My monitors are behind a round robin dns:
dig ceph-mon.cluster.local
ceph-mon.cluster.local. 30 IN A 10.244.10.11
ceph-mon.cluster.local. 30 IN A 10.244.20.6
ceph-mon.cluster.local. 30 IN A 10.244.30.13
but libceph still use an old ip previously returned by the dns:
libceph: resolve 'ceph-mon.cluster.local' (ret=0): 10.244.10.6:0
the monitor with the ip 10.244.10.6 is now down so i can't mount any volume :(
is there a way to clear the dns cache ? Does it have something to do with libceph ?
Thanks.
CONFIG_CEPH_LIB_USE_DNS_RESOLVER is disabled by default. Did you build your own libceph module?
I think the userspace component (request-key -> key.dns_resolver in your case?) is responsible for setting the TTL. Try enabling the in-kernel dns resolver logging with "echo 1 >/sys/module/dns_resolver/parameters/debug".
There should be a way to inspect and invalidate the in-kernel cache with keyctl.
i use CoreOS Container Linux and indeed ceph module is compiled with dns resolver module.
the userspace component is key.dns_resolver as you guessed, and it seems to no support ttl for a or aaaa :(
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/keyutils.git/tree/key.dns_resolver.c
if this can be useful to someone else here the command i use to clear the kernel dns resolver cache:
sudo keyctl clear $((16#$(sudo cat /proc/keys | grep .dns_resolver | awk '{print $1;}')))
Thanks
Looks right... If you have three mons, you can specify three ips on the mount command line and avoid the in-kernel dns resolver.
- Category set to libceph
- Status changed from New to Closed
- Assignee set to Ilya Dryomov
Also available in: Atom
PDF