Ceph » CephFS

ceph version: jewel(10.2.2)
mds mode: active/hot-standby

bug description:
stack:

#0  Client::get_caps (this=0xba32000, in=0xbb31200, need=2048, want=1024, phave=0x7f7bd14961fc, endoff=-1)
    at /var/ws/orig-mh38-ceph/src/client/Client.cc:3399
#1  0x0000000000ffc58a in Client::_read (this=0xba32000, f=0xb9dc700, offset=0, size=131072, bl=0x7f7bd1496420)
    at /var/ws/orig-mh38-ceph/src/client/Client.cc:8294
#2  0x000000000101b8ce in Client::ll_read (this=0xba32000, fh=0xb9dc700, off=0, len=131072, bl=0x7f7bd1496420)
    at /var/ws/orig-mh38-ceph/src/client/Client.cc:12338
#3  0x0000000000facc0c in fuse_ll_read (req=0xb9a4c80, ino=1099511635413, size=131072, off=0, fi=0x7f7bd14964b0)
    at /var/ws/orig-mh38-ceph/src/client/fuse_ll.cc:491
#4  0x00007f7be1fde13e in do_read (req=<optimized out>, nodeid=<optimized out>, inarg=<optimized out>) at fuse_lowlevel.c:1232
#5  0x00007f7be1fdebdb in fuse_ll_process_buf (data=0xbaca000, buf=0x7f7bd1496680, ch=<optimized out>) at fuse_lowlevel.c:2441
#6  0x00007f7be1fdb471 in fuse_do_work (data=0xba9af60) at fuse_loop_mt.c:117
#7  0x00007f7be072fdc5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f7bde3cf73d in clone () from /lib64/libc.so.6

(gdb) p in->caps
$2 = std::map with 1 elements = {[0] = 0xbfa5680}
(gdb) p (Capability*)0xbfa5680
$3 = (Capability *) 0xbfa5680

p *(Cap*)0xbfa5680
$7 = {session = 0xc052000, inode = 0xbfe8d00, cap_item = {_item = 0xbfa5680, _prev = 0x0, _next = 0xbfa51b0, _list = 0xc0520f0},
  cap_id = 1582, issued = 3413, implemented = 3413, wanted = 3329, seq = 0, issue_seq = 0, mseq = 0, *gen = 0*}

(gdb) p *((Cap*)0xbfa5680)->session
$9 = {mds_num = 0, con = {px = 0xbf79800}, seq = 0, *cap_gen = 1*, cap_ttl = {tv = {tv_sec = 1533030280, tv_nsec = 129590720}},
  last_cap_renew_request = {tv = {tv_sec = 1533030220, tv_nsec = 129590720}}, cap_renew_seq = 46, num_caps = 17, inst = {name = {
      _type = 2 '\002', _num = 0, static TYPE_MON = 1, static TYPE_MDS = 2, static TYPE_OSD = 4, static TYPE_CLIENT = 8,
      static NEW = -1}, addr = {type = 0, nonce = 30130, {addr = {ss_family = 2, __ss_align = 0,
          __ss_padding = '\000' <repeats 111 times>}, addr4 = {sin_family = 2, sin_port = 36890, sin_addr = {s_addr = 3389827264},
          sin_zero = "\000\000\000\000\000\000\000"}, addr6 = {sin6_family = 2, sin6_port = 36890, sin6_flowinfo = 3389827264,
          sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
                0}}}, sin6_scope_id = 0}}}}, state = MetaSession::STATE_OPEN, readonly = false, waiting_for_open = empty std::list,
  caps = {_front = 0xbfa5690, _back = 0xbfa51b0, _size = 2}, flushing_caps = {_front = 0x0, _back = 0x0, _size = 0}, flushing_capsnaps = {
    _front = 0x0, _back = 0x0, _size = 0}, requests = {_front = 0x0, _back = 0x0, _size = 0}, unsafe_requests = {_front = 0x0,
    _back = 0x0, _size = 0}, flushing_caps_tids = std::set with 0 elements, kicked_flush_tids = std::set with 0 elements,
  s_cap_iterator = 0x0, release = 0x0}

As the above stack info shows, when cap->gen behind session->cap_gen we can't read or write data continually because our caps is invalid.
What's even worse the cap->gen will not be updated unless we getattr or reopen it,so the application may hung all the time.How could this happen?

Time series is as fllows:
1. we open one file in client
2. mds found the client become stale will send CEPH_SESSION_STALE to client
3. client session->cap_gen++ and will send CEPH_SESSION_REQUEST_RENEWCAPS to mds.But the active/hot-stanby switch occurred before this message
could be processed.
4. the client start reconnect to mds with its cap(seq,issue_seq,wanted,issued...)
5. when client reconnect success and the hot-standby mds become active we found we couldn't read data any more because the inode's cap_gen is invalid.