Project

General

Profile

Bug #36189

ceph-fuse client can't read or write due to backward cap_gen

Added by Ivan Guan 3 months ago. Updated about 7 hours ago.

Status:
Need More Info
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
1.00 h
Source:
Community (user)
Tags:
Backport:
mimic,luminous
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client
Labels (FS):
Pull request ID:

Description

ceph version: jewel(10.2.2)
mds mode: active/hot-standby

bug description:
stack:

#0  Client::get_caps (this=0xba32000, in=0xbb31200, need=2048, want=1024, phave=0x7f7bd14961fc, endoff=-1)
    at /var/ws/orig-mh38-ceph/src/client/Client.cc:3399
#1  0x0000000000ffc58a in Client::_read (this=0xba32000, f=0xb9dc700, offset=0, size=131072, bl=0x7f7bd1496420)
    at /var/ws/orig-mh38-ceph/src/client/Client.cc:8294
#2  0x000000000101b8ce in Client::ll_read (this=0xba32000, fh=0xb9dc700, off=0, len=131072, bl=0x7f7bd1496420)
    at /var/ws/orig-mh38-ceph/src/client/Client.cc:12338
#3  0x0000000000facc0c in fuse_ll_read (req=0xb9a4c80, ino=1099511635413, size=131072, off=0, fi=0x7f7bd14964b0)
    at /var/ws/orig-mh38-ceph/src/client/fuse_ll.cc:491
#4  0x00007f7be1fde13e in do_read (req=<optimized out>, nodeid=<optimized out>, inarg=<optimized out>) at fuse_lowlevel.c:1232
#5  0x00007f7be1fdebdb in fuse_ll_process_buf (data=0xbaca000, buf=0x7f7bd1496680, ch=<optimized out>) at fuse_lowlevel.c:2441
#6  0x00007f7be1fdb471 in fuse_do_work (data=0xba9af60) at fuse_loop_mt.c:117
#7  0x00007f7be072fdc5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f7bde3cf73d in clone () from /lib64/libc.so.6

(gdb) p in->caps
$2 = std::map with 1 elements = {[0] = 0xbfa5680}
(gdb) p (Capability*)0xbfa5680
$3 = (Capability *) 0xbfa5680

p *(Cap*)0xbfa5680
$7 = {session = 0xc052000, inode = 0xbfe8d00, cap_item = {_item = 0xbfa5680, _prev = 0x0, _next = 0xbfa51b0, _list = 0xc0520f0},
  cap_id = 1582, issued = 3413, implemented = 3413, wanted = 3329, seq = 0, issue_seq = 0, mseq = 0, *gen = 0*}

(gdb) p *((Cap*)0xbfa5680)->session
$9 = {mds_num = 0, con = {px = 0xbf79800}, seq = 0, *cap_gen = 1*, cap_ttl = {tv = {tv_sec = 1533030280, tv_nsec = 129590720}},
  last_cap_renew_request = {tv = {tv_sec = 1533030220, tv_nsec = 129590720}}, cap_renew_seq = 46, num_caps = 17, inst = {name = {
      _type = 2 '\002', _num = 0, static TYPE_MON = 1, static TYPE_MDS = 2, static TYPE_OSD = 4, static TYPE_CLIENT = 8,
      static NEW = -1}, addr = {type = 0, nonce = 30130, {addr = {ss_family = 2, __ss_align = 0,
          __ss_padding = '\000' <repeats 111 times>}, addr4 = {sin_family = 2, sin_port = 36890, sin_addr = {s_addr = 3389827264},
          sin_zero = "\000\000\000\000\000\000\000"}, addr6 = {sin6_family = 2, sin6_port = 36890, sin6_flowinfo = 3389827264,
          sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
                0}}}, sin6_scope_id = 0}}}}, state = MetaSession::STATE_OPEN, readonly = false, waiting_for_open = empty std::list,
  caps = {_front = 0xbfa5690, _back = 0xbfa51b0, _size = 2}, flushing_caps = {_front = 0x0, _back = 0x0, _size = 0}, flushing_capsnaps = {
    _front = 0x0, _back = 0x0, _size = 0}, requests = {_front = 0x0, _back = 0x0, _size = 0}, unsafe_requests = {_front = 0x0,
    _back = 0x0, _size = 0}, flushing_caps_tids = std::set with 0 elements, kicked_flush_tids = std::set with 0 elements,
  s_cap_iterator = 0x0, release = 0x0}

As the above stack info shows, when cap->gen behind session->cap_gen we can't read or write data continually because our caps is invalid.
What's even worse the cap->gen will not be updated unless we getattr or reopen it,so the application may hung all the time.How could this happen?

Time series is as fllows:
1. we open one file in client
2. mds found the client become stale will send CEPH_SESSION_STALE to client
3. client session->cap_gen++ and will send CEPH_SESSION_REQUEST_RENEWCAPS to mds.But the active/hot-stanby switch occurred before this message
could be processed.
4. the client start reconnect to mds with its cap(seq,issue_seq,wanted,issued...)
5. when client reconnect success and the hot-standby mds become active we found we couldn't read data any more because the inode's cap_gen is invalid.


Related issues

Copied to fs - Backport #36462: luminous: ceph-fuse client can't read or write due to backward cap_gen Need More Info
Copied to fs - Backport #36463: mimic: ceph-fuse client can't read or write due to backward cap_gen Need More Info

History

#1 Updated by Greg Farnum 2 months ago

  • Project changed from Ceph to fs
  • Component(FS) ceph-fuse added

#2 Updated by Patrick Donnelly 2 months ago

  • Description updated (diff)
  • Due date deleted (09/26/2018)
  • Start date deleted (09/26/2018)
  • Source set to Community (user)
  • Backport set to mimic,luminous
  • Affected Versions deleted (v14.0.0)
  • Component(FS) Client added
  • Component(FS) deleted (ceph-fuse)

#3 Updated by Patrick Donnelly 2 months ago

  • Status changed from New to Need Review
  • Assignee set to Ivan Guan

#4 Updated by Patrick Donnelly about 2 months ago

  • Status changed from Need Review to Pending Backport

#5 Updated by Nathan Cutler about 2 months ago

  • Copied to Backport #36462: luminous: ceph-fuse client can't read or write due to backward cap_gen added

#6 Updated by Nathan Cutler about 2 months ago

  • Copied to Backport #36463: mimic: ceph-fuse client can't read or write due to backward cap_gen added

#7 Updated by Nathan Cutler 17 days ago

  • Status changed from Pending Backport to Need More Info

Zheng writes: "If cap is invalid during reconnect, mds should consider issued caps is empty (just CEPH_CAP_PIN)"

and, in the mimic backport: "don't merge this. The original commit may have issue"

#8 Updated by Patrick Donnelly about 7 hours ago

  • Assignee changed from Ivan Guan to Zheng Yan

Also available in: Atom PDF