Project

General

Profile

Bug #11482

kclient: intermittent log warnings "client.XXXX isn't responding to mclientcaps(revoke)"

Added by Greg Farnum almost 4 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Correctness/Safety
Target version:
-
Start date:
04/27/2015
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
hammer,infernalis
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client, Common/Protocol, MDS
Labels (FS):
Pull request ID:

Description

http://pulpito-rdu.front.sepia.ceph.com/teuthology-2015-04-25_16:08:02-kcephfs-hammer-testing-basic-typica/5139/ for instance, but we've seen it elsewhere with the kernel client. I initially guessed the test or something might just be racy when I saw it at http://pulpito.ceph.com/teuthology-2015-04-12_23:08:01-kcephfs-hammer-testing-basic-multi/845893/ but we're still seeing it and things are all cleaned up now.


Related issues

Copied to fs - Backport #13931: infernalis: kclient: intermittent log warnings "client.XXXX isn't responding to mclientcaps(revoke)" Resolved
Copied to fs - Backport #13932: hammer: kclient: intermittent log warnings "client.XXXX isn't responding to mclientcaps(revoke)" Rejected

Associated revisions

Revision 86788c4e (diff)
Added by Yan, Zheng almost 4 years ago

mds: remove caps from revoking list when caps are voluntarily released

Fixes: #11482
Signed-off-by: Yan, Zheng <>

Revision 3a58e30d (diff)
Added by Yan, Zheng almost 4 years ago

mds: remove caps from revoking list when caps are voluntarily released

Fixes: #11482
Signed-off-by: Yan, Zheng <>
(cherry picked from commit 86788c4ea4155f0388b7ebaf475a3d3c37d39331)

Revision ce9a596d (diff)
Added by Yan, Zheng over 3 years ago

mds: consider client's flushing caps when choosing lock states

Client may flush and drop caps at the same time. If client need to
send cap reconnect before the caps get flushed. The issued caps in
the cap reconnect does not include the flushing caps. When choosing
lock states, MDS only consider the issued caps in cap reconnect, it
may choose wrong states.

Fixes: #11482
Signed-off-by: Yan, Zheng <>

Revision 6c68971d (diff)
Added by Yan, Zheng over 3 years ago

mds: consider client's flushing caps when choosing lock states

Client may flush and drop caps at the same time. If client need to
send cap reconnect before the caps get flushed. The issued caps in
the cap reconnect does not include the flushing caps. When choosing
lock states, MDS only consider the issued caps in cap reconnect, it
may choose wrong states.

Fixes: #11482
Signed-off-by: Yan, Zheng <>
(cherry picked from commit ce9a596dcaf95dd4af0a3a9e28871462a6bcb930)

History

#1 Updated by Zheng Yan almost 4 years ago

  • Category changed from 53 to 47
  • Status changed from New to Need Review

#2 Updated by John Spray almost 4 years ago

  • Status changed from Need Review to Pending Backport

Fix looks good, let's backport to hammer too

#4 Updated by Zheng Yan almost 4 years ago

  • Status changed from Pending Backport to Resolved
  • Regression set to No

#5 Updated by John Spray over 3 years ago

  • Status changed from Resolved to Verified

Reopening this because we're still seeing it on kcephfs, unknown whether underlying cause is in the same area:
http://pulpito.ceph.com/teuthology-2015-08-07_23:08:02-kcephfs-next-testing-basic-multi/1005956/

#6 Updated by John Spray over 3 years ago

We need to turn on MDS logs to get a more useful bit of information

#7 Updated by Greg Farnum over 3 years ago

http://pulpito.ceph.com/teuthology-2015-08-17_23:08:05-kcephfs-master-testing-basic-multi/1020518/

Doesn't have the MDS debugging unfortunately, but it happened again.

#8 Updated by Greg Farnum over 3 years ago

For some reason the override we provided isn't being added to the configs, I created #12869 for that. :/

#9 Updated by Greg Farnum over 3 years ago

Okay, added debug mds 20 to the other kcephfs subsuites so we should be good next time we see it for jobs scheduled after now.

commit 8bac7213cb0f387f335fcf49ba5581e972495db9
Author: Greg Farnum <gfarnum@redhat.com>
Date:   Thu Sep 3 04:04:47 2015 -0700

    kcephfs: include debug mds 20 everywhere

    Signed-off-by: Greg Farnum <gfarnum@redhat.com>

diff --git a/suites/kcephfs/mixed-clients/conf.yaml b/suites/kcephfs/mixed-clients/conf.yaml
index 30da870..75b8558 100644
--- a/suites/kcephfs/mixed-clients/conf.yaml
+++ b/suites/kcephfs/mixed-clients/conf.yaml
@@ -3,3 +3,5 @@ overrides:
     conf:
       global:
         ms die on skipped message: false
+      mds:
+        debug mds: 20
\ No newline at end of file
diff --git a/suites/kcephfs/thrash/conf.yaml b/suites/kcephfs/thrash/conf.yaml
index 30da870..75b8558 100644
--- a/suites/kcephfs/thrash/conf.yaml
+++ b/suites/kcephfs/thrash/conf.yaml
@@ -3,3 +3,5 @@ overrides:
     conf:
       global:
         ms die on skipped message: false
+      mds:
+        debug mds: 20
\ No newline at end of file

#10 Updated by Greg Farnum over 3 years ago

  • Assignee set to Zheng Yan

Still happening: http://pulpito.ceph.com/teuthology-2015-10-23_23:08:01-kcephfs-master-testing-basic-multi/1122923/

Zheng, can you take a look now that we have debug logging?

#11 Updated by Zheng Yan over 3 years ago

seems like caused by caps reconnect issues.

#12 Updated by Zheng Yan over 3 years ago

  • Status changed from Verified to Need Review

#13 Updated by John Spray over 3 years ago

  • Status changed from Need Review to Pending Backport

Merged to master

commit 304965f17ead56f1e687a85c2f09bade46397656
Merge: 1ada4c3 e242d84
Author: John Spray <jcspray@gmail.com>
Date:   Mon Nov 30 17:53:51 2015 +0000

    Merge pull request #6432 from ukernel/wip-11482

    mds: fix client capabilities during reconnect (client.XXXX isn't responding to mclientcaps(revoke))

    Reviewed-by: John Spray <john.spray@redhat.com>

#14 Updated by Nathan Cutler over 3 years ago

Hi John: Backport to which major versions - hammer, infernalis?

#15 Updated by Loic Dachary over 3 years ago

  • Backport set to hammer,infernalis

#16 Updated by Loic Dachary over 3 years ago

  • Copied to Backport #13931: infernalis: kclient: intermittent log warnings "client.XXXX isn't responding to mclientcaps(revoke)" added

#17 Updated by Loic Dachary over 3 years ago

  • Copied to Backport #13932: hammer: kclient: intermittent log warnings "client.XXXX isn't responding to mclientcaps(revoke)" added

#18 Updated by Greg Farnum over 2 years ago

  • Category changed from 47 to Correctness/Safety
  • Component(FS) Client, Common/Protocol, MDS added

#19 Updated by Loic Dachary over 2 years ago

  • Backport changed from hammer,infernalis to hammer

infernalis is EOL

#20 Updated by Loic Dachary over 2 years ago

  • Backport changed from hammer to hammer,infernalis

#21 Updated by Nathan Cutler about 2 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF