Ceph » CephFS

     0> 2017-11-10 17:04:37.887573 7f7bc2d64700 -1 /build/ceph-12.2.1/src/include/xlist.h: In function 'xlist<T>::iterator& xlist<T>::iterator::operator++() [with T = Cap*]' thread 7f7bc2d64700 time 2017-11-10 17:04:37.674883
/build/ceph-12.2.1/src/include/xlist.h: 166: FAILED assert(cur->_list)

 ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7f7bcc73aa4e]
 2: (Client::trim_caps(MetaSession*, int)+0x7cd) [0x7f7bcc6acadd]
 3: (Client::handle_client_session(MClientSession*)+0x3a1) [0x7f7bcc6ace91]
 4: (Client::ms_dispatch(Message*)+0x54f) [0x7f7bcc6d9faf]
 5: (DispatchQueue::entry()+0x78b) [0x7f7bcca2cf2b]
 6: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7bcc7cb2dd]
 7: (()+0x8184) [0x7f7bcb1b8184]
 8: (clone()+0x6d) [0x7f7bca085bed]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

There are two ways I can see this happening:

1) We are trimming the auth_cap for the inode and there are other caps for the inode:

https://github.com/ceph/ceph/blob/f870729aa15ad1156fcd86b66ed601e659d3cf7a/src/client/Client.cc#L4069

The iterator points to one of those other caps which are removed during trim_dentry here:

https://github.com/ceph/ceph/blob/f870729aa15ad1156fcd86b66ed601e659d3cf7a/src/client/Client.cc#L4093

2) (Most likely) We are trimming the cap for the last dentry in a directory and the trim_caps iterator is currently pointing to the directory inode's cap. Client::trim_dentry will drop the dir inode/cap when calling Client::unlink:

https://github.com/ceph/ceph/blob/f870729aa15ad1156fcd86b66ed601e659d3cf7a/src/client/Client.cc#L672

This is probably caused by this refactoring patch:

commit d6a6b03a0dd239a889332fd641cc6a3e99b961c1 HEAD (upstream/pull/12161/head)
Author: John Spray <john.spray@redhat.com>
Date:   Tue Nov 29 17:13:29 2016 +0000

    client: simplify remove_cap interface

    All this complexity only existed to handle the
    case where trim wanted to iterate over an xlist
    without getting disturbed by a deletion.  We
    can simply increment the iterator before
    maybe-deleting the Cap.

    Signed-off-by: John Spray <john.spray@redhat.com>

BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1514298