Ceph » Linux kernel client

fs/ceph/mds_client.c

static void dispatch(struct ceph_connection *con, struct ceph_msg *msg)
{
    struct ceph_mds_session *s = con->private;
    struct ceph_mds_client *mdsc = s->s_mdsc;
    int type = le16_to_cpu(msg->hdr.type);

    mutex_lock(&mdsc->mutex);
    if (__verify_registered_session(mdsc, s) < 0) {
        mutex_unlock(&mdsc->mutex);
        goto out;      <== should return? if mds session does not exist. e.g. https://github.com/gmayyyha/ceph-client/blob/1c789249578895bb14ab62b4327306439b754857/net/ceph/mon_client.c#L1208
    }
    mutex_unlock(&mdsc->mutex);

    switch (type) {
    ...
    default:
        pr_err("received unknown message type %d %s\n", type,
               ceph_msg_type_name(type));
    }
out:
    ceph_msg_put(msg);
}

Looks like slabcache corruption at first glance, or maybe a bogus pointer got passed to kfree. 4.19.15 is pretty old at this point, do you see this on newer v4.19 kernels too?

Jeff Layton wrote:

Looks like slabcache corruption at first glance, or maybe a bogus pointer got passed to kfree. 4.19.15 is pretty old at this point, do you see this on newer v4.19 kernels too?

The newer v4.19.79 kernel, same as the old v4.19.15.

(cc'ing Ilya since he knows this code better)

I don't know the mon code as well, but the monitor dispatch function looks leaky to me too. process_message takes in_msg from the con, and in the case where monc->private is NULL, it just drops that msg onto the floor without releasing the reference. Maybe there is something that mitigates that but I'm not seeing it right offhand.

I think you meant con->private. That check is bogus, con->private is set when ceph_connection is initialized and is never cleared. I'll remove it.

Ilya Dryomov wrote:

I think you meant con->private. That check is bogus, con->private is set when ceph_connection is initialized and is never cleared. I'll remove it.

Yes, I meant con->private. Great, that was my take too after going over the code.

So to summarize, I think the patch proposed to fix this is not right. We need to put the reference to the message so it doesn't leak. If there's a bogus pointer in the message that's causing this, then we probably need to deal with it another way.

ceph_msg_release
  -> ceph_msg_data_destroy <= foreach
    -> ceph_pagelist_release   <= data->type == CEPH_MSG_DATA_PAGELIST
      -> kfree <== crash here ?
  -> kvfree  <= !m->pool
    -> kfree  <== crash here ? 

I only have vmcore-dmesg.txt, so I'm not pretty sure which kfree caused the crash.

Do you have a way to reproduce this at will?

If you have the debug symbols, you may be able to open the module with gdb and determine where it was:

$ gdb path/to/libceph.ko
...
gdb> list *(ceph_msg_release+0x15d)

That should show you the next instruction after the kfree returns, and you can use that to infer which one it was. That said this relies on having debug symbols available and the same sources, so ymmv.

Only wait for the production environment to reappear.

Reading symbols from /export/linux-4.19.15/net/ceph/libceph.ko...done.
(gdb) list *(ceph_msg_release+0x15d)
0x348d is in ceph_msg_release (net/ceph/messenger.c:3534).
3529
3530        if (m->pool)
3531            ceph_msgpool_put(m->pool, m);
3532        else
3533            ceph_msg_free(m);   <== here  
3534    }
3535

Do you have a coredump and a way to analyze it?

If so, then it'd be good to track down the msg and see what you can determine about the pointer that's being freed.

This may be unrelated to MDS traffic as well. It'd be good to figure out what sort of socket the thread was servicing. If you can identify the ceph_connection in some of the earlier stack frames, then you may be able to peek at it and determine the socket address of the peer.

It also wouldn't hurt to try a newer kernel on the off chance that this may already be fixed. I'm not aware of anything specific that might address this though.

So in ceph_msg_free:

static void ceph_msg_free(struct ceph_msg *m)                                   
{                                                                               
        dout("%s %p\n", __func__, m);                                           
        kvfree(m->front.iov_base);                                              
        kfree(m->data);                                                         
        kmem_cache_free(ceph_msg_cache, m);                                     
}

"m" comes from a fixed-size slabcache, so the crash probably happened before it got freed. It'd be interesting to see what "m" looks like at this point.

No coredump file.

Reading symbols from /export/linux-4.19.15/fs/ceph/ceph.ko...done.
(gdb) list *(dispatch+0x66)
0x29266 is in dispatch (fs/ceph/mds_client.c:4149).
4144            pr_err("received unknown message type %d %s\n", type,
4145                   ceph_msg_type_name(type));
4146        }
4147    out:
4148        ceph_msg_put(msg);
4149    }

It should be related to MDS according to the next instruction returned by ceph_msg_free, and there is only mds_client.c: dispatch in the CEPH module.

struct ceph_msg *ceph_msg_new(int type, int front_len, gfp_t flags,
                  bool can_fail)
{
    struct ceph_msg *m;

    m = kmem_cache_zalloc(ceph_msg_cache, flags);
    ...
    /* front */
    if (front_len) {
        m->front.iov_base = ceph_kvmalloc(front_len, flags);
        if (m->front.iov_base == NULL) {
            dout("ceph_msg_new can't allocate %d bytes\n",
                 front_len);
            goto out2;
        }
    } else {
        m->front.iov_base = NULL;   <== Can be judged whether it is NULL when kvfree(m->front.iov_base)?
    }
    ...
}

void *ceph_kvmalloc(size_t size, gfp_t flags)
{
    if (size <= (PAGE_SIZE << PAGE_ALLOC_COSTLY_ORDER)) {
        void *ptr = kmalloc(size, flags | __GFP_NOWARN);
        if (ptr)
            return ptr;
    }

    return __vmalloc(size, flags | __GFP_HIGHMEM, PAGE_KERNEL);  <== Why use __GFP_HIGHMEM?
}

The __GFP_HIGHMEM flag indicates that "the allocated memory may be located in high memory". High Memory is the part of physical memory in a computer which is not directly mapped by the page tables of its operating system kernel.

This zone of memory is not mapped in the kernel's virtual address space, and this prevents the kernel from being capable of directly referring it.

Yanhu Cao wrote:

[...]

The __GFP_HIGHMEM flag indicates that "the allocated memory may be located in high memory". High Memory is the part of physical memory in a computer which is not directly mapped by the page tables of its operating system kernel.

This zone of memory is not mapped in the kernel's virtual address space, and this prevents the kernel from being capable of directly referring it.

Does not match the kernel version used. Ignore this.

Yanhu Cao wrote:

No coredump file.

Too bad. That would help expedite this.

It should be related to MDS according to the next instruction returned by ceph_msg_free, and there is only mds_client.c: dispatch in the CEPH module.

Quite right. I missed that that address was in ceph.ko...and yes, x86_64 doesn't really have high memory since it's a 64-bit arch.

I suspect there are two possibilities here:

1. some pointer in the msg structure has been double-freed
2. some pointer in the msg structure has been scribbled over

Determining that without a core will be difficult. Are you building your own kernels? One possibility is to turn on KASAN. It will impact performance, but it may help detect what's going wrong earlie, and provide some clues about what it is.