Ceph » Linux kernel client

Please see whether will the following commit will fix it:

commit 708c87168b6121abc74b2a57d0c498baaf70cbea
Author: Dan Carpenter <dan.carpenter@oracle.com>
Date:   Mon Sep 6 12:43:01 2021 +0300

    ceph: fix off by one bugs in unsafe_request_wait()

    The "> max" tests should be ">= max" to prevent an out of bounds access
    on the next lines.

    Fixes: e1a4541ec0b9 ("ceph: flush the mdlog before waiting on unsafe reqs")
    Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
    Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
    Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

And also please try the lasted upstream kernel if possible to see could you still see the crash, it works well for me.

Xiubo Li wrote:

Please see whether will the following commit will fix it:

According to https://github.com/torvalds/linux/commit/708c87168b6121abc74b2a57d0c498baaf70cbea, that patch is already in tag v5.15, so the kernel I was running that crash should already have it.

I didn't see any crash by using your test script by using https://github.com/ceph/ceph-client, please check whether has this been fixed in upstream ?

Sorry for the misunderstanding:

The test script I posted in https://tracker.ceph.com/issues/53809 cannot be used to reproduce this crash; it is not what crashed my nodes here.

The slowness of that script was fixed by kernel kernel 5.15. So I deployed 5.15 to my general production cluster that runs all kinds of tasks (including some fsyncs). That cluster crashed after ~13 hours of operation on the 5.15 kernel.

I don't have a reproducer to immediately crash the cluster, it just happened after normal operations. The only reason why I think it is related to https://tracker.ceph.com/issues/53809 is because there is fsync in the stack trace.

So far I couldn't get a setup to reproduce it easily, since the crash happened on my production cluseter only, and experimenting with that needs some effort.

On the crash call traces, I posted a trace in the screenshot of the crashed machine which shows the kernel panic. Is that the type of trace you are looking for?

Xiubo Li wrote:

The `mail@...` in the ceph project is your mail right ? I will ping Jeff to add the 'Reported-by:' tag in that patch.

Yes, I'm https://github.com/nh2

Niklas Hambuechen wrote:

Xiubo Li wrote:

The `mail@...` in the ceph project is your mail right ? I will ping Jeff to add the 'Reported-by:' tag in that patch.

Yes, I'm https://github.com/nh2

Sure, thanks.

Xiubo Li wrote:

I have one fix in https://patchwork.kernel.org/project/ceph-devel/list/?series=604729, this patch will fix some memory leakage issues and also one possible ZERO_SIZE_PTR dereference bug, which could cause the NULL pointer crash.

The link is no longer working. could you provide a stable link to the supposed fix?

I think this is a stable link: https://patchwork.kernel.org/project/ceph-devel/patch/20220112042904.8557-1-xiubli@redhat.com/

The commit message title is:

ceph: put the requests/sessions when it fails to alloc memory

Niklas Hambuechen wrote:

I think this is a stable link: https://patchwork.kernel.org/project/ceph-devel/patch/20220112042904.8557-1-xiubli@redhat.com/

The commit message title is:

[...]

It's just archived and you need to change your filter.

Hey Xiubo,

I deployed a kernel that has the linked fix in, and the kernel null pointer derefecence has not occured since then.

So it does look like your fix did resove it.

Thank you!