Project

General

Profile

Actions
Copy link

Bug #53710

closed

ceph_direct_read_write handle NULL pointer dereference

Added by tao song over 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Jeff Layton
Category:
fs/ceph
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
v3.0
ceph-qa-suite:
fs
Crash signature (v1):
Crash signature (v2):

Description

Enable cephfs kernel client debug logging during FIO test，a null pointer in the kernel causes a crash

fio：direct + read +libaio
fio -filename=./test4 -direct=1 -iodepth 1 -thread -rw=read -ioengine=libaio -bs=1M -size=1G -numjobs=1 -runtime=60 -group_reporting -name=mytest

open kclient ceph.ko debug:
echo 'module ceph +p' > /sys/kernel/debug/dynamic_debug/control

kernel & kclient version:
4.14.113-1.el7.x86_64 4.19

[ 1874.024995] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[ 1874.025138] IP: ceph_direct_read_write+0xa4a/0xbd0 [ceph]
[ 1874.025178] PGD 80000001270a0067 P4D 80000001270a0067 PUD 135d54067 PMD 0
[ 1874.025228] Oops: 0000 [#1] SMP PTI
[ 1874.025299] Modules linked in: ceph(OE) libceph(OE) dns_resolver(E) fscache(E) sunrpc(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) pcbc(E) aesni_intel(E) crypto_simd(E) glue_helper(E) cryptd(E) joydev(E) virtio_rng(E) sg(E) virtio_balloon(E) pcspkr(E) i2c_piix4(E) ip_tables(E) xfs(E) libcrc32c(E) sr_mod(E) cdrom(E) virtio_net(E) virtio_blk(E) virtio_console(E) ata_generic(E) pata_acpi(E) crc32c_intel(E) serio_raw(E) cirrus(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ata_piix(E) floppy(E) ttm(E) libata(E) drm(E) virtio_pci(E) virtio_ring(E) virtio(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[ 1874.025744] CPU: 2 PID: 80059 Comm: fio Tainted: G OE 4.14.113-1.el7.x86_64 #1
[ 1874.025799] Hardware name: Red Hat KVM, BIOS 1.11.0-2.el7 04/01/2014
[ 1874.025842] task: ffff8e5e380b4500 task.stack: ffff99cd82130000
[ 1874.025891] RIP: 0010:ceph_direct_read_write+0xa4a/0xbd0 [ceph]
[ 1874.025932] RSP: 0018:ffff99cd82133b08 EFLAGS: 00010246
[ 1874.025968] RAX: 0000000000000001 RBX: ffff8e5e37f98800 RCX: ffff8e5e37f98800
[ 1874.026015] RDX: ffffffffc05b1b1a RSI: ffffffffc05a68a8 RDI: 0000000000000000
[ 1874.026062] RBP: ffff99cd82133c48 R08: 0000000000000000 R09: 0000000000100000
[ 1874.026109] R10: 0000000000000004 R11: 0000000000000000 R12: ffff8e5d9b674600
[ 1874.026156] R13: 0000000000000000 R14: 0000000000000000 R15: ffff99cd82133d68
[ 1874.026204] FS: 00007f15d14db700(0000) GS:ffff8e5e3fd00000(0000) knlGS:0000000000000000
[ 1874.026257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1874.026296] CR2: 0000000000000008 CR3: 000000011db92004 CR4: 00000000001606e0
[ 1874.026360] Call Trace:
[ 1874.026518] ? printk+0x58/0x6f
[ 1874.026555] ? ceph_read_iter+0x8a9/0xb80 [ceph]
[ 1874.026619] ? handle_mm_fault+0xc4/0x1d0
[ 1874.026666] ? aio_read+0xe8/0x140
[ 1874.026703] ? kmem_cache_alloc+0xd2/0x1a0
[ 1874.026736] ? do_io_submit+0x3e3/0x690
[ 1874.026789] ? do_syscall_64+0x74/0x1a0
[ 1874.026849] ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[ 1874.026887] Code: 8b 4c 24 60 45 84 f6 4c 8b 84 24 c0 00 00 00 48 c7 c1 14 1b 5b c0 48 c7 c2 1a 1b 5b c0 48 0f 45 d1 48 c7 c6 a8 68 5a c0 48 89 d9 <48> 8b 47 08 48 89 3c 24 48 c7 c7 f0 df 5b c0 48 89 44 24 08 31
[ 1874.027055] RIP: ceph_direct_read_write+0xa4a/0xbd0 [ceph] RSP: ffff99cd82133b08
[ 1874.027104] CR2: 0000000000000008

Files

vmcore-dmesg.txt.tar.gz (85.9 KB) vmcore-dmesg.txt.tar.gz tao song, 12/23/2021 09:18 AM
Actions
Copy link
 #1

Updated by tao song over 2 years ago

The read operation entry function ceph_read_iter in file.c. The parameter snapc is null,when called ceph_direct_read_write .And in ceph_direct_read_write ,a null pointer was referenced when "dout" print log.


......
if (ci->i_inline_version == CEPH_INLINE_NONE) {
if (!retry_op && (iocb->ki_flags & IOCB_DIRECT)) {
ret = ceph_direct_read_write(iocb, to,
NULL, NULL, need_qos);
if (ret >= 0 && ret < len)
retry_op = CHECK_EOF;
} else {
ret = ceph_sync_read(iocb, to, &retry_op, need_qos);
}
} else {
retry_op = READ_INLINE;
}
......

@
if (write && ceph_snap(file_inode(file)) != CEPH_NOSNAP)
return -EROFS;

dout("sync_direct_%s on file %p %lld~%u snapc %p seq %lld\n",
         (write ? "write" : "read"), file, pos, (unsigned)count,
         snapc, snapc->seq);

@

Actions
Copy link
 #2

Updated by Jeff Layton almost 2 years ago

  • Status changed from New to Resolved
  • Assignee set to Jeff Layton

This was fixed in mainline with this commit:

commit 40e7e2c0e86464bca839cdf891bd58a6d41b60b4
Author: Jeff Layton <jlayton@kernel.org>
Date:   Tue Apr 23 14:18:45 2019 -0400

    ceph: fix NULL pointer deref when debugging is enabled

    Signed-off-by: Jeff Layton <jlayton@kernel.org>
    Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Actions
Copy link

Also available in: Atom PDF