Bug #48749
openUse after free in osdmap handling
0%
Description
I got the following KASAN pop when testing with the new msgr2 code:
[ 7741.292758] ================================================================== [ 7741.301082] BUG: KASAN: use-after-free in __ceph_open_session+0x609/0x680 [libceph] [ 7741.309891] Read of size 4 at addr ffff8881fc0be410 by task mount.ceph/259885 [ 7741.317461] [ 7741.322723] CPU: 4 PID: 259885 Comm: mount.ceph Not tainted 4.18.0-269.el8.bz1893177.3.x86_64+debug #1 [ 7741.330911] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-1.fc33 04/01/2014 [ 7741.338959] Call Trace: [ 7741.344406] dump_stack+0x8e/0xd0 [ 7741.350428] ? __ceph_open_session+0x609/0x680 [libceph] [ 7741.356222] print_address_description.constprop.5+0x1e/0x230 [ 7741.359147] ? kmsg_dump_rewind_nolock+0xd9/0xd9 [ 7741.362102] ? lockdep_hardirqs_on_prepare+0x343/0x4f0 [ 7741.364909] ? __ceph_open_session+0x609/0x680 [libceph] [ 7741.367733] ? __ceph_open_session+0x609/0x680 [libceph] [ 7741.370478] ? __ceph_open_session+0x609/0x680 [libceph] [ 7741.373239] __kasan_report.cold.7+0x37/0x86 [ 7741.375811] ? __ceph_open_session+0x609/0x680 [libceph] [ 7741.378548] kasan_report+0x37/0x50 [ 7741.380943] __ceph_open_session+0x609/0x680 [libceph] [ 7741.383655] ? ceph_reset_client_addr+0x30/0x30 [libceph] [ 7741.386342] ? finish_wait+0x280/0x280 [ 7741.388714] ? sget+0x3be/0x4c0 [ 7741.390972] ceph_mount+0x1144/0x1b30 [ceph] [ 7741.393406] ? selinux_sb_eat_lsm_opts+0x38e/0x780 [ 7741.395925] ? param_set_metrics+0x150/0x150 [ceph] [ 7741.398443] legacy_get_tree+0x105/0x200 [ 7741.400791] ? security_capable+0x58/0x90 [ 7741.403102] vfs_get_tree+0x89/0x3c0 [ 7741.405406] ? ns_capable_common+0x64/0xe0 [ 7741.407730] do_mount+0xe33/0x15b0 [ 7741.409990] ? copy_mount_string+0x20/0x20 [ 7741.412367] ? kasan_unpoison_shadow+0x30/0x40 [ 7741.414772] ? __kasan_kmalloc.constprop.9+0xc1/0xd0 [ 7741.417579] ? copy_mount_options+0x53/0x2a0 [ 7741.419839] ? kmem_cache_alloc_trace+0x159/0x360 [ 7741.422154] ksys_mount+0xb6/0xd0 [ 7741.424210] __x64_sys_mount+0xba/0x150 [ 7741.426314] do_syscall_64+0xa5/0x420 [ 7741.428369] entry_SYSCALL_64_after_hwframe+0x6a/0xdf [ 7741.430628] RIP: 0033:0x7f7a60e672ce [ 7741.432642] Code: 48 8b 0d bd eb 2b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8a eb 2b 00 f7 d8 64 89 01 48 [ 7741.438659] RSP: 002b:00007ffe4e227fd8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a5 [ 7741.441652] RAX: ffffffffffffffda RBX: 00007ffe4e228068 RCX: 00007f7a60e672ce [ 7741.444795] RDX: 0000561c9e354f98 RSI: 00007ffe4e2298ce RDI: 0000561c9f768080 [ 7741.447771] RBP: 0000561c9f768080 R08: 0000561c9f7712b0 R09: 00007ffe4e228074 [ 7741.450584] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffe4e228010 [ 7741.453387] R13: 00007ffe4e228070 R14: 0000000000000053 R15: 00007ffe4e228070 [ 7741.456163] [ 7741.457915] Allocated by task 259885: [ 7741.460001] kasan_save_stack+0x19/0x80 [ 7741.462110] __kasan_kmalloc.constprop.9+0xc1/0xd0 [ 7741.464364] kmem_cache_alloc_trace+0x159/0x360 [ 7741.466539] ceph_osdmap_alloc+0x3f/0x2b0 [libceph] [ 7741.468712] ceph_osdc_init+0x424/0x990 [libceph] [ 7741.470889] ceph_create_client+0x237/0x300 [libceph] [ 7741.473106] ceph_mount+0x6b6/0x1b30 [ceph] [ 7741.475186] legacy_get_tree+0x105/0x200 [ 7741.477170] vfs_get_tree+0x89/0x3c0 [ 7741.479148] do_mount+0xe33/0x15b0 [ 7741.481060] ksys_mount+0xb6/0xd0 [ 7741.482981] __x64_sys_mount+0xba/0x150 [ 7741.485003] do_syscall_64+0xa5/0x420 [ 7741.486960] entry_SYSCALL_64_after_hwframe+0x6a/0xdf [ 7741.489181] [ 7741.490828] Freed by task 243052: [ 7741.492786] kasan_save_stack+0x19/0x80 [ 7741.494821] kasan_set_track+0x1c/0x30 [ 7741.496851] kasan_set_free_info+0x1b/0x30 [ 7741.499021] __kasan_slab_free+0x108/0x150 [ 7741.501099] slab_free_freelist_hook+0x7e/0x140 [ 7741.503293] kfree+0xd6/0x2d0 [ 7741.505174] handle_one_map+0x3c6/0x7d0 [libceph] [ 7741.507324] ceph_osdc_handle_map+0xfbc/0x16d0 [libceph] [ 7741.509551] mon_dispatch+0x3ae/0x1290 [libceph] [ 7741.511663] ceph_con_process_message+0x1f6/0x640 [libceph] [ 7741.514027] process_message+0xe/0xa0 [libceph] [ 7741.516162] ceph_con_v2_try_read+0x16d4/0x32a0 [libceph] [ 7741.518455] ceph_con_workfn+0x30b/0x13f0 [libceph] [ 7741.520643] process_one_work+0x8f0/0x1770 [ 7741.522680] worker_thread+0x87/0xb40 [ 7741.524628] kthread+0x342/0x410 [ 7741.526522] ret_from_fork+0x27/0x50 [ 7741.528492] [ 7741.530095] The buggy address belongs to the object at ffff8881fc0be400 [ 7741.530095] which belongs to the cache kmalloc-512 of size 512 [ 7741.535152] The buggy address is located 16 bytes inside of [ 7741.535152] 512-byte region [ffff8881fc0be400, ffff8881fc0be600) [ 7741.540002] The buggy address belongs to the page: [ 7741.542162] page:ffffea0007f02e00 refcount:1 mapcount:0 mapping:ffff888107c0e100 index:0x0 compound_mapcount: 0 [ 7741.545190] flags: 0x17ffffc0008100(slab|head) [ 7741.547331] raw: 0017ffffc0008100 0000000000000000 0000000100000001 ffff888107c0e100 [ 7741.550005] raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000 [ 7741.552635] page dumped because: kasan: bad access detected [ 7741.554974] [ 7741.556688] Memory state around the buggy address: [ 7741.558900] ffff8881fc0be300: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 7741.561699] ffff8881fc0be380: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 7741.564432] >ffff8881fc0be400: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 7741.567158] ^ [ 7741.569264] ffff8881fc0be480: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 7741.572016] ffff8881fc0be500: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 7741.574703] ==================================================================
I think this is a long-standing bug and not a regression, but I'm not certain. I think there are a couple of potential fixes -- we could have have_mon_and_osd_map() take the osdc->lock before checking the pointers (and probably take the monc->mutex for that bit).
Alternately, this may be a place where we could convert these objects to being freed via RCU and simply take the rcu_read_lock() when accessing them.
Updated by Jeff Layton over 3 years ago
I took a quick stab at a patch to take the locks before checking this, but that won't work. osdc->lock and monc->mutex are both sleeping locks, and we can't take them in the context of a wait condition.
We could hold the mutexes across the sleep in __ceph_open_session, but that's pretty nasty. I think that leaves us with converting these objects to be freed via RCU.
It's a bit of work to convert all of the existing accessors of these fields to use RCU-capable wrappers. Ilya, do you have any thoughts on an alternate approach before I dive into doing this conversion?
Updated by Ilya Dryomov over 3 years ago
- Category set to libceph
Jeff and I discussed this on IRC yesterday. It is indeed a long standing bug and the splat just highlights the mess around osdmap and, to a much lesser extent, monmap locking and parsing (!). While we could come up with something to cover up this particular use-after-free, it's just one manifestation of the problem (and also not the only use-after-free because some osdmap accessors aren't locked). I think we can probably live with it for now and prioritize osdmap/monmap handling rework.