Project

General

Profile

Bug #11555

lock inversion related to memory reclaim

Added by Zheng Yan almost 9 years ago. Updated almost 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
libceph
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

crypto_alloc_blkcipher() call malloc funtion with GFP_KERNEL. we call crypto_alloc_blkcipher() while holding monc->mutex

[ 2163.217944] =========================================================
[ 2163.218006] [ INFO: possible irq lock inversion dependency detected ]
[ 2163.218006] 4.1.0-rc2+ #60 Not tainted
[ 2163.218006] ---------------------------------------------------------
[ 2163.218006] kswapd0/585 just changed the state of lock:
[ 2163.218006] (&osdc->map_sem){++++.-}, at: [<ffffffffa01721fd>] ceph_osdc_start_request+0x2d/0x70 [libceph]
[ 2163.218006] but this lock took another, RECLAIM_FS-unsafe lock in the past:
[ 2163.218006] (&monc->mutex){+.+.+.}

and interrupts could create inverse lock ordering between them.

[ 2163.218006]
other info that might help us debug this:
[ 2163.218006] Possible interrupt unsafe locking scenario:

[ 2163.218006] CPU0 CPU1
[ 2163.218006] ---- ----
[ 2163.218006] lock(&monc->mutex);
[ 2163.218006] local_irq_disable();
[ 2163.218006] lock(&osdc->map_sem);
[ 2163.218006] lock(&monc->mutex);
[ 2163.218006] <Interrupt>
[ 2163.218006] lock(&osdc->map_sem);
[ 2163.218006] * DEADLOCK *

[ 2163.218006] no locks held by kswapd0/585.
[ 2163.218006]
the shortest dependencies between 2nd lock and 1st lock:
[ 2163.218006] -> (&monc->mutex){+.+.+.} ops: 384 {
[ 2163.218006] HARDIRQ-ON-W at:
[ 2163.218006] [<ffffffff810a4137>] __lock_acquire+0x9a7/0x1d30
[ 2163.218006] [<ffffffff810a5d6c>] lock_acquire+0xbc/0x270
[ 2163.218006] [<ffffffff8194c040>] mutex_lock_nested+0x60/0x3b0
[ 2163.218006] [<ffffffffa016d168>] ceph_monc_open_session+0x18/0x40 [libceph]
[ 2163.218006] [<ffffffffa0165651>] __ceph_open_session+0x31/0x2a0 [libceph]
[ 2163.218006] [<ffffffffa01ae1a5>] 0xffffffffa01ae1a5
[ 2163.218006] [<ffffffff811a9844>] mount_fs+0x34/0x180
[ 2163.218006] [<ffffffff811c8ac6>] vfs_kern_mount+0x66/0x160
[ 2163.218006] [<ffffffff811cb814>] do_mount+0x204/0xc20
[ 2163.218006] [<ffffffff811cc536>] SyS_mount+0x86/0xd0
[ 2163.218006] [<ffffffff81950997>] system_call_fastpath+0x12/0x6f
[ 2163.218006] SOFTIRQ-ON-W at:
[ 2163.218006] [<ffffffff810a3c31>] __lock_acquire+0x4a1/0x1d30
[ 2163.218006] [<ffffffff810a5d6c>] lock_acquire+0xbc/0x270
[ 2163.218006] [<ffffffff8194c040>] mutex_lock_nested+0x60/0x3b0
[ 2163.218006] [<ffffffffa016d168>] ceph_monc_open_session+0x18/0x40 [libceph]
[ 2163.218006] [<ffffffffa0165651>] __ceph_open_session+0x31/0x2a0 [libceph]
[ 2163.218006] [<ffffffffa01ae1a5>] 0xffffffffa01ae1a5
[ 2163.218006] [<ffffffff811a9844>] mount_fs+0x34/0x180
[ 2163.218006] [<ffffffff811c8ac6>] vfs_kern_mount+0x66/0x160
[ 2163.218006] [<ffffffff811cb814>] do_mount+0x204/0xc20
[ 2163.218006] [<ffffffff811cc536>] SyS_mount+0x86/0xd0
[ 2163.218006] [<ffffffff81950997>] system_call_fastpath+0x12/0x6f
[ 2163.218006] RECLAIM_FS-ON-W at:
[ 2163.218006] [<ffffffff810a32da>] mark_held_locks+0x6a/0x90
[ 2163.218006] [<ffffffff810a67d3>] lockdep_trace_alloc+0x63/0xc0
[ 2163.218006] [<ffffffff8119acee>] kmem_cache_alloc_trace+0x2e/0x2e0
[ 2163.218006] [<ffffffff8106f5d4>] __request_module+0x134/0x3f0
[ 2163.218006] [<ffffffff812e90dc>] crypto_larval_lookup+0x7c/0x160
[ 2163.218006] [<ffffffff812e91ed>] crypto_alg_mod_lookup+0x2d/0xb0
[ 2163.218006] [<ffffffff812e92ff>] crypto_alloc_base+0x2f/0xa0
[ 2163.218006] [<ffffffffa017c19c>] ceph_aes_encrypt2+0x4c/0x2b0 [libceph]
[ 2163.218006] [<ffffffffa017c9e5>] ceph_encrypt2+0x55/0xd0 [libceph]
[ 2163.218006] [<ffffffffa017ce37>] ceph_x_encrypt+0x47/0x60 [libceph]
[ 2163.218006] [<ffffffffa017e0a1>] ceph_x_build_request+0xc1/0x2d0 [libceph]
[ 2163.218006] [<ffffffffa017ae2f>] ceph_build_auth_request+0x3f/0xa0 [libceph]
[ 2163.218006] [<ffffffffa017b418>] ceph_handle_auth_reply+0x228/0x2b0 [libceph]
[ 2163.218006] [<ffffffffa016e33f>] dispatch+0x23f/0x7a0 [libceph]
[ 2163.218006] [<ffffffffa016a7ec>] con_work+0x17ac/0x2c10 [libceph]
[ 2163.218006] [<ffffffff81074500>] process_one_work+0x1e0/0x790
[ 2163.218006] [<ffffffff81074bc9>] worker_thread+0x119/0x460
[ 2163.218006] [<ffffffff8107ad3a>] kthread+0xea/0x100
[ 2163.218006] [<ffffffff81950d92>] ret_from_fork+0x42/0x70
[ 2163.218006] INITIAL USE at:
[ 2163.218006] [<ffffffff810a3c99>] __lock_acquire+0x509/0x1d30
[ 2163.218006] [<ffffffff810a5d6c>] lock_acquire+0xbc/0x270
[ 2163.218006] [<ffffffff8194c040>] mutex_lock_nested+0x60/0x3b0
[ 2163.218006] [<ffffffffa016d168>] ceph_monc_open_session+0x18/0x40 [libceph]
[ 2163.218006] [<ffffffffa0165651>] __ceph_open_session+0x31/0x2a0 [libceph]
[ 2163.218006] [<ffffffffa01ae1a5>] 0xffffffffa01ae1a5
[ 2163.218006] [<ffffffff811a9844>] mount_fs+0x34/0x180
[ 2163.218006] [<ffffffff811c8ac6>] vfs_kern_mount+0x66/0x160
[ 2163.218006] [<ffffffff811cb814>] do_mount+0x204/0xc20
[ 2163.218006] [<ffffffff811cc536>] SyS_mount+0x86/0xd0
[ 2163.218006] [<ffffffff81950997>] system_call_fastpath+0x12/0x6f
[ 2163.218006] }
[ 2163.218006] ... key at: [<ffffffffa0190068>] __key.30445+0x0/0xfffffffffffef5eb [libceph]
[ 2163.218006] ... acquired at:
[ 2163.218006] [<ffffffff810a5d6c>] lock_acquire+0xbc/0x270
[ 2163.218006] [<ffffffff8194c040>] mutex_lock_nested+0x60/0x3b0
[ 2163.218006] [<ffffffffa016e931>] ceph_monc_got_osdmap+0x21/0x50 [libceph]
[ 2163.218006] [<ffffffffa01732be>] ceph_osdc_handle_map+0x29e/0x5e0 [libceph]
[ 2163.218006] [<ffffffffa016e14f>] dispatch+0x4f/0x7a0 [libceph]
[ 2163.218006] [<ffffffffa016a7ec>] con_work+0x17ac/0x2c10 [libceph]
[ 2163.218006] [<ffffffff81074500>] process_one_work+0x1e0/0x790
[ 2163.218006] [<ffffffff81074bc9>] worker_thread+0x119/0x460
[ 2163.218006] [<ffffffff8107ad3a>] kthread+0xea/0x100
[ 2163.218006] [<ffffffff81950d92>] ret_from_fork+0x42/0x70

[ 2163.218006] > (&osdc->map_sem){++++.} ops: 239556 {
[ 2163.218006] HARDIRQ-ON-W at:
[ 2163.218006] [<ffffffff810a4137>] __lock_acquire+0x9a7/0x1d30
[ 2163.218006] [<ffffffff810a5d6c>] lock_acquire+0xbc/0x270
[ 2163.218006] [<ffffffff8194e384>] down_write+0x44/0x80
[ 2163.218006] [<ffffffffa01730d6>] ceph_osdc_handle_map+0xb6/0x5e0 [libceph]
[ 2163.218006] [<ffffffffa016e14f>] dispatch+0x4f/0x7a0 [libceph]
[ 2163.218006] [<ffffffffa016a7ec>] con_work+0x17ac/0x2c10 [libceph]
[ 2163.218006] [<ffffffff81074500>] process_one_work+0x1e0/0x790
[ 2163.218006] [<ffffffff81074bc9>] worker_thread+0x119/0x460
[ 2163.218006] [<ffffffff8107ad3a>] kthread+0xea/0x100
[ 2163.218006] [<ffffffff81950d92>] ret_from_fork+0x42/0x70
[ 2163.218006] HARDIRQ-ON-R at:
[ 2163.218006] [<ffffffff810a3c08>] __lock_acquire+0x478/0x1d30
[ 2163.218006] [<ffffffff810a5d6c>] lock_acquire+0xbc/0x270
[ 2163.218006] [<ffffffff8194e327>] down_read+0x47/0x60
[ 2163.218006] [<ffffffffa01721fd>] ceph_osdc_start_request+0x2d/0x70 [libceph]
[ 2163.218006] [<ffffffffa01bd0e6>] 0xffffffffa01bd0e6
[ 2163.218006] [<ffffffffa01c4037>] 0xffffffffa01c4037
[ 2163.218006] [<ffffffffa01b657f>] 0xffffffffa01b657f
[ 2163.218006] [<ffffffff811a5389>] __vfs_write+0xa9/0xe0
[ 2163.218006] [<ffffffff811a59d4>] vfs_write+0xa4/0x1b0
[ 2163.218006] [<ffffffff811a6784>] SyS_write+0x44/0xb0
[ 2163.218006] [<ffffffff81950997>] system_call_fastpath+0x12/0x6f
[ 2163.218006] SOFTIRQ-ON-W at:
[ 2163.218006] [<ffffffff810a3c31>] __lock_acquire+0x4a1/0x1d30
[ 2163.218006] [<ffffffff810a5d6c>] lock_acquire+0xbc/0x270
[ 2163.218006] [<ffffffff8194e384>] down_write+0x44/0x80
[ 2163.218006] [<ffffffffa01730d6>] ceph_osdc_handle_map+0xb6/0x5e0 [libceph]
[ 2163.218006] [<ffffffffa016e14f>] dispatch+0x4f/0x7a0 [libceph]
[ 2163.218006] [<ffffffffa016a7ec>] con_work+0x17ac/0x2c10 [libceph]
[ 2163.218006] [<ffffffff81074500>] process_one_work+0x1e0/0x790
[ 2163.218006] [<ffffffff81074bc9>] worker_thread+0x119/0x460
[ 2163.218006] [<ffffffff8107ad3a>] kthread+0xea/0x100
[ 2163.218006] [<ffffffff81950d92>] ret_from_fork+0x42/0x70
[ 2163.218006] SOFTIRQ-ON-R at:
[ 2163.218006] [<ffffffff810a3c31>] __lock_acquire+0x4a1/0x1d30
[ 2163.218006] [<ffffffff810a5d6c>] lock_acquire+0xbc/0x270
[ 2163.218006] [<ffffffff8194e327>] down_read+0x47/0x60
[ 2163.218006] [<ffffffffa01721fd>] ceph_osdc_start_request+0x2d/0x70 [libceph]
[ 2163.218006] [<ffffffffa01bd0e6>] 0xffffffffa01bd0e6
[ 2163.218006] [<ffffffffa01c4037>] 0xffffffffa01c4037
[ 2163.218006] [<ffffffffa01b657f>] 0xffffffffa01b657f
[ 2163.218006] [<ffffffff811a5389>] __vfs_write+0xa9/0xe0
[ 2163.218006] [<ffffffff811a59d4>] vfs_write+0xa4/0x1b0
[ 2163.218006] [<ffffffff811a6784>] SyS_write+0x44/0xb0
[ 2163.218006] [<ffffffff81950997>] system_call_fastpath+0x12/0x6f
[ 2163.218006] IN-RECLAIM_FS-R at:
[ 2163.218006] [<ffffffff810a3c73>] __lock_acquire+0x4e3/0x1d30
[ 2163.218006] [<ffffffff810a5d6c>] lock_acquire+0xbc/0x270
[ 2163.218006] [<ffffffff8194e327>] down_read+0x47/0x60
[ 2163.218006] [<ffffffffa01721fd>] ceph_osdc_start_request+0x2d/0x70 [libceph]
[ 2163.218006] [<ffffffffa017250c>] ceph_osdc_writepages+0x10c/0x1b0 [libceph]
[ 2163.218006] [<ffffffffa0279abb>] writepage_nounlock+0x25b/0x450 [ceph]
[ 2163.218006] [<ffffffffa027a152>] ceph_writepage+0x32/0x60 [ceph]
[ 2163.218006] [<ffffffff8115c6c2>] shrink_page_list+0x472/0xb60
[ 2163.218006] [<ffffffff8115d38e>] shrink_inactive_list+0x25e/0x680
[ 2163.218006] [<ffffffff8115e0c5>] shrink_lruvec+0x5c5/0x710
[ 2163.218006] [<ffffffff8115e269>] shrink_zone+0x59/0x180
[ 2163.218006] [<ffffffff8115f309>] kswapd+0x4c9/0xc10
[ 2163.218006] [<ffffffff8107ad3a>] kthread+0xea/0x100
[ 2163.218006] [<ffffffff81950d92>] ret_from_fork+0x42/0x70
[ 2163.218006] INITIAL USE at:
[ 2163.218006] [<ffffffff810a3c99>] __lock_acquire+0x509/0x1d30
[ 2163.218006] [<ffffffff810a5d6c>] lock_acquire+0xbc/0x270
[ 2163.218006] [<ffffffff8194e384>] down_write+0x44/0x80
[ 2163.218006] [<ffffffffa01730d6>] ceph_osdc_handle_map+0xb6/0x5e0 [libceph]
[ 2163.218006] [<ffffffffa016e14f>] dispatch+0x4f/0x7a0 [libceph]
[ 2163.218006] [<ffffffffa016a7ec>] con_work+0x17ac/0x2c10 [libceph]
[ 2163.218006] [<ffffffff81074500>] process_one_work+0x1e0/0x790
[ 2163.218006] [<ffffffff81074bc9>] worker_thread+0x119/0x460
[ 2163.218006] [<ffffffff8107ad3a>] kthread+0xea/0x100
[ 2163.218006] [<ffffffff81950d92>] ret_from_fork+0x42/0x70
[ 2163.218006] }
[ 2163.218006] ... key at: [<ffffffffa01900b0>] __key.31877+0x0/0xfffffffffffef5a3 [libceph]
[ 2163.218006] ... acquired at:
[ 2163.218006] [<ffffffff810a2906>] check_usage_forwards+0x126/0x130
[ 2163.218006] [<ffffffff810a3166>] mark_lock+0x1a6/0x2b0
[ 2163.218006] [<ffffffff810a3c73>] __lock_acquire+0x4e3/0x1d30
[ 2163.218006] [<ffffffff810a5d6c>] lock_acquire+0xbc/0x270
[ 2163.218006] [<ffffffff8194e327>] down_read+0x47/0x60
[ 2163.218006] [<ffffffffa01721fd>] ceph_osdc_start_request+0x2d/0x70 [libceph]
[ 2163.218006] [<ffffffffa017250c>] ceph_osdc_writepages+0x10c/0x1b0 [libceph]
[ 2163.218006] [<ffffffffa0279abb>] writepage_nounlock+0x25b/0x450 [ceph]
[ 2163.218006] [<ffffffffa027a152>] ceph_writepage+0x32/0x60 [ceph]
[ 2163.218006] [<ffffffff8115c6c2>] shrink_page_list+0x472/0xb60
[ 2163.218006] [<ffffffff8115d38e>] shrink_inactive_list+0x25e/0x680
[ 2163.218006] [<ffffffff8115e0c5>] shrink_lruvec+0x5c5/0x710
[ 2163.218006] [<ffffffff8115e269>] shrink_zone+0x59/0x180
[ 2163.218006] [<ffffffff8115f309>] kswapd+0x4c9/0xc10
[ 2163.218006] [<ffffffff8107ad3a>] kthread+0xea/0x100
[ 2163.218006] [<ffffffff81950d92>] ret_from_fork+0x42/0x70

[ 2163.218006]
stack backtrace:
[ 2163.218006] CPU: 0 PID: 585 Comm: kswapd0 Not tainted 4.1.0-rc2+ #60
[ 2163.218006] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 2163.218006] ffff88007c0611e0 ffff88007a8175d8 ffffffff81943170 0000000000000000
[ 2163.218006] ffffffff82b6fce0 ffff88007a817628 ffffffff8193f1de ffff88007a817678
[ 2163.218006] ffffffff81c7ccf2 ffff88007a81763c 0000000000000000 ffff88007c061a18
[ 2163.218006] Call Trace:
[ 2163.218006] [<ffffffff81943170>] dump_stack+0x4c/0x65
[ 2163.218006] [<ffffffff8193f1de>] print_irq_inversion_bug.part.40+0x1ae/0x1bd
[ 2163.218006] [<ffffffff810a2906>] check_usage_forwards+0x126/0x130
[ 2163.218006] [<ffffffff810a27e0>] ? check_usage_backwards+0x130/0x130
[ 2163.218006] [<ffffffff810a3166>] mark_lock+0x1a6/0x2b0
[ 2163.218006] [<ffffffff810a3c73>] __lock_acquire+0x4e3/0x1d30
[ 2163.218006] [<ffffffff8119abc3>] ? kmem_cache_alloc+0x203/0x300
[ 2163.218006] [<ffffffffa0166d71>] ? ceph_msg_data_create+0x41/0x60 [libceph]
[ 2163.218006] [<ffffffff810a5d6c>] lock_acquire+0xbc/0x270
[ 2163.218006] [<ffffffffa01721fd>] ? ceph_osdc_start_request+0x2d/0x70 [libceph]
[ 2163.218006] [<ffffffff8194e327>] down_read+0x47/0x60
[ 2163.218006] [<ffffffffa01721fd>] ? ceph_osdc_start_request+0x2d/0x70 [libceph]
[ 2163.218006] [<ffffffffa01721fd>] ceph_osdc_start_request+0x2d/0x70 [libceph]
[ 2163.218006] [<ffffffffa017250c>] ceph_osdc_writepages+0x10c/0x1b0 [libceph]
[ 2163.218006] [<ffffffffa0279abb>] writepage_nounlock+0x25b/0x450 [ceph]
[ 2163.218006] [<ffffffffa027a152>] ceph_writepage+0x32/0x60 [ceph]
[ 2163.218006] [<ffffffff8115c6c2>] shrink_page_list+0x472/0xb60
[ 2163.218006] [<ffffffff8115d38e>] shrink_inactive_list+0x25e/0x680
[ 2163.218006] [<ffffffff8115e0c5>] shrink_lruvec+0x5c5/0x710
[ 2163.218006] [<ffffffff810c8e00>] ? detach_if_pending+0x250/0x250
[ 2163.218006] [<ffffffff8115e269>] shrink_zone+0x59/0x180
[ 2163.218006] [<ffffffff8115f309>] kswapd+0x4c9/0xc10
[ 2163.218006] [<ffffffff8115ee40>] ? shrink_all_memory+0xb0/0xb0
[ 2163.218006] [<ffffffff8107ad3a>] kthread+0xea/0x100
[ 2163.218006] [<ffffffff8107ac50>] ? kthread_create_on_node+0x220/0x220
[ 2163.218006] [<ffffffff81950d92>] ret_from_fork+0x42/0x70
[ 2163.218006] [<ffffffff8107ac50>] ? kthread_create_on_node+0x220/0x220
[ 3288.216430] libceph: client898731 fsid 46574da0-8084-4405-95fe-cb546a197920
[ 3288.220293] libceph: mon0 10.0.0.10:6789 session established

History

#1 Updated by Josh Durgin almost 9 years ago

  • Priority changed from Normal to High

#2 Updated by Ilya Dryomov over 8 years ago

  • Priority changed from High to Normal

#3 Updated by Ilya Dryomov almost 7 years ago

  • Category set to libceph
  • Status changed from New to Resolved
  • Assignee set to Ilya Dryomov

We were allocating (with GFP_KERNEL) and destroying the cipher context on each encrypt/decrypt operation:

https://www.spinics.net/lists/ceph-devel/msg33806.html
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7af3ea189a9a13f090de51c97f676215dabc1205

This was fixed in 4.10. We are still calling crypto_alloc_skcipher() under monc->mutex, but only when the new key needs to be installed -- that's like 1/1000th of the original bug.

Also available in: Atom PDF