Actions
Bug #1795
closedbreak d_lock > s_cap_lock ordering
% Done:
0%
Source:
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
Dec 7 15:06:27 tgpro1 kernel: ======================================================= Dec 7 15:06:27 tgpro1 kernel: [ INFO: possible circular locking dependency detected ] Dec 7 15:06:27 tgpro1 kernel: ------------------------------------------------------- Dec 7 15:06:27 tgpro1 kernel: kworker/4:2/3143 is trying to acquire lock: Dec 7 15:06:27 tgpro1 kernel: (&sb->s_type->i_lock_key#16){+.+...}, at: [<000ff721>] igrab+0x11/0x41 Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: but task is already holding lock: Dec 7 15:06:27 tgpro1 kernel: (&(&s->s_cap_lock)->rlock){+.+...}, at: [<006d0dd0>] iterate_session_caps+0x40/0x17d [ceph] Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: which lock already depends on the new lock. Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: the existing dependency chain (in reverse order) is: Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: -> #2 (&(&s->s_cap_lock)->rlock){+.+...}: Dec 7 15:06:27 tgpro1 kernel: [<00091456>] lock_acquire+0x42/0x59 Dec 7 15:06:27 tgpro1 kernel: [<0053f187>] _raw_spin_lock+0x24/0x33 Dec 7 15:06:27 tgpro1 kernel: [<006c180f>] ceph_d_revalidate+0xf4/0x29e [ceph] Dec 7 15:06:27 tgpro1 kernel: [<000f3339>] do_lookup+0x223/0x2a6 Dec 7 15:06:27 tgpro1 kernel: [<000f4d99>] path_lookupat+0x118/0x6e9 Dec 7 15:06:27 tgpro1 kernel: [<000f5386>] do_path_lookup+0x1c/0x4e Dec 7 15:06:27 tgpro1 kernel: [<000f544c>] user_path_at_empty+0x3e/0x69 Dec 7 15:06:27 tgpro1 kernel: [<000f5484>] user_path_at+0xd/0xf Dec 7 15:06:27 tgpro1 kernel: [<000ecf98>] vfs_fstatat+0x40/0x67 Dec 7 15:06:27 tgpro1 kernel: [<000ed003>] vfs_lstat+0x16/0x18 Dec 7 15:06:27 tgpro1 kernel: [<000ed019>] sys_lstat64+0x14/0x28 Dec 7 15:06:27 tgpro1 kernel: [<0053f9ca>] syscall_call+0x7/0xb Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: -> #1 (&(&dentry->d_lock)->rlock){+.+.-.}: Dec 7 15:06:27 tgpro1 kernel: [<00091456>] lock_acquire+0x42/0x59 Dec 7 15:06:27 tgpro1 kernel: [<0053f187>] _raw_spin_lock+0x24/0x33 Dec 7 15:06:27 tgpro1 kernel: [<000fca43>] __d_instantiate+0x14/0xbc Dec 7 15:06:27 tgpro1 kernel: [<000fcf4e>] d_instantiate+0x30/0x40 Dec 7 15:06:27 tgpro1 kernel: [<000fe357>] d_alloc_root+0x27/0x2d Dec 7 15:06:27 tgpro1 kernel: [<006bd2e2>] open_root_dentry+0xfd/0x143 [ceph] Dec 7 15:06:27 tgpro1 kernel: [<006bde68>] ceph_mount+0x338/0x4c4 [ceph] Dec 7 15:06:27 tgpro1 kernel: [<000eaac4>] mount_fs+0xe/0x95 Dec 7 15:06:27 tgpro1 kernel: [<00103266>] vfs_kern_mount+0x4c/0x80 Dec 7 15:06:27 tgpro1 kernel: [<001046aa>] do_kern_mount+0x2f/0xaf Dec 7 15:06:27 tgpro1 kernel: [<001051f0>] do_mount+0x26b/0x2aa Dec 7 15:06:27 tgpro1 kernel: [<00105290>] sys_mount+0x61/0x94 Dec 7 15:06:27 tgpro1 kernel: [<0053f9ca>] syscall_call+0x7/0xb Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: -> #0 (&sb->s_type->i_lock_key#16){+.+...}: Dec 7 15:06:27 tgpro1 kernel: [<00090b06>] __lock_acquire+0xe7b/0x1428 Dec 7 15:06:27 tgpro1 kernel: [<00091456>] lock_acquire+0x42/0x59 Dec 7 15:06:27 tgpro1 kernel: [<0053f187>] _raw_spin_lock+0x24/0x33 Dec 7 15:06:27 tgpro1 kernel: [<000ff721>] igrab+0x11/0x41 Dec 7 15:06:27 tgpro1 kernel: [<006d0e00>] iterate_session_caps+0x70/0x17d [ceph] Dec 7 15:06:27 tgpro1 kernel: [<006d3bb4>] send_mds_reconnect+0x319/0x46a [ceph] Dec 7 15:06:27 tgpro1 kernel: [<006d401f>] ceph_mdsc_handle_map+0x2f2/0x42d [ceph] Dec 7 15:06:27 tgpro1 kernel: [<006be00c>] extra_mon_dispatch+0x18/0x1c [ceph] Dec 7 15:06:27 tgpro1 kernel: [<006963ed>] dispatch+0x4f7/0x52a [libceph] Dec 7 15:06:27 tgpro1 kernel: [<00694459>] con_work+0x14f4/0x16c3 [libceph] Dec 7 15:06:27 tgpro1 kernel: [<0007b331>] process_one_work+0x229/0x397 Dec 7 15:06:27 tgpro1 kernel: [<0007b80e>] worker_thread+0x182/0x2d8 Dec 7 15:06:27 tgpro1 kernel: [<0007e843>] kthread+0x62/0x67 Dec 7 15:06:27 tgpro1 kernel: [<00540c86>] kernel_thread_helper+0x6/0xd Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: other info that might help us debug this: Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: Chain exists of: Dec 7 15:06:27 tgpro1 kernel: &sb->s_type->i_lock_key --> &(&dentry->d_lock)->rlock --> &(&s->s_cap_lock)->rlock Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: Possible unsafe locking scenario: Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: CPU0 CPU1 Dec 7 15:06:27 tgpro1 kernel: ---- ---- Dec 7 15:06:27 tgpro1 kernel: lock(&(&s->s_cap_lock)->rlock); Dec 7 15:06:27 tgpro1 kernel: lock(&(&dentry->d_lock)->rlock); Dec 7 15:06:27 tgpro1 kernel: lock(&(&s->s_cap_lock)->rlock); Dec 7 15:06:27 tgpro1 kernel: lock(&sb->s_type->i_lock_key); Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: *** DEADLOCK *** Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: 5 locks held by kworker/4:2/3143: Dec 7 15:06:27 tgpro1 kernel: #0: (ceph-msgr){++++.+}, at: [<0007b2e5>] process_one_work+0x1dd/0x397 Dec 7 15:06:27 tgpro1 kernel: #1: ((&(&con->work)->work)){+.+.+.}, at: [<0007b2e5>] process_one_work+0x1dd/0x397 Dec 7 15:06:27 tgpro1 kernel: #2: (&s->s_mutex){+.+.+.}, at: [<006d393e>] send_mds_reconnect+0xa3/0x46a [ceph] Dec 7 15:06:27 tgpro1 kernel: #3: (&mdsc->snap_rwsem){+++++.}, at: [<006d3a36>] send_mds_reconnect+0x19b/0x46a [ceph] Dec 7 15:06:27 tgpro1 kernel: #4: (&(&s->s_cap_lock)->rlock){+.+...}, at: [<006d0dd0>] iterate_session_caps+0x40/0x17d [ceph] Dec 7 15:06:27 tgpro1 kernel: Dec 7 15:06:27 tgpro1 kernel: stack backtrace: Dec 7 15:06:27 tgpro1 kernel: Call Trace: Dec 7 15:06:27 tgpro1 kernel: [<0008f840>] print_circular_bug+0x219/0x226 Dec 7 15:06:27 tgpro1 kernel: [<00090b06>] __lock_acquire+0xe7b/0x1428 Dec 7 15:06:27 tgpro1 kernel: [<00091456>] lock_acquire+0x42/0x59 Dec 7 15:06:27 tgpro1 kernel: [<000ff721>] ? igrab+0x11/0x41 Dec 7 15:06:27 tgpro1 kernel: [<0053f187>] _raw_spin_lock+0x24/0x33 Dec 7 15:06:27 tgpro1 kernel: [<000ff721>] ? igrab+0x11/0x41 Dec 7 15:06:27 tgpro1 kernel: [<000ff721>] igrab+0x11/0x41 Dec 7 15:06:27 tgpro1 kernel: [<006d0e00>] iterate_session_caps+0x70/0x17d [ceph] Dec 7 15:06:27 tgpro1 kernel: [<006d5475>] ? ceph_mdsc_submit_request+0x55/0x55 [ceph] Dec 7 15:06:27 tgpro1 kernel: [<00694d13>] ? ceph_pagelist_append+0xbc/0xf9 [libceph] Dec 7 15:06:27 tgpro1 kernel: [<006d3bb4>] send_mds_reconnect+0x319/0x46a [ceph] Dec 7 15:06:27 tgpro1 kernel: [<0008eb6f>] ? trace_hardirqs_on_caller+0x10b/0x13c Dec 7 15:06:27 tgpro1 kernel: [<0008ebab>] ? trace_hardirqs_on+0xb/0xd Dec 7 15:06:27 tgpro1 kernel: [<006d401f>] ceph_mdsc_handle_map+0x2f2/0x42d [ceph] Dec 7 15:06:27 tgpro1 kernel: [<00c40015>] ? 0xc40014 Dec 7 15:06:27 tgpro1 kernel: [<006be00c>] extra_mon_dispatch+0x18/0x1c [ceph] Dec 7 15:06:27 tgpro1 kernel: [<006963ed>] dispatch+0x4f7/0x52a [libceph] Dec 7 15:06:27 tgpro1 kernel: [<0008ebab>] ? trace_hardirqs_on+0xb/0xd Dec 7 15:06:27 tgpro1 kernel: [<00694459>] con_work+0x14f4/0x16c3 [libceph] Dec 7 15:06:27 tgpro1 kernel: [<0007b331>] process_one_work+0x229/0x397 Dec 7 15:06:27 tgpro1 kernel: [<0007b2e5>] ? process_one_work+0x1dd/0x397 Dec 7 15:06:27 tgpro1 kernel: [<00692f65>] ? ceph_fault+0x262/0x262 [libceph] Dec 7 15:06:27 tgpro1 kernel: [<0007b80e>] worker_thread+0x182/0x2d8 Dec 7 15:06:27 tgpro1 kernel: [<00002b40>] ? setup_sigcontext+0x24/0x24 Dec 7 15:06:27 tgpro1 kernel: [<00002b40>] ? setup_sigcontext+0x24/0x24 Dec 7 15:06:27 tgpro1 kernel: [<0007b68c>] ? rescuer_thread+0x1ed/0x1ed Dec 7 15:06:27 tgpro1 kernel: [<0007e843>] kthread+0x62/0x67 Dec 7 15:06:27 tgpro1 kernel: [<0007e7e1>] ? __init_kthread_worker+0x42/0x42 Dec 7 15:06:27 tgpro1 kernel: [<00540c86>] kernel_thread_helper+0x6/0xd
Actions