Actions
Bug #3058
closedxfs: possible circular locking dependency detected
Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
Logs: ubuntu@teuthology:/a/teuthology-2012-08-28_02:00:05-regression-testing-testing-basic/10406
2012-08-28T03:23:52.209167-07:00 plana40 kernel: [23405.556763] ========================================= ============= 2012-08-28T03:23:52.209175-07:00 plana40 kernel: [23405.584315] [ INFO: possible circular locking depende ncy detected ] 2012-08-28T03:23:52.209177-07:00 plana40 kernel: [23405.611861] 3.6.0-rc2-ceph-00143-g995fc06 #1 Not tain ted 2012-08-28T03:23:52.209179-07:00 plana40 kernel: [23405.638127] ----------------------------------------- -------------- 2012-08-28T03:23:52.209180-07:00 plana40 kernel: [23405.638129] fill2/7976 is trying to acquire lock: 2012-08-28T03:23:52.209183-07:00 plana40 kernel: [23405.638139] ((&mp->m_flush_work)){+.+.+.}, at: [<fff fffff81072060>] wait_on_work+0x0/0x160 2012-08-28T03:23:52.209184-07:00 plana40 kernel: [23405.638140] 2012-08-28T03:23:52.209186-07:00 plana40 kernel: [23405.638140] but task is already holding lock: 2012-08-28T03:23:52.209191-07:00 plana40 kernel: [23405.638174] (sb_internal#2){.+.+.+}, at: [<ffffffffa 03afe5d>] xfs_trans_alloc+0x2d/0x50 [xfs] 2012-08-28T03:23:52.209192-07:00 plana40 kernel: [23405.638175] 2012-08-28T03:23:52.209194-07:00 plana40 kernel: [23405.638175] which lock already depends on the new loc k. 2012-08-28T03:23:52.209195-07:00 plana40 kernel: [23405.638175] 2012-08-28T03:23:52.209196-07:00 plana40 kernel: [23405.638175] 2012-08-28T03:23:52.209197-07:00 plana40 kernel: [23405.638175] the existing dependency chain (in reverse order) is: 2012-08-28T03:23:52.209199-07:00 plana40 kernel: [23405.638179] 2012-08-28T03:23:52.209200-07:00 plana40 kernel: [23405.638179] -> #1 (sb_internal#2){.+.+.+}: 2012-08-28T03:23:52.209204-07:00 plana40 kernel: [23405.638183] [<ffffffff810b2c82>] lock_acquire+ 0xa2/0x140 2012-08-28T03:23:52.209207-07:00 plana40 kernel: [23405.638186] [<ffffffff816318eb>] mutex_lock_ne sted+0x4b/0x320 2012-08-28T03:23:52.209221-07:00 plana40 kernel: [23405.638210] [<ffffffffa03aee89>] xfs_icsb_modi fy_counters+0x119/0x1b0 [xfs] 2012-08-28T03:23:52.209224-07:00 plana40 kernel: [23405.638228] [<ffffffffa0363346>] xfs_reserve_b locks+0x96/0x170 [xfs] 2012-08-28T03:23:52.209226-07:00 plana40 kernel: [23405.638252] [<ffffffffa03aec75>] xfs_unmountfs +0x95/0x190 [xfs] 2012-08-28T03:23:52.209229-07:00 plana40 kernel: [23405.638268] [<ffffffffa036cd95>] xfs_fs_put_su per+0x25/0x70 [xfs] 2012-08-28T03:23:52.209231-07:00 plana40 kernel: [23405.638273] [<ffffffff8117de12>] generic_shutdown_super+0x62/0xf0 2012-08-28T03:23:52.209233-07:00 plana40 kernel: [23405.638276] [<ffffffff8117ded0>] kill_block_super+0x30/0x80 2012-08-28T03:23:52.209237-07:00 plana40 kernel: [23405.638279] [<ffffffff8117e1a5>] deactivate_locked_super+0x45/0x70 2012-08-28T03:23:52.209239-07:00 plana40 kernel: [23405.638283] [<ffffffff8117ee4e>] deactivate_super+0x4e/0x70 2012-08-28T03:23:52.209241-07:00 plana40 kernel: [23405.638287] [<ffffffff8119b1d6>] mntput_no_expire+0x106/0x160 2012-08-28T03:23:52.209243-07:00 plana40 kernel: [23405.638289] [<ffffffff8119c1fe>] sys_umount+0x6e/0x3b0 2012-08-28T03:23:52.209245-07:00 plana40 kernel: [23405.638293] [<ffffffff8163d569>] system_call_fastpath+0x16/0x1b 2012-08-28T03:23:52.209246-07:00 plana40 kernel: [23405.638296] 2012-08-28T03:23:52.209248-07:00 plana40 kernel: [23405.638296] -> #0 ((&mp->m_flush_work)){+.+.+.}: 2012-08-28T03:23:52.209252-07:00 plana40 kernel: [23405.638298] [<ffffffff810b25e8>] __lock_acquire+0x1ac8/0x1b90 2012-08-28T03:23:52.209254-07:00 plana40 kernel: [23405.638301] [<ffffffff810b2c82>] lock_acquire+0xa2/0x140 2012-08-28T03:23:52.209256-07:00 plana40 kernel: [23405.638304] [<ffffffff810720a1>] wait_on_work+0x41/0x160 2012-08-28T03:23:52.209258-07:00 plana40 kernel: [23405.638307] [<ffffffff81072203>] flush_work_sync+0x43/0x90 2012-08-28T03:23:52.209260-07:00 plana40 kernel: [23405.638323] [<ffffffffa036ec7f>] xfs_flush_inodes+0x2f/0x40 [xfs] 2012-08-28T03:23:52.209262-07:00 plana40 kernel: [23405.638341] [<ffffffffa0371d2e>] xfs_create+0x3be/0x640 [xfs] 2012-08-28T03:23:52.209264-07:00 plana40 kernel: [23405.638357] [<ffffffffa036888f>] xfs_vn_mknod+0x8f/0x1c0 [xfs] 2012-08-28T03:23:52.209266-07:00 plana40 kernel: [23405.638372] [<ffffffffa03689f3>] xfs_vn_create+0x13/0x20 [xfs] 2012-08-28T03:23:52.209270-07:00 plana40 kernel: [23405.638375] [<ffffffff8118aeb5>] vfs_create+0xb5/0x120 2012-08-28T03:23:52.209272-07:00 plana40 kernel: [23405.638378] [<ffffffff8118bcc0>] do_last+0xda0/0xf00 2012-08-28T03:23:52.209274-07:00 plana40 kernel: [23405.638380] [<ffffffff8118bed3>] path_openat+0 xb3/0x4c0 2012-08-28T03:23:52.209276-07:00 plana40 kernel: [23405.638383] [<ffffffff8118c6f2>] do_filp_open+0x42/0xa0 2012-08-28T03:23:52.209278-07:00 plana40 kernel: [23405.638386] [<ffffffff8117b040>] do_sys_open+0x100/0x1e0 2012-08-28T03:23:52.209279-07:00 plana40 kernel: [23405.638389] [<ffffffff8117b141>] sys_open+0x21/0x30 2012-08-28T03:23:52.209281-07:00 plana40 kernel: [23405.638391] [<ffffffff8163d569>] system_call_fastpath+0x16/0x1b 2012-08-28T03:23:52.209285-07:00 plana40 kernel: [23405.638392] 2012-08-28T03:23:52.209286-07:00 plana40 kernel: [23405.638392] other info that might help us debug this: 2012-08-28T03:23:52.209287-07:00 plana40 kernel: [23405.638392] 2012-08-28T03:23:52.209289-07:00 plana40 kernel: [23405.638393] Possible unsafe locking scenario: 2012-08-28T03:23:52.209290-07:00 plana40 kernel: [23405.638393] 2012-08-28T03:23:52.209292-07:00 plana40 kernel: [23405.638394] CPU0 CPU1 2012-08-28T03:23:52.209293-07:00 plana40 kernel: [23405.638394] ---- ---- 2012-08-28T03:23:52.209294-07:00 plana40 kernel: [23405.638396] lock(sb_internal#2); 2012-08-28T03:23:52.209298-07:00 plana40 kernel: [23405.638398] lock((&mp->m_flush_work)); 2012-08-28T03:23:52.209300-07:00 plana40 kernel: [23405.638400] lock(sb_internal#2); 2012-08-28T03:23:52.209301-07:00 plana40 kernel: [23405.638402] lock((&mp->m_flush_work)); 2012-08-28T03:23:52.209302-07:00 plana40 kernel: [23405.638402] 2012-08-28T03:23:52.209304-07:00 plana40 kernel: [23405.638402] *** DEADLOCK *** 2012-08-28T03:23:52.209305-07:00 plana40 kernel: [23405.638402] 2012-08-28T03:23:52.209306-07:00 plana40 kernel: [23405.638404] 3 locks held by fill2/7976: 2012-08-28T03:23:52.209310-07:00 plana40 kernel: [23405.638409] #0: (sb_writers#14){.+.+.+}, at: [<ffffffff8119b5b4>] mnt_want_write+0x24/0x50 2012-08-28T03:23:52.209313-07:00 plana40 kernel: [23405.638414] #1: (&type->i_mutex_dir_key#9){+.+.+.}, at: [<ffffffff8118b22b>] do_last+0x30b/0xf00 2012-08-28T03:23:52.209316-07:00 plana40 kernel: [23405.638440] #2: (sb_internal#2){.+.+.+}, at: [<ffffffffa03afe5d>] xfs_trans_alloc+0x2d/0x50 [xfs] 2012-08-28T03:23:52.209317-07:00 plana40 kernel: [23405.638441] 2012-08-28T03:23:52.209318-07:00 plana40 kernel: [23405.638441] stack backtrace: 2012-08-28T03:23:52.209359-07:00 plana40 kernel: [23405.638443] Pid: 7976, comm: fill2 Not tainted 3.6.0-rc2-ceph-00143-g995fc06 #1 2012-08-28T03:23:52.209360-07:00 plana40 kernel: [23405.638444] Call Trace: 2012-08-28T03:23:52.209364-07:00 plana40 kernel: [23405.638448] [<ffffffff8162a77c>] print_circular_bug+ 0x1fb/0x20c 2012-08-28T03:23:52.209367-07:00 plana40 kernel: [23405.638451] [<ffffffff810b25e8>] __lock_acquire+0x1ac8/0x1b90 2012-08-28T03:23:52.209369-07:00 plana40 kernel: [23405.638455] [<ffffffff81050500>] ? __mmdrop+0x60/0x90 2012-08-28T03:23:52.209371-07:00 plana40 kernel: [23405.638459] [<ffffffff8108494a>] ? finish_task_switch+0x10a/0x110 2012-08-28T03:23:52.209373-07:00 plana40 kernel: [23405.638463] [<ffffffff81072060>] ? busy_worker_rebind_fn+0x100/0x100 2012-08-28T03:23:52.209374-07:00 plana40 kernel: [23405.638465] [<ffffffff810b2c82>] lock_acquire+0xa2/0x140 2012-08-28T03:23:52.209376-07:00 plana40 kernel: [23405.638468] [<ffffffff81072060>] ? busy_worker_rebind_fn+0x100/0x100 2012-08-28T03:23:52.209378-07:00 plana40 kernel: [23405.638472] [<ffffffff81634c30>] ? _raw_spin_unlock_irq+0x30/0x40 2012-08-28T03:23:52.209382-07:00 plana40 kernel: [23405.638475] [<ffffffff810720a1>] wait_on_work+0x41/0x160 2012-08-28T03:23:52.209384-07:00 plana40 kernel: [23405.638477] [<ffffffff81072060>] ? busy_worker_rebind_fn+0x100/0x100 2012-08-28T03:23:52.209386-07:00 plana40 kernel: [23405.638480] [<ffffffff810710a8>] ? start_flush_work+0x108/0x180 2012-08-28T03:23:52.209388-07:00 plana40 kernel: [23405.638483] [<ffffffff81634e5f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80 2012-08-28T03:23:52.209391-07:00 plana40 kernel: [23405.638486] [<ffffffff81072203>] flush_work_sync+0x43/0x90 2012-08-28T03:23:52.209393-07:00 plana40 kernel: [23405.638488] [<ffffffff810b379d>] ? trace_hardirqs_on+0xd/0x10 2012-08-28T03:23:52.209394-07:00 plana40 kernel: [23405.638491] [<ffffffff810706c4>] ? __queue_work+0xe4/0x3b0 2012-08-28T03:23:52.209401-07:00 plana40 kernel: [23405.638509] [<ffffffffa036ec7f>] xfs_flush_inodes+0x2f/0x40 [xfs] 2012-08-28T03:23:52.209403-07:00 plana40 kernel: [23405.638527] [<ffffffffa0371d2e>] xfs_create+0x3be/0x640 [xfs] 2012-08-28T03:23:52.209405-07:00 plana40 kernel: [23405.638529] [<ffffffff81192254>] ? d_rehash+0x24/0x40 2012-08-28T03:23:52.209407-07:00 plana40 kernel: [23405.638545] [<ffffffffa036888f>] xfs_vn_mknod+0x8f/0x1c0 [xfs] 2012-08-28T03:23:52.209409-07:00 plana40 kernel: [23405.638561] [<ffffffffa03689f3>] xfs_vn_create+0x13/ 0x20 [xfs] 2012-08-28T03:23:52.209411-07:00 plana40 kernel: [23405.638564] [<ffffffff8118aeb5>] vfs_create+0xb5/0x120 2012-08-28T03:23:52.209413-07:00 plana40 kernel: [23405.638567] [<ffffffff8118bcc0>] do_last+0xda0/0xf00 2012-08-28T03:23:52.209414-07:00 plana40 kernel: [23405.638570] [<ffffffff8118bed3>] path_openat+0xb3/0x4c0 2012-08-28T03:23:52.209418-07:00 plana40 kernel: [23405.638573] [<ffffffff8118c6f2>] do_filp_open+0x42/0xa0 2012-08-28T03:23:52.209420-07:00 plana40 kernel: [23405.638577] [<ffffffff8132babd>] ? do_raw_spin_unlock+0x5d/0xb0 2012-08-28T03:23:52.209422-07:00 plana40 kernel: [23405.638579] [<ffffffff81634c6b>] ? _raw_spin_unlock+0x2b/0x40 2012-08-28T03:23:52.209424-07:00 plana40 kernel: [23405.638582] [<ffffffff81199a22>] ? alloc_fd+0xd2/0x120 2012-08-28T03:23:52.209426-07:00 plana40 kernel: [23405.638585] [<ffffffff8117b040>] do_sys_open+0x100/0x1e0 2012-08-28T03:23:52.209428-07:00 plana40 kernel: [23405.638588] [<ffffffff8117b141>] sys_open+0x21/0x30 2012-08-28T03:23:52.209429-07:00 plana40 kernel: [23405.638590] [<ffffffff8163d569>] system_call_fastpath+0x16/0x1b 2012-08-28T03:23:56.017269-07:00 plana40 kernel: [23409.414754] XFS (rbd2): Mounting Filesystem 2012-08-28T03:23:56.322868-07:00 plana40 kernel: [23409.720118] XFS (rbd2): Ending clean mount 2012-08-28T03:24:09.726925-07:00 plana40 kernel: [23423.101047] XFS (rbd2): Mounting Filesystem 2012-08-28T03:24:10.069138-07:00 plana40 kernel: [23423.443036] XFS (rbd2): Ending clean mount 2012-08-28T03:24:25.706773-07:00 plana40 kernel: [23439.053799] XFS (rbd2): Mounting Filesystem 2012-08-28T03:24:25.967883-07:00 plana40 kernel: [23439.314846] XFS (rbd2): Ending clean mount 2012-08-28T03:24:41.274859-07:00 plana40 kernel: [23454.595433] XFS (rbd1): Mounting Filesystem 2012-08-28T03:24:41.581621-07:00 plana40 kernel: [23454.902059] XFS (rbd1): Ending clean mount 2012-08-28T03:24:44.026073-07:00 plana40 kernel: [23457.341122] XFS (rbd2): unknown mount option [foobar]. 2012-08-28T03:24:44.333401-07:00 plana40 kernel: [23457.622354] XFS (rbd2): Filesystem has duplicate UUID 53a04ae9-103d-4894-981f-892ad659f533 - can't mount 2012-08-28T03:24:44.859334-07:00 plana40 kernel: [23458.173026] XFS (rbd1): Mounting Filesystem 2012-08-28T03:24:45.142515-07:00 plana40 kernel: [23458.456009] XFS (rbd1): Ending clean mount 2012-08-28T03:24:46.582102-07:00 plana40 kernel: [23459.893012] XFS (rbd2): Mounting Filesystem 2012-08-28T03:24:46.837651-07:00 plana40 kernel: [23460.148478] XFS (rbd2): Ending clean mount ubuntu@teuthology:/a/teuthology-2012-08-28_02:00:05-regression-testing-testing-basic/10406$ cat config.yaml kernel: &id001 kdb: true sha1: 995fc068ddf675260098c60591989bf2ee184338 nuke-on-error: true overrides: ceph: fs: btrfs log-whitelist: - slow request sha1: c03ca95d235c9a072dcd8a77ad5274a52e93ae30 workunit: sha1: c03ca95d235c9a072dcd8a77ad5274a52e93ae30 roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mds.a - osd.3 - osd.4 - osd.5 - - client.0 targets: ubuntu@plana39.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo+Kh24vRxeTQ6/n5PIIGuxrPHPRO/xMQlwoLHi7mR01cIXJMG5wet7mp2om3/5SZSDcLBHduDKrdWL142Sg5fC0zZPUggbxS7nz/UCjYBzMsOtHEUAU5Gs0KFopOCHXNEveK95ezsroMAD5+jS/IEpiooYCkrR3H+NSvUU0Ae352PlXqV0vamkYzyQyEMmhFE50ALhUXbKMve3d2mxJee5sqVZSBmQTbze9RKUA96t9iiwiheflXbN1i9WHlbBOIue5pZ5fM3/vqPWgaShfFpa0pT56QKJfjyFcDeCLOislo23E5qKAJOi5vn5BoYVtG3niNQpt/YbYGfDEHVeqt9 ubuntu@plana40.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEwyNlwC9Utqf3PCjL2JR4wwDkzpdEJuW93DOW82vYVisYEGod454JwXeNkjqzTUk6tXeRoUM9f/C6sZS3LFgHcMYt6m0sxP8DC4qU+q0YxCw9zLY8bXKe4DDjijM62h/SnyqyOWIh9amGT7wRwZEHBV1BKvZbNxQIJ7ESkuKsk/tJfWKhq7dSw6E/+MZ4yQtXvTyaJ3pK96Hq2uoUkawv+FxXBrzG3FtTTYA8gqA1SIiV3erEIQuBK/WD74i5yK4rwpfGTo7jNc0V6wrwO1BKFj/OGjSC+2LSAkBgf8WLe6UL/dHr3bBEyzm0V4xMf5Iqb8JGvkaXNEfbFqzKC2Wv ubuntu@plana80.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjbUncT43IZIcpSlFXnWDaagYNdnaLTfgsN8TKi3J2QYOB1BlMpaSRBByXr5MW2ZGuWrB77USjJLdRW/feuwtAjDbwc7y5woxcxn1u9eqPh4RyQWqyCmKWvi3GpdM4/4vh+L+7X5rNzJPStTjiPXCxHK+stPGCk9pg+J+KKg/GlJ1Lwx1DeWetdForGqEVJrTSSWj8RM+Nyw/V+c+t2d6gW1SzjY2NvaDdVHfduM+8X9F9aTf3KLitnIvNQWXzfqoEDNPSwrrO2uvTDyqpFkW6HLpZDnWm+io8Y9P7XMrEhWCTlQX2SHeQv/BhaA1btYAw7qfLC9n8RbeXLlQnc5yn tasks: - internal.lock_machines: 3 - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.check_conflict: null - kernel: *id001 - internal.base: null - internal.archive: null - internal.coredump: null - internal.syslog: null - internal.timer: null - chef: null - clock: null - ceph: null - rbd.xfstests: null
Actions