Project

General

Profile

Actions

Bug #22652

closed

client: fails to release to revoking Fc

Added by Zheng Yan over 6 years ago. Updated about 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
luminous,jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://pulpito.ceph.com/pdonnell-2018-01-09_21:14:38-multimds-wip-pdonnell-testing-20180109.193634-testing-basic-smithi/

[ 2538.478643] INFO: task fn_anonymous:25446 blocked for more than 120 seconds.
[ 2538.485812]       Not tainted 4.15.0-rc2-ceph-g6105a4a28d07 #1
[ 2538.491775] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2538.499729] fn_anonymous    D    0 25446  25425 0x00000000
[ 2538.499733] Call Trace:
[ 2538.499743]  ? __schedule+0x9be/0xa20
[ 2538.499748]  ? mark_held_locks+0x61/0x80
[ 2538.499751]  ? __lock_page+0xd3/0x140
[ 2538.499753]  schedule+0x67/0x80
[ 2538.499757]  io_schedule+0x12/0x40
[ 2538.499759]  __lock_page+0x113/0x140
[ 2538.499762]  ? page_cache_tree_insert+0x100/0x100
[ 2538.499771]  invalidate_inode_pages2_range+0x125/0x420
[ 2538.499781]  ? find_held_lock+0x90/0xa0
[ 2538.499787]  ? _cond_resched+0x20/0x30
[ 2538.499792]  ? ilookup5+0x67/0xc0
[ 2538.499799]  fuse_reverse_inval_inode+0x68/0x90
[ 2538.499804]  fuse_dev_do_write+0x160/0xe30
[ 2538.499811]  ? find_held_lock+0x31/0xa0
[ 2538.499814]  fuse_dev_write+0x59/0x70
[ 2538.499819]  do_iter_readv_writev+0xed/0x130
[ 2538.499823]  do_iter_write+0x7f/0x180
[ 2538.499825]  vfs_writev+0x9c/0xf0
[ 2538.499832]  ? __fget+0x1a0/0x1c0
[ 2538.499836]  ? do_writev+0x4d/0xd0
[ 2538.499837]  do_writev+0x4d/0xd0
[ 2538.499842]  entry_SYSCALL_64_fastpath+0x23/0x9a
[ 2538.499844] RIP: 0033:0x7f147224efbd
[ 2538.499845] RSP: 002b:00007f146712ed00 EFLAGS: 00000293 ORIG_RAX: 0000000000000014
[ 2538.499847] RAX: ffffffffffffffda RBX: 00005635ee9521a0 RCX: 00007f147224efbd
[ 2538.499849] RDX: 0000000000000002 RSI: 00007f146712ed60 RDI: 0000000000000018
[ 2538.499850] RBP: 00007f147c0a8040 R08: 0000000000000028 R09: 0000000000000003
[ 2538.499851] R10: 00005635eb094470 R11: 0000000000000293 R12: 00007f14737abc40
[ 2538.499852] R13: 00007f1467130c28 R14: 000000000000000a R15: 0000000000000001
[ 2538.499860]
               Showing all locks held in the system:
[ 2538.499865] 1 lock held by khungtaskd/66:
[ 2538.499868]  #0:  (tasklist_lock){.+.+}, at: [<00000000f40de66e>] debug_show_all_locks+0x3f/0x170
[ 2538.499890] 1 lock held by fn_anonymous/25446:
[ 2538.499892]  #0:  (&fc->killsb){.+.+}, at: [<00000000c3570ddc>] fuse_dev_do_write+0x133/0xe30

looks like invalidate_inode_pages2_range() was waiting for a locked page, but the page never got unlocked.

The attached file was generated by 'echo t > /proc/sysrq-tigger; dmesg > dmesg.txt'


Files

dmesg.txt (246 KB) dmesg.txt Zheng Yan, 01/10/2018 10:56 AM

Related issues 2 (0 open2 closed)

Copied to CephFS - Backport #22688: luminous: client: fails to release to revoking FcResolvedPatrick DonnellyActions
Copied to CephFS - Backport #22689: jewel: client: fails to release to revoking FcResolvedPrashant DActions
Actions #1

Updated by Zheng Yan over 6 years ago

I reproduced it locally. it seems like kernel issue. The issue happens only when fuse_use_invalidate_cb is true.

Actions #2

Updated by Sage Weil over 6 years ago

  • Status changed from New to 12
Actions #3

Updated by Zheng Yan over 6 years ago

  • Project changed from Linux kernel client to CephFS
  • Subject changed from fuse_reverse_inval_inode() hangs to client fails to release to revoking Fc

hang fuse_reverse_inval_inode() was caused by hang page writeback.

Actions #4

Updated by Zheng Yan over 6 years ago

  • Status changed from 12 to Fix Under Review
Actions #5

Updated by Patrick Donnelly over 6 years ago

  • Subject changed from client fails to release to revoking Fc to client: fails to release to revoking Fc
  • Status changed from Fix Under Review to Pending Backport
  • Assignee set to Zheng Yan
  • Source set to Development
  • Backport set to luminous,jewel
  • Component(FS) Client added
Actions #6

Updated by Nathan Cutler over 6 years ago

  • Copied to Backport #22688: luminous: client: fails to release to revoking Fc added
Actions #7

Updated by Nathan Cutler over 6 years ago

  • Copied to Backport #22689: jewel: client: fails to release to revoking Fc added
Actions #8

Updated by Nathan Cutler about 6 years ago

  • Status changed from Pending Backport to Resolved
Actions

Also available in: Atom PDF