Ceph » Linux kernel client

In a newly setup ceph system, while doing some basic performance test, we hit a kernel bug with the following trace:

Jun 12 12:51:54 rndcl15 kernel: [511788.438303] ------------[ cut here ]------------
Jun 12 12:51:54 rndcl15 kernel: [511788.439744] kernel BUG at /build/linux-lts-xenial-7RlTta/linux-lts-xenial-4.4.0/fs/ceph/addr.c:92!
Jun 12 12:51:54 rndcl15 kernel: [511788.442541] invalid opcode: 0000 [#1] SMP 
Jun 12 12:51:54 rndcl15 kernel: [511788.443876] Modules linked in: ceph libceph fscache btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c bonding intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ipmi_ssif ipmi_devintf aesni_intel aes_x86_64 lrw gf128mul glue_helper ipmi_si mei_me sb_edac 8250_fintek ablk_helper ipmi_msghandler mei edac_core cryptd dcdbas shpchp acpi_power_meter mxm_wmi lpc_ich mac_hid wmi lp parport tg3 ahci ptp libahci megaraid_sas pps_core fjes
Jun 12 12:51:54 rndcl15 kernel: [511788.459712] CPU: 6 PID: 3076691 Comm: python Not tainted 4.4.0-22-generic #40~14.04.1-Ubuntu
Jun 12 12:51:54 rndcl15 kernel: [511788.462345] Hardware name: Dell Inc. PowerEdge R730/0WCJNT, BIOS 2.0.1 02/12/2016
Jun 12 12:51:54 rndcl15 kernel: [511788.464681] task: ffff8813af586040 ti: ffff880186b60000 task.ti: ffff880186b60000
Jun 12 12:51:54 rndcl15 kernel: [511788.467017] RIP: 0010:[<ffffffffc060af1f>]  [<ffffffffc060af1f>] ceph_set_page_dirty+0x1bf/0x240 [ceph]
Jun 12 12:51:54 rndcl15 kernel: [511788.469987] RSP: 0018:ffff880186b63b68  EFLAGS: 00010246
Jun 12 12:51:54 rndcl15 kernel: [511788.471644] RAX: 0000000000000000 RBX: ffffea002a7ba180 RCX: 0000000000001000
Jun 12 12:51:54 rndcl15 kernel: [511788.473871] RDX: 0000000000000000 RSI: 0000000000001000 RDI: ffff880187b605b0
Jun 12 12:51:54 rndcl15 kernel: [511788.476098] RBP: ffff880186b63bc8 R08: 0000000000001000 R09: ffffea002a7ba180
Jun 12 12:51:54 rndcl15 kernel: [511788.478325] R10: 0000000000000000 R11: ffffffff81ac9018 R12: ffff880187b60908
Jun 12 12:51:54 rndcl15 kernel: [511788.590836] R13: ffff880187b60a70 R14: ffff880187b605b0 R15: 0000000000000000
Jun 12 12:51:54 rndcl15 kernel: [511788.707503] FS:  00007f83604c1740(0000) GS:ffff88142e6c0000(0000) knlGS:0000000000000000
Jun 12 12:51:54 rndcl15 kernel: [511788.824543] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 12 12:51:54 rndcl15 kernel: [511788.883948] CR2: 00007f745eb3dce0 CR3: 0000001aa3bae000 CR4: 00000000001406e0
Jun 12 12:51:54 rndcl15 kernel: [511789.000942] Stack:
Jun 12 12:51:54 rndcl15 kernel: [511789.057598]  000010000000000f 0000000000001000 0000000000001000 ffffffff8118311a
Jun 12 12:51:54 rndcl15 kernel: [511789.171365]  ffffffffc061080d ffff880186b63c78 0000000000001000 ffffea002a7ba180
Jun 12 12:51:54 rndcl15 kernel: [511789.285376]  0000000000001000 ffff880187b60908 0000000000000000 0000000000000000
Jun 12 12:51:54 rndcl15 kernel: [511789.399598] Call Trace:
Jun 12 12:51:54 rndcl15 kernel: [511789.455153]  [<ffffffff8118311a>] ? pagecache_get_page+0x8a/0x1c0
Jun 12 12:51:54 rndcl15 kernel: [511789.511495]  [<ffffffffc061080d>] ? try_get_cap_refs+0x1fd/0x530 [ceph]
Jun 12 12:51:54 rndcl15 kernel: [511789.567333]  [<ffffffff8118cdcd>] set_page_dirty+0x3d/0x70
Jun 12 12:51:54 rndcl15 kernel: [511789.622068]  [<ffffffffc060b1c6>] ceph_write_end+0x56/0x170 [ceph]
Jun 12 12:51:54 rndcl15 kernel: [511789.675957]  [<ffffffff813e16a0>] ? iov_iter_copy_from_user_atomic+0x90/0x220
Jun 12 12:51:54 rndcl15 kernel: [511789.781431]  [<ffffffff81182432>] generic_perform_write+0x102/0x1a0
Jun 12 12:51:54 rndcl15 kernel: [511789.835486]  [<ffffffffc0608017>] ceph_write_iter+0xda7/0xfd0 [ceph]
Jun 12 12:51:54 rndcl15 kernel: [511789.888718]  [<ffffffffc0606057>] ? ceph_read_iter+0x127/0x7d0 [ceph]
Jun 12 12:51:54 rndcl15 kernel: [511789.941084]  [<ffffffff81191467>] ? lru_cache_add_active_or_unevictable+0x27/0x90
Jun 12 12:51:54 rndcl15 kernel: [511790.043021]  [<ffffffff811fd458>] new_sync_write+0x88/0xb0
Jun 12 12:51:54 rndcl15 kernel: [511790.094155]  [<ffffffff811fd4a7>] __vfs_write+0x27/0x40
Jun 12 12:51:54 rndcl15 kernel: [511790.143986]  [<ffffffff811fdab2>] vfs_write+0xa2/0x1a0
Jun 12 12:51:54 rndcl15 kernel: [511790.192601]  [<ffffffff811fd95f>] ? vfs_read+0x7f/0x130
Jun 12 12:51:54 rndcl15 kernel: [511790.239842]  [<ffffffff811fe7d6>] SyS_write+0x46/0xa0
Jun 12 12:51:54 rndcl15 kernel: [511790.286042]  [<ffffffff817efcf6>] entry_SYSCALL_64_fastpath+0x16/0x75
Jun 12 12:51:54 rndcl15 kernel: [511790.331797] Code: 10 49 8b 55 00 48 89 f9 48 c7 c6 e0 bc 62 c0 48 c7 c7 d0 ac 63 c0 31 c0 e8 5f d3 de c0 e9 84 fe ff ff 0f 0b b8 01 00 00 00 eb ca <0f> 0b be 7d 00 00 00 48 c7 c7 b0 b5 62 c0 89 45 d0 e8 fb 29 a7 
Jun 12 12:51:54 rndcl15 kernel: [511790.470904] RIP  [<ffffffffc060af1f>] ceph_set_page_dirty+0x1bf/0x240 [ceph]
Jun 12 12:51:54 rndcl15 kernel: [511790.516994]  RSP <ffff880186b63b68>
Jun 12 12:51:54 rndcl15 kernel: [511790.628253] ---[ end trace 997f54ad6f4952b1 ]---

OS info:

• Ubuntu 14.04.4
• Btrfs for the underlying FS
• ceph installed using apt source http://download.ceph.com/debian-jewel, version 10.2.1-1trusty
• Kernel from package linux-image-generic-lts-xenial, `uname -a` output:
  

  Linux rndcl15 4.4.0-22-generic #40~14.04.1-Ubuntu SMP Fri May 13 17:27:45 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  

We are doing some simple concurrent read/write test on 8 hosts, each with 3 osds. Our test may write to the same file concurrently.

The attached file contains trace after hitting the bug.
We are new to ceph, please let us know if we should provide more information, thanks.