Ceph

[Sun Jun 12 13:32:56 2016] kernel BUG at /build/linux-lts-xenial-7RlTta/linux-lts-xenial-4.4.0/fs/ceph/file.c:1083!
[Sun Jun 12 13:32:56 2016] invalid opcode: 0000 [#1] SMP
[Sun Jun 12 13:32:56 2016] Modules linked in: ceph libceph fscache btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c bonding intel_rapl x86_pkg_temp_thermal intel_powerclamp
coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ipmi_ssif aesni_intel ipmi_devintf aes_x86_64 lrw gf128mul glue_helper dcdbas sb_edac mei_me ablk_helper edac_core mei mxm_wmi cryptd
lpc_ich shpchp 8250_fintek acpi_power_meter ipmi_si mac_hid wmi ipmi_msghandler lp parport tg3 ahci ptp libahci megaraid_sas pps_core fjes
[Sun Jun 12 13:32:56 2016] CPU: 7 PID: 3011089 Comm: python Not tainted 4.4.0-22-generic #40~14.04.1-Ubuntu
[Sun Jun 12 13:32:56 2016] Hardware name: Dell Inc. PowerEdge R730/0WCJNT, BIOS 2.0.1 02/12/2016
[Sun Jun 12 13:32:56 2016] task: ffff880b49fe44c0 ti: ffff88078e6b0000 task.ti: ffff88078e6b0000
[Sun Jun 12 13:32:56 2016] RIP: 0010:[<ffffffffc05ae23b>] [<ffffffffc05ae23b>] ceph_write_iter+0xfcb/0xfd0 [ceph]
[Sun Jun 12 13:32:56 2016] RSP: 0018:ffff88078e6b3cc0 EFLAGS: 00010246
[Sun Jun 12 13:32:56 2016] RAX: ffff88029eb957c8 RBX: 0000000000000000 RCX: ffff88078e6b3db8
[Sun Jun 12 13:32:56 2016] RDX: 0000000000000001 RSI: ffff88029eb957c8 RDI: 0000000000000000
[Sun Jun 12 13:32:56 2016] RBP: ffff88078e6b3e38 R08: 0000000000400000 R09: ffff88078e6b3c58
[Sun Jun 12 13:32:56 2016] R10: 0000000000001855 R11: 0000000000000000 R12: ffff88078e6b3f20
[Sun Jun 12 13:32:56 2016] R13: 000000000000eaed R14: ffff88078e6b3f20 R15: ffff88029eb957c8
[Sun Jun 12 13:32:56 2016] FS: 00007fe3fe865740(0000) GS:ffff88282dec0000(0000) knlGS:0000000000000000
[Sun Jun 12 13:32:56 2016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Sun Jun 12 13:32:56 2016] CR2: 00007fe3fe72f000 CR3: 00000018c984d000 CR4: 00000000001406e0
[Sun Jun 12 13:32:56 2016] Stack:
[Sun Jun 12 13:32:56 2016] ffff88029e9d6550 ffff880cffff4002 0000000000000000 ffff88141d4af000
[Sun Jun 12 13:32:56 2016] 00000d010000bffd ffffffff00000400 0000000000000000 00000d550000000e
[Sun Jun 12 13:32:57 2016] 0000000000000000 000000000000040f 000000000000eaed ffff88029eb95470
[Sun Jun 12 13:32:57 2016] Call Trace:
[Sun Jun 12 13:32:57 2016] [<ffffffffc05ac057>] ? ceph_read_iter+0x127/0x7d0 [ceph]
[Sun Jun 12 13:32:57 2016] [<ffffffff811fd458>] new_sync_write+0x88/0xb0
[Sun Jun 12 13:32:57 2016] [<ffffffff811fd4a7>] __vfs_write+0x27/0x40
[Sun Jun 12 13:32:57 2016] [<ffffffff811fdab2>] vfs_write+0xa2/0x1a0
[Sun Jun 12 13:32:57 2016] [<ffffffff811fd95f>] ? vfs_read+0x7f/0x130
[Sun Jun 12 13:32:57 2016] [<ffffffff811fe7d6>] SyS_write+0x46/0xa0
[Sun Jun 12 13:32:57 2016] [<ffffffff817efcf6>] entry_SYSCALL_64_fastpath+0x16/0x75
[Sun Jun 12 13:32:57 2016] Code: 00 00 00 41 89 d8 4c 89 ea 48 c7 c6 68 0a 5d c0 48 c7 c7 e0 01 5e c0 31 c0 e8 42 a0 e4 c0 e9 47 f4 ff ff 41 89 c5 e9 5d fb ff ff <0f> 0b 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41
[Sun Jun 12 13:32:57 2016] RIP [<ffffffffc05ae23b>] ceph_write_iter+0xfcb/0xfd0 [ceph]
[Sun Jun 12 13:32:57 2016] RSP <ffff88078e6b3cc0>
[Sun Jun 12 13:32:57 2016] ---[ end trace 7972388097b2c541 ]---

OS info:

• Ubuntu 14.04.4
• Btrfs for the underlying FS
• ceph installed using apt source http://download.ceph.com/debian-jewel, version 10.2.1-1trusty
• Kernel from package linux-image-generic-lts-xenial, `uname -a` output:

Linux rndcl12 4.4.0-22-generic #40~14.04.1-Ubuntu SMP Fri May 13 17:27:45 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

We are doing some simple concurrent read/write test on 8 hosts, each with 3 osds. Our test may write to the same file concurrently.

The attached file contains trace after hitting the bug.

We are new to ceph, please let us know if we should provide more information, thanks.

We got this issue and #16242 at the same time