Project

General

Profile

Bug #40862

kclient: crashed after evicted twice

Added by Erqi Chen over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
fs/ceph
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

After kernel mount client evicted twice due to not responded to mds cap revoke, it's crashed.
kernel version is 3.10 with latest patch ported.

vmcore-dmesg.txt
[4058680.824852] libceph: mds0 10.48.148.42:6800 socket closed (con state OPEN)
[4058683.098344] libceph: mds0 10.48.148.42:6800 connection reset
[4058683.098634] libceph: reset on mds0
[4058683.098635] ceph: mds0 closed our session
[4058683.098637] ceph: mds0 reconnect start
[4058683.106593] ceph: mds0 reconnect denied
[4058683.106608] ceph: dropping dirty XxFw state for ffff88262e694b90 1101208305803
[4058683.107115] libceph: mds0 10.48.148.42:6800 socket closed (con state NEGOTIATING)
[4058710.463132] ceph: get_quota_realm: ino (1001a35bfb1.fffffffffffffffe) null i_snap_realm
[4058710.463672] ceph: get_quota_realm: ino (1001a35bfb1.fffffffffffffffe) null i_snap_realm
[4058717.231387] libceph: mds0 10.48.148.42:6800 socket closed (con state OPEN)
[4058717.233585] libceph: mds0 10.48.148.42:6800 connection reset
[4058717.233915] libceph: reset on mds0
[4058717.233917] ceph: mds0 closed our session
[4058717.233918] ceph: mds0 reconnect start
[4058717.236841] ceph: mds0 reconnect denied
[4058717.236846] ceph: dropping dirty+flushing XxFw state for ffff88262e6bf280 1101208305806
[4058717.236849] ceph: dropping dirty+flushing Fw state for ffff882329bbd6b0 1101208305807
[4058717.237776] libceph: mds0 10.48.148.42:6800 socket closed (con state NEGOTIATING)
[4058749.968287] ------------[ cut here ]------------
[4058749.968581] kernel BUG at fs/ceph/addr.c:82!
[4058749.968807] invalid opcode: 0000 [#1] SMP
[4058749.969011] Modules linked in: kpatch_ceph_request(OEK) kpatch(OE) ipmi_watchdog ipmi_poweroff nvidia_modeset(POE) nvidia_uvm(OE) nvidia(POE) tcp_diag inet_diag veth ceph libceph dns_resolver br_netfilter bridge stp llc overlay(T) nf_conntrack_netlink xt_statistic xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 xt_mark xt_comment xt_conntrack ip_set nfnetlink xt_addrtype iptable_nat nf_nat_ipv4 nf_nat iptable_filter nf_conntrack_ipv4 nf_defrag_ipv4 vxlan ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6_udp_tunnel nf_conntrack udp_tunnel libcrc32c binfmt_misc sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support mei_me mei lpc_ich pcspkr
[4058749.970945] i2c_i801 sg ioatdma ipmi_si shpchp ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci libata ixgbe igb crct10dif_pclmul crct10dif_common crc32c_intel mdio ptp pps_core i2c_algo_bit i2c_core dca megaraid_sas mxm_wmi wmi
[4058749.972736] CPU: 31 PID: 14668 Comm: python Tainted: P OE K------------ T 3.10.0-1.0.2.el7.x86_64 #1
[4058749.973248] Hardware name: Supermicro SYS-4028GR-TR2/X10DRG-O+-CPU, BIOS 2.0a 04/24/2017
[4058749.973775] task: ffff881f70ea0fd0 ti: ffff881fd3a40000 task.ti: ffff881fd3a40000
[4058749.974342] RIP: 0010:[<ffffffffc073687e>] [<ffffffffc073687e>] ceph_set_page_dirty+0x1ce/0x250 [ceph]
[4058749.974924] RSP: 0018:ffff881fd3a43b08 EFLAGS: 00010246
[4058749.975499] RAX: 002fffff00000039 RBX: ffffea0043e429c0 RCX: 0000000001000000
[4058749.976093] RDX: ffffffff81226bd0 RSI: 0000000000400000 RDI: ffffea0043e429c0
[4058749.976698] RBP: ffff881fd3a43b68 R08: 0000000000001000 R09: ffffea0043e429c0
[4058749.977311] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000001000
[4058749.977963] R13: ffff882329bbd800 R14: 0000000000000000 R15: 0000000000001000
[4058749.978587] FS: 00007f601ef33700(0000) GS:ffff881fffa40000(0000) knlGS:0000000000000000
[4058749.979233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4058749.979884] CR2: 00007f5ea4327928 CR3: 0000003fb0e5a000 CR4: 00000000003407e0
[4058749.980553] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[4058749.981235] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[4058749.981947] Stack:
[4058749.982629] 0000100029bbd800 0000000000001000 0000000000401000 ffff881fd3a43b38
[4058749.983350] ffffffff8117f712 ffffea0043e429c0 ffff881fd3a43b78 ffffea0043e429c0
[4058749.984080] 0000000000001000 ffff882329bbd6b0 0000000000000000 0000000000001000
[4058749.984823] Call Trace:
[4058749.985570] [<ffffffff8117f712>] ? wait_for_stable_page+0x12/0x40
[4058749.986393] [<ffffffff8117f687>] set_page_dirty+0x37/0x60
[4058749.987169] [<ffffffffc0736403>] ceph_write_end+0xd3/0x140 [ceph]
[4058749.987960] [<ffffffff81174599>] generic_file_buffered_write+0x189/0x2a0
[4058749.988757] [<ffffffffc07337ca>] ceph_aio_write+0xb0a/0xd40 [ceph]
[4058749.989562] [<ffffffff811833ee>] ? lru_cache_add+0xe/0x10
[4058749.990414] [<ffffffff81187c22>] ? putback_lru_page+0x52/0xb0
[4058749.991234] [<ffffffff811d88a7>] ? migrate_pages+0x2d7/0x7a0
[4058749.992067] [<ffffffff811d5f00>] ? migrate_vma_collect.constprop.51+0xe0/0xe0
[4058749.992919] [<ffffffff811a2909>] ? do_numa_page+0x159/0x1e0
[4058749.993766] [<ffffffff811f1c8d>] do_sync_write+0x8d/0xd0
[4058749.994631] [<ffffffff811f274d>] vfs_write+0xbd/0x1e0
[4058749.995453] [<ffffffff811f355f>] SyS_write+0x7f/0xe0
[4058749.996285] [<ffffffff81692f09>] system_call_fastpath+0x16/0x1b
[4058749.997102] Code: c0 e9 9f fe ff ff 4c 8b 47 10 49 8b 55 00 48 89 f9 48 c7 c6 f8 c5 75 c0 48 c7 c7 70 d0 76 c0 31 c0 e8 07 58 c0 c0 e9 6c fe ff ff <0f> 0b 0f 0b be 7c 00 00 00 48 c7 c7 33 51 76 c0 89 45 d0 e8 da
[4058749.998892] RIP [<ffffffffc073687e>] ceph_set_page_dirty+0x1ce/0x250 [ceph]
[4058749.999759] RSP <ffff881fd3a43b08>

History

#1 Updated by Patrick Donnelly over 1 year ago

  • Project changed from Ceph to Linux kernel client
  • Subject changed from ceph client is crashed after evicted twice to kclient: crashed after evicted twice
  • Assignee set to Jeff Layton
  • Start date deleted (07/22/2019)
  • Source set to Community (user)

#2 Updated by Jeff Layton over 1 year ago

It oopsed because of this in ceph_set_page_dirty:

        if (PageDirty(page)) {
                dout("%p set_page_dirty %p idx %lu -- already dirty\n",
                     mapping->host, page, page->index);
                BUG_ON(!PagePrivate(page));          <<<< CRASH HERE
                return 0;
        }

We clear out the PagePrivate bit (and associated private info) when the page is invalidated. I suspect that this page got invalidated, but the dirty bit was left intact. We then tried to redirty the page. We probably ought to clear the dirty bit (and clean up the accounting) when the page is invalidated.

It looks like this can happen in ceph_writepages_start in certain situations. We call invalidatepage directly, but don't clear the dirty bit. There may be other ways that this can happen as well.

Do we have a reliable reproducer for this?

#3 Updated by Erqi Chen over 1 year ago

Jeff Layton wrote:

It oopsed because of this in ceph_set_page_dirty:

[...]

We clear out the PagePrivate bit (and associated private info) when the page is invalidated. I suspect that this page got invalidated, but the dirty bit was left intact. We then tried to redirty the page. We probably ought to clear the dirty bit (and clean up the accounting) when the page is invalidated.

It looks like this can happen in ceph_writepages_start in certain situations. We call invalidatepage directly, but don't clear the dirty bit. There may be other ways that this can happen as well.

Do we have a reliable reproducer for this?

Yes, invalidatepage in ceph_writepages_start may cause this issue, and I have proposed a patch to ceph-devel group.
It happened only once in our situation when client failed to response mds's cap revoke request.

#4 Updated by Jeff Layton over 1 year ago

Patch merged into ceph-client/testing branch:

https://marc.info/?l=ceph-devel&m=156393518802502&w=2

#5 Updated by Ilya Dryomov over 1 year ago

  • Category set to fs/ceph
  • Status changed from New to Pending Backport

#6 Updated by Ilya Dryomov over 1 year ago

  • Status changed from Pending Backport to Resolved

In 4.19.69 and 5.2.11.

Also available in: Atom PDF