Support #43809: ceph osd server issues (page allocation failure) - Linux kernel client - Ceph

Actions

Copy link

Support #43809

closed

ceph osd server issues (page allocation failure)

Added by Kenneth Waegeman about 4 years ago. Updated almost 4 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Category:

Target version:

% Done:

Tags:

Reviewed:

Affected Versions:

Ceph - v14.2.0, Ceph - v14.2.1, Ceph - v14.2.2, Ceph - v14.2.3, Ceph - v14.2.4, Ceph - v14.2.5, Ceph - v14.2.6, Ceph - v14.2.7

Description

Hi,

I'm constantly getting these messages on our osd servers:


[Fri Jan 24 12:00:11 2020] warn_alloc_failed: 4748 callbacks suppressed
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd64ca4e>] ? prune_super+0x3e/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d0cae>] shrink_slab+0xae/0x340
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd63f9b1>] ? vmpressure+0x21/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d403a>] do_try_to_free_pages+0x3ca/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd64ca4e>] ? prune_super+0x3e/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d0cae>] shrink_slab+0xae/0x340
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd63f9b1>] ? vmpressure+0x21/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d403a>] do_try_to_free_pages+0x3ca/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd64ca4e>] ? prune_super+0x3e/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d0cae>] shrink_slab+0xae/0x340
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd63f9b1>] ? vmpressure+0x21/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d403a>] do_try_to_free_pages+0x3ca/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd64ca4e>] ? prune_super+0x3e/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d0cae>] shrink_slab+0xae/0x340
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd63f9b1>] ? vmpressure+0x21/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d403a>] do_try_to_free_pages+0x3ca/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd64ca4e>] ? prune_super+0x3e/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d0cae>] shrink_slab+0xae/0x340
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd63f9b1>] ? vmpressure+0x21/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d403a>] do_try_to_free_pages+0x3ca/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd64ca4e>] ? prune_super+0x3e/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d0cae>] shrink_slab+0xae/0x340
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd63f9b1>] ? vmpressure+0x21/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d403a>] do_try_to_free_pages+0x3ca/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd5fe239>] ? __page_check_address+0x99/0x250
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5fe43d>] page_referenced_one+0x4d/0x1b0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd600473>] rmap_walk+0xf3/0x360
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd600793>] page_referenced+0xb3/0x130
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5fe3f0>] ? __page_check_address+0x250/0x250
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd600270>] ? page_get_anon_vma+0xa0/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d2f44>] shrink_active_list+0x1f4/0x370
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d34b1>] shrink_lruvec+0x3f1/0x740
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d3876>] shrink_zone+0x76/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d3d60>] do_try_to_free_pages+0xf0/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd5fe239>] ? __page_check_address+0x99/0x250
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5fe43d>] page_referenced_one+0x4d/0x1b0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd600473>] rmap_walk+0xf3/0x360
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd600793>] page_referenced+0xb3/0x130
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5fe3f0>] ? __page_check_address+0x250/0x250
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd600270>] ? page_get_anon_vma+0xa0/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d2f44>] shrink_active_list+0x1f4/0x370
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d34b1>] shrink_lruvec+0x3f1/0x740
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d3876>] shrink_zone+0x76/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d3d60>] do_try_to_free_pages+0xf0/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] swapper/0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d06e7>] ? bnx2x_set_led+0x157/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd9c1077>] ? cpuidle_enter_state+0x57/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd9c106d>] ? cpuidle_enter_state+0x4d/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd9c11ce>] cpuidle_idle_call+0xde/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd437c6e>] arch_cpu_idle+0xe/0xc0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd500cda>] cpu_startup_entry+0x14a/0x1e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb68c07>] rest_init+0x77/0x80
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe1881cb>] start_kernel+0x450/0x471
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe187b7b>] ? repair_env_string+0x5c/0x5c
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe187120>] ? early_idt_handler_array+0x120/0x120
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe18772f>] x86_64_start_reservations+0x24/0x26
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe187885>] x86_64_start_kernel+0x154/0x177
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4000d5>] start_cpu+0x5/0x14
[Fri Jan 24 12:00:11 2020] swapper/0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d06e7>] ? bnx2x_set_led+0x157/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd9c1077>] ? cpuidle_enter_state+0x57/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd9c106d>] ? cpuidle_enter_state+0x4d/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd9c11ce>] cpuidle_idle_call+0xde/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd437c6e>] arch_cpu_idle+0xe/0xc0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd500cda>] cpu_startup_entry+0x14a/0x1e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb68c07>] rest_init+0x77/0x80
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe1881cb>] start_kernel+0x450/0x471
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe187b7b>] ? repair_env_string+0x5c/0x5c
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe187120>] ? early_idt_handler_array+0x120/0x120
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe18772f>] x86_64_start_reservations+0x24/0x26
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe187885>] x86_64_start_kernel+0x154/0x177
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4000d5>] start_cpu+0x5/0x14
[Fri Jan 24 12:00:16 2020] warn_alloc_failed: 5926 callbacks suppressed
[Fri Jan 24 12:00:16 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:16 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:16 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:16 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]

I'v had set this setting long time ago:

vm.min_free_kbytes = 3145728

I am running 14.2.6 cluster, but I also saw this with earlier versions. I'm also seeing osds going down or even hanging now and then.

Is there something else to configure?

Thanks!!

Actions

Copy link

Updated by Greg Farnum about 4 years ago

Project changed from Ceph to RADOS
Category deleted (~~OSD~~)

Actions

Copy link

Updated by Greg Farnum about 4 years ago

Tracker changed from Bug to Support
Project changed from RADOS to rbd

Oh, you're running kernel clients on the same servers as the OSDs. This is generally asking for a memory allocation deadlock where the kernel tries to flush out dirty pages and that results in the OSD attempting to allocate memory (which is unavailable).

You can improve the odds in various ways such as by increasing the target free memory, but it's never going to be foolproof...

Actions

Copy link