Ceph » Linux kernel client

Hi,

I'm constantly getting these messages on our osd servers:

[Fri Jan 24 12:00:11 2020] warn_alloc_failed: 4748 callbacks suppressed
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd64ca4e>] ? prune_super+0x3e/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d0cae>] shrink_slab+0xae/0x340
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd63f9b1>] ? vmpressure+0x21/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d403a>] do_try_to_free_pages+0x3ca/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd64ca4e>] ? prune_super+0x3e/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d0cae>] shrink_slab+0xae/0x340
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd63f9b1>] ? vmpressure+0x21/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d403a>] do_try_to_free_pages+0x3ca/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd64ca4e>] ? prune_super+0x3e/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d0cae>] shrink_slab+0xae/0x340
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd63f9b1>] ? vmpressure+0x21/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d403a>] do_try_to_free_pages+0x3ca/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd64ca4e>] ? prune_super+0x3e/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d0cae>] shrink_slab+0xae/0x340
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd63f9b1>] ? vmpressure+0x21/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d403a>] do_try_to_free_pages+0x3ca/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd64ca4e>] ? prune_super+0x3e/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d0cae>] shrink_slab+0xae/0x340
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd63f9b1>] ? vmpressure+0x21/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d403a>] do_try_to_free_pages+0x3ca/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd64ca4e>] ? prune_super+0x3e/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d0cae>] shrink_slab+0xae/0x340
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd63f9b1>] ? vmpressure+0x21/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d403a>] do_try_to_free_pages+0x3ca/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd5fe239>] ? __page_check_address+0x99/0x250
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5fe43d>] page_referenced_one+0x4d/0x1b0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd600473>] rmap_walk+0xf3/0x360
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd600793>] page_referenced+0xb3/0x130
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5fe3f0>] ? __page_check_address+0x250/0x250
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd600270>] ? page_get_anon_vma+0xa0/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d2f44>] shrink_active_list+0x1f4/0x370
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d34b1>] shrink_lruvec+0x3f1/0x740
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d3876>] shrink_zone+0x76/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d3d60>] do_try_to_free_pages+0xf0/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d0861>] ? bnx2x_set_led+0x2d1/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd5fe239>] ? __page_check_address+0x99/0x250
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5fe43d>] page_referenced_one+0x4d/0x1b0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd600473>] rmap_walk+0xf3/0x360
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd600793>] page_referenced+0xb3/0x130
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5fe3f0>] ? __page_check_address+0x250/0x250
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd600270>] ? page_get_anon_vma+0xa0/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d2f44>] shrink_active_list+0x1f4/0x370
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d34b1>] shrink_lruvec+0x3f1/0x740
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d3876>] shrink_zone+0x76/0x1a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d3d60>] do_try_to_free_pages+0xf0/0x520
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5d428c>] try_to_free_pages+0xfc/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c7eb1>] __alloc_pages_nodemask+0x831/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0778426>] ceph_kvmalloc+0x26/0x60 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b9da>] ceph_msg_new+0xca/0x1a0 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc0a80a7c>] mds_alloc_msg+0x4c/0x80 [ceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077b102>] read_partial_message+0x2b2/0x900 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e56a>] ? kernel_recvmsg+0x3a/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077bf78>] try_read+0x4c8/0x1300 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e134e>] ? account_entity_dequeue+0xae/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4e4cdc>] ? dequeue_entity+0x11c/0x5e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda2e417>] ? kernel_sendmsg+0x37/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc077cfb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb7f608>] ? __schedule+0x448/0x9c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4be1bf>] process_one_work+0x17f/0x440
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf2d6>] worker_thread+0x126/0x3c0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4bf1b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c6191>] kthread+0xd1/0xe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4c60c0>] ? insert_kthread_work+0x40/0x40
[Fri Jan 24 12:00:11 2020] swapper/0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d06e7>] ? bnx2x_set_led+0x157/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd9c1077>] ? cpuidle_enter_state+0x57/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd9c106d>] ? cpuidle_enter_state+0x4d/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd9c11ce>] cpuidle_idle_call+0xde/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd437c6e>] arch_cpu_idle+0xe/0xc0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd500cda>] cpu_startup_entry+0x14a/0x1e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb68c07>] rest_init+0x77/0x80
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe1881cb>] start_kernel+0x450/0x471
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe187b7b>] ? repair_env_string+0x5c/0x5c
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe187120>] ? early_idt_handler_array+0x120/0x120
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe18772f>] x86_64_start_reservations+0x24/0x26
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe187885>] x86_64_start_kernel+0x154/0x177
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4000d5>] start_cpu+0x5/0x14
[Fri Jan 24 12:00:11 2020] swapper/0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:11 2020] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:11 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:11 2020] Call Trace:
[Fri Jan 24 12:00:11 2020]  <IRQ>  [<ffffffffbdb79c45>] dump_stack+0x19/0x1b
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c3450>] warn_alloc_failed+0x110/0x180
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5c805f>] __alloc_pages_nodemask+0x9df/0xbe0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd616208>] alloc_pages_current+0x98/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd5e3208>] kmalloc_order+0x18/0x40
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd621736>] kmalloc_order_trace+0x26/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] ? __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd625cf1>] __kmalloc+0x211/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d9d7a>] bnx2x_frag_alloc.isra.58+0x2a/0x40 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da4b0>] bnx2x_alloc_rx_data.isra.66+0x50/0x220 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04db3ee>] bnx2x_rx_int+0x87e/0x1950 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda37c24>] ? consume_skb+0x34/0x90
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4cb9d>] ? __dev_kfree_skb_any+0x3d/0x50
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04da898>] ? bnx2x_free_tx_pkt+0x218/0x300 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04d06e7>] ? bnx2x_set_led+0x157/0x620 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffc04de43d>] bnx2x_poll+0x1dd/0x260 [bnx2x]
[Fri Jan 24 12:00:11 2020]  [<ffffffffbda4f57f>] net_rx_action+0x26f/0x390
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a52a5>] __do_softirq+0xf5/0x280
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb9042c>] call_softirq+0x1c/0x30
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd42f715>] do_softirq+0x65/0xa0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4a5625>] irq_exit+0x105/0x110
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb91886>] do_IRQ+0x56/0xf0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb8336a>] common_interrupt+0x16a/0x16a
[Fri Jan 24 12:00:11 2020]  <EOI>  [<ffffffffbd9c1077>] ? cpuidle_enter_state+0x57/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd9c106d>] ? cpuidle_enter_state+0x4d/0xd0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd9c11ce>] cpuidle_idle_call+0xde/0x230
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd437c6e>] arch_cpu_idle+0xe/0xc0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd500cda>] cpu_startup_entry+0x14a/0x1e0
[Fri Jan 24 12:00:11 2020]  [<ffffffffbdb68c07>] rest_init+0x77/0x80
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe1881cb>] start_kernel+0x450/0x471
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe187b7b>] ? repair_env_string+0x5c/0x5c
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe187120>] ? early_idt_handler_array+0x120/0x120
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe18772f>] x86_64_start_reservations+0x24/0x26
[Fri Jan 24 12:00:11 2020]  [<ffffffffbe187885>] x86_64_start_kernel+0x154/0x177
[Fri Jan 24 12:00:11 2020]  [<ffffffffbd4000d5>] start_cpu+0x5/0x14
[Fri Jan 24 12:00:16 2020] warn_alloc_failed: 5926 callbacks suppressed
[Fri Jan 24 12:00:16 2020] kworker/0:0: page allocation failure: order:2, mode:0x4020
[Fri Jan 24 12:00:16 2020] CPU: 0 PID: 663485 Comm: kworker/0:0 Not tainted 3.10.0-1062.4.3.el7.ug.x86_64 #1
[Fri Jan 24 12:00:16 2020] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.10.5 07/25/2019
[Fri Jan 24 12:00:16 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph]

I'v had set this setting long time ago:

vm.min_free_kbytes = 3145728

I am running 14.2.6 cluster, but I also saw this with earlier versions. I'm also seeing osds going down or even hanging now and then.

Is there something else to configure?

Thanks!!