Bug #40482
closed[libceph] page allocation failure when parsing new OSD map
0%
Description
Jun 01 02:03:49 rhnode kernel: kworker/15:1: page allocation failure: order:7, mode:0x104050
Jun 01 02:03:49 rhnode kernel: CPU: 15 PID: 6313 Comm: kworker/15:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-862.6.3.el7.x86_64 #1
Jun 01 02:03:49 rhnode kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Jun 01 02:03:49 rhnode kernel: Workqueue: ceph-msgr ceph_con_workfn [libceph]
Jun 01 02:03:49 rhnode kernel: Call Trace:
Jun 01 02:03:49 rhnode kernel: [<ffffffffb570e80e>] dump_stack+0x19/0x1b
Jun 01 02:03:49 rhnode kernel: [<ffffffffb519a5b0>] warn_alloc_failed+0x110/0x180
Jun 01 02:03:49 rhnode kernel: [<ffffffffb519f134>] __alloc_pages_nodemask+0x9b4/0xbb0
Jun 01 02:03:49 rhnode kernel: [<ffffffffb51e8ce8>] alloc_pages_current+0x98/0x110
Jun 01 02:03:49 rhnode kernel: [<ffffffffb519934e>] __get_free_pages+0xe/0x40
Jun 01 02:03:49 rhnode kernel: [<ffffffffb51f4f9e>] kmalloc_order_trace+0x2e/0xa0
Jun 01 02:03:49 rhnode kernel: [<ffffffffb51fa511>] ? __kmalloc_track_caller+0x221/0x240
Jun 01 02:03:49 rhnode kernel: [<ffffffffb51fa511>] __kmalloc_track_caller+0x221/0x240
Jun 01 02:03:49 rhnode kernel: [<ffffffffc0aa4056>] ? osdmap_set_max_osd+0x76/0x1d0 [libceph]
Jun 01 02:03:49 rhnode kernel: [<ffffffffb51b395f>] krealloc+0x4f/0xa0
Jun 01 02:03:49 rhnode kernel: [<ffffffffc0aa4056>] osdmap_set_max_osd+0x76/0x1d0 [libceph]
Jun 01 02:03:49 rhnode kernel: [<ffffffffc0aa7255>] ceph_osdmap_decode+0x195/0x860 [libceph]
Jun 01 02:03:49 rhnode kernel: [<ffffffffc0a9daf4>] handle_one_map+0x224/0x250 [libceph]
Jun 01 02:03:49 rhnode kernel: [<ffffffffc0aa233c>] ceph_osdc_handle_map+0x7dc/0x8c0 [libceph]
Jun 01 02:03:49 rhnode kernel: [<ffffffffc0a96490>] dispatch+0x350/0x790 [libceph]
Jun 01 02:03:49 rhnode kernel: [<ffffffffc0a91ff4>] try_read+0x4e4/0x1210 [libceph]
Jun 01 02:03:49 rhnode kernel: [<ffffffffb50d972e>] ? dequeue_task_fair+0x41e/0x660
Jun 01 02:03:49 rhnode kernel: [<ffffffffb502959e>] ? __switch_to+0xce/0x580
Jun 01 02:03:49 rhnode kernel: [<ffffffffb50c9e50>] ? finish_task_switch+0x50/0x170
Jun 01 02:03:49 rhnode kernel: [<ffffffffc0a92dd9>] ceph_con_workfn+0xb9/0x670 [libceph]
Jun 01 02:03:49 rhnode kernel: [<ffffffffb50b35ef>] process_one_work+0x17f/0x440
Jun 01 02:03:49 rhnode kernel: [<ffffffffb50b4686>] worker_thread+0x126/0x3c0
Jun 01 02:03:49 rhnode kernel: [<ffffffffb50b4560>] ? manage_workers.isra.24+0x2a0/0x2a0
Jun 01 02:03:49 rhnode kernel: [<ffffffffb50bb621>] kthread+0xd1/0xe0
Jun 01 02:03:49 rhnode kernel: [<ffffffffb50bb550>] ? insert_kthread_work+0x40/0x40
Jun 01 02:03:49 rhnode kernel: [<ffffffffb57205f7>] ret_from_fork_nospec_begin+0x21/0x21
Jun 01 02:03:49 rhnode kernel: [<ffffffffb50bb550>] ? insert_kthread_work+0x40/0x40
Jun 01 02:03:49 rhnode kernel: Mem-Info:
...
...
Jun 01 02:03:49 rhnode kernel: lowmem_reserve[]: 0 0 0 0
Jun 01 02:03:49 rhnode kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Jun 01 02:03:49 rhnode kernel: Node 0 DMA32: 3980*4kB (UEM) 3460*8kB (UEM) 1895*16kB (UEM) 610*32kB (UEM) 277*64kB (UEM) 314*128kB (UEM) 72*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 169792kB
Jun 01 02:03:49 rhnode kernel: Node 0 Normal: 8145*4kB (UEM) 44558*8kB (UEM) 2521*16kB (UM) 516*32kB (UM) 549*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 481028kB
Jun 01 02:03:49 rhnode kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 01 02:03:49 rhnode kernel: 7209999 total pagecache pages
Jun 01 02:03:49 rhnode kernel: 0 pages in swap cache
Jun 01 02:03:49 rhnode kernel: Swap cache stats: add 0, delete 0, find 0/0
Jun 01 02:03:49 rhnode kernel: Free swap = 4194300kB
Jun 01 02:03:49 rhnode kernel: Total swap = 4194300kB
Jun 01 02:03:49 rhnode kernel: 8388496 pages RAM
Jun 01 02:03:49 rhnode kernel: 0 pages HighMem/MovableOnly
Jun 01 02:03:49 rhnode kernel: 193671 pages reserved
Jun 01 02:03:49 rhnode kernel: libceph: corrupt full osdmap (-12) epoch 345678 off 13481 (ffff9fe419c414c9 of ffff9fe419c3e020-ffff9fe419da13d1)
Jun 01 02:03:49 rhnode journal: Missed 7896 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 4393 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 2298 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 4266 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 2098 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 26193 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 6110 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 13598 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 3671 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 3696 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 11856 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 1787 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 11848 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 4408 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 4336 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 5350 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 5066 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 9397 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 2144 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 6545 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 8105 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 13765 kernel messages
Jun 01 02:03:49 rhnode journal: Missed 23058 kernel messages
Jun 10 14:48:10 rhnode kernel: libceph: mon2 10.x.x.3:6789 io error
Jun 10 14:48:10 rhnode kernel: libceph: mon2 10.x.x.3:6789 session lost, hunting for new mon
Jun 10 14:48:18 rhnode kernel: libceph: mon0 10.x.x.1:6789 session established
Updated by Prashant D almost 5 years ago
- Project changed from Ceph to Linux kernel client
Updated by Ilya Dryomov almost 5 years ago
- Is duplicate of Bug #40481: osdmap->osd_addr allocation is susceptible to memory fragmentation added