Project

General

Profile

Actions
Copy link

Bug #8000

closed

SLAB: Unable to allocate memory on node 0

Added by Dmitry Smirnov about 10 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Ilya Dryomov
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I'm getting the following kernel errors with ext4 on rbd:

Apr  6 11:43:38 debworm kernel: [103549.359234] jbd2/rbd1-8: page allocation failure: order:1, mode:0x204020
Apr  6 11:43:38 debworm kernel: [103549.359239] CPU: 1 PID: 5608 Comm: jbd2/rbd1-8 Tainted: P           O 3.13-1-amd64 #1 Debian 3.13.7-1
Apr  6 11:43:38 debworm kernel: [103549.359240] Hardware name: Gigabyte Technology Co., Ltd. EP45-DS4P/EP45-DS4P, BIOS F9A 04/21/2009
Apr  6 11:43:38 debworm kernel: [103549.359242]  ffff880096bf78c8 ffffffff814a1327 0000000000204020 ffffffff8111b503
Apr  6 11:43:38 debworm kernel: [103549.359245]  0000000000000000 0000000000000000 ffff88022fff6b00 000000012fff5000
Apr  6 11:43:38 debworm kernel: [103549.359247]  ffff88022fff8fb0 0000000200000046 ffff88022fff6b08 0000000000000001
Apr  6 11:43:38 debworm kernel: [103549.359250] Call Trace:
Apr  6 11:43:38 debworm kernel: [103549.359257]  [<ffffffff814a1327>] ? dump_stack+0x41/0x51
Apr  6 11:43:38 debworm kernel: [103549.359260]  [<ffffffff8111b503>] ? warn_alloc_failed+0xe3/0x130
Apr  6 11:43:38 debworm kernel: [103549.359262]  [<ffffffff8111f5ea>] ? __alloc_pages_nodemask+0x81a/0x9d0
Apr  6 11:43:38 debworm kernel: [103549.359265]  [<ffffffff811607ef>] ? kmem_getpages+0x5f/0x140
Apr  6 11:43:38 debworm kernel: [103549.359268]  [<ffffffff81162007>] ? fallback_alloc+0x1c7/0x200
Apr  6 11:43:38 debworm kernel: [103549.359270]  [<ffffffff811625ad>] ? kmem_cache_alloc+0x11d/0x430
Apr  6 11:43:38 debworm kernel: [103549.359281]  [<ffffffffa108cc0e>] ? ceph_osdc_alloc_request+0x4e/0x2d0 [libceph]
Apr  6 11:43:38 debworm kernel: [103549.359284]  [<ffffffffa11571de>] ? rbd_osd_req_create+0x5e/0x1c0 [rbd]
Apr  6 11:43:38 debworm kernel: [103549.359287]  [<ffffffffa1159803>] ? rbd_img_request_fill+0x273/0x8b0 [rbd]
Apr  6 11:43:38 debworm kernel: [103549.359290]  [<ffffffffa115b47d>] ? rbd_request_fn+0x1dd/0x2b0 [rbd]
Apr  6 11:43:38 debworm kernel: [103549.359293]  [<ffffffff812485ea>] ? __blk_run_queue+0x2a/0x40
Apr  6 11:43:38 debworm kernel: [103549.359295]  [<ffffffff81248690>] ? queue_unplugged+0x20/0x90
Apr  6 11:43:38 debworm kernel: [103549.359298]  [<ffffffff8124b95f>] ? blk_flush_plug_list+0x1ff/0x240
Apr  6 11:43:38 debworm kernel: [103549.359300]  [<ffffffff8124bcfb>] ? blk_finish_plug+0xb/0x30
Apr  6 11:43:38 debworm kernel: [103549.359318]  [<ffffffffa0299a07>] ? jbd2_journal_commit_transaction+0xd67/0x17d0 [jbd2]
Apr  6 11:43:38 debworm kernel: [103549.359321]  [<ffffffff81019895>] ? sched_clock+0x5/0x10
Apr  6 11:43:38 debworm kernel: [103549.359324]  [<ffffffff8108b8d5>] ? sched_clock_local+0x15/0x80
Apr  6 11:43:38 debworm kernel: [103549.359330]  [<ffffffffa029e0af>] ? kjournald2+0xaf/0x240 [jbd2]
Apr  6 11:43:38 debworm kernel: [103549.359333]  [<ffffffff81096f40>] ? prepare_to_wait_event+0xf0/0xf0
Apr  6 11:43:38 debworm kernel: [103549.359338]  [<ffffffffa029e000>] ? commit_timeout+0x10/0x10 [jbd2]
Apr  6 11:43:38 debworm kernel: [103549.359341]  [<ffffffff8107bb41>] ? kthread+0xc1/0xe0
Apr  6 11:43:38 debworm kernel: [103549.359343]  [<ffffffff8107ba80>] ? kthread_create_on_node+0x180/0x180
Apr  6 11:43:38 debworm kernel: [103549.359346]  [<ffffffff814ae44c>] ? ret_from_fork+0x7c/0xb0
Apr  6 11:43:38 debworm kernel: [103549.359348]  [<ffffffff8107ba80>] ? kthread_create_on_node+0x180/0x180
Apr  6 11:43:38 debworm kernel: [103549.359349] Mem-Info:
Apr  6 11:43:38 debworm kernel: [103549.359350] Node 0 DMA per-cpu:
Apr  6 11:43:38 debworm kernel: [103549.359352] CPU    0: hi:    0, btch:   1 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359353] CPU    1: hi:    0, btch:   1 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359355] CPU    2: hi:    0, btch:   1 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359356] CPU    3: hi:    0, btch:   1 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359357] Node 0 DMA32 per-cpu:
Apr  6 11:43:38 debworm kernel: [103549.359359] CPU    0: hi:  186, btch:  31 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359360] CPU    1: hi:  186, btch:  31 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359361] CPU    2: hi:  186, btch:  31 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359363] CPU    3: hi:  186, btch:  31 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359364] Node 0 Normal per-cpu:
Apr  6 11:43:38 debworm kernel: [103549.359365] CPU    0: hi:  186, btch:  31 usd:   8
Apr  6 11:43:38 debworm kernel: [103549.359367] CPU    1: hi:  186, btch:  31 usd: 213
Apr  6 11:43:38 debworm kernel: [103549.359368] CPU    2: hi:  186, btch:  31 usd: 198
Apr  6 11:43:38 debworm kernel: [103549.359369] CPU    3: hi:  186, btch:  31 usd: 185
Apr  6 11:43:38 debworm kernel: [103549.359372] active_anon:869609 inactive_anon:216123 isolated_anon:0
Apr  6 11:43:38 debworm kernel: [103549.359372]  active_file:380431 inactive_file:414833 isolated_file:0
Apr  6 11:43:38 debworm kernel: [103549.359372]  unevictable:0 dirty:13334 writeback:0 unstable:0
Apr  6 11:43:38 debworm kernel: [103549.359372]  free:96767 slab_reclaimable:25267 slab_unreclaimable:13377
Apr  6 11:43:38 debworm kernel: [103549.359372]  mapped:50681 shmem:177434 pagetables:14845 bounce:0
Apr  6 11:43:38 debworm kernel: [103549.359372]  free_cma:0
Apr  6 11:43:38 debworm kernel: [103549.359375] Node 0 DMA free:15904kB min:252kB low:312kB high:376kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr  6 11:43:38 debworm kernel: [103549.359380] lowmem_reserve[]: 0 3236 7984 7984
Apr  6 11:43:38 debworm kernel: [103549.359382] Node 0 DMA32 free:135116kB min:53040kB low:66300kB high:79560kB active_anon:1271424kB inactive_anon:317936kB active_file:819608kB inactive_file:732796kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3390144kB managed:3316452kB mlocked:0kB dirty:392kB writeback:0kB mapped:98084kB shmem:287868kB slab_reclaimable:664kB slab_unreclaimable:100kB kernel_stack:2264kB pagetables:23540kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr  6 11:43:38 debworm kernel: [103549.359386] lowmem_reserve[]: 0 0 4748 4748
Apr  6 11:43:38 debworm kernel: [103549.359389] Node 0 Normal free:236048kB min:77772kB low:97212kB high:116656kB active_anon:2207012kB inactive_anon:546556kB active_file:702116kB inactive_file:926536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:4980736kB managed:4862776kB mlocked:4kB dirty:52944kB writeback:0kB mapped:104640kB shmem:421868kB slab_reclaimable:100404kB slab_unreclaimable:53408kB kernel_stack:4048kB pagetables:35840kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr  6 11:43:38 debworm kernel: [103549.359393] lowmem_reserve[]: 0 0 0 0
Apr  6 11:43:38 debworm kernel: [103549.359395] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15904kB
Apr  6 11:43:38 debworm kernel: [103549.359405] Node 0 DMA32: 32135*4kB (UM) 310*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 135116kB
Apr  6 11:43:38 debworm kernel: [103549.359412] Node 0 Normal: 57959*4kB (UE) 6*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 235980kB
Apr  6 11:43:38 debworm kernel: [103549.359420] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr  6 11:43:38 debworm kernel: [103549.359421] 973008 total pagecache pages
Apr  6 11:43:38 debworm kernel: [103549.359423] 287 pages in swap cache
Apr  6 11:43:38 debworm kernel: [103549.359424] Swap cache stats: add 26110, delete 25823, find 332/417
Apr  6 11:43:38 debworm kernel: [103549.359425] Free swap  = 12474668kB
Apr  6 11:43:38 debworm kernel: [103549.359426] Total swap = 12574716kB
Apr  6 11:43:38 debworm kernel: [103549.359428] SLAB: Unable to allocate memory on node 0 (gfp=0x4020)
Apr  6 11:43:38 debworm kernel: [103549.359429]   cache: ceph_osd_request, object size: 864, order: 1
Apr  6 11:43:38 debworm kernel: [103549.359431]   node 0: slabs: 2/2, objs: 18/18, free: 0
Apr  6 11:43:38 debworm kernel: [103549.359457] ------------[ cut here ]------------
Apr  6 11:43:38 debworm kernel: [103549.359505] invalid opcode: 0000 [#1] SMP 
Apr  6 11:43:38 debworm kernel: [103549.359522] Modules linked in: nls_utf8 nls_cp437 usb_storage ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs rbd ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_powersave snd_hrtimer cpufreq_stats cpufreq_conservative cpufreq_userspace pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) binfmt_misc cbc ceph libceph rpcsec_gss_krb5 nfsv4 dns_resolver nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc bonding xts gf128mul fuse it87 hwmon_vid parport_pc ppdev lp parport loop dm_crypt dm_mod fglrx(PO) snd_hda_codec_hdmi snd_usb_audio snd_hda_codec_realtek snd_usbmidi_lib snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core videodev media iTCO_wdt iTCO_vendor_support snd_timer snd acpi_cpufreq pcspkr radeon soundcore ttm drm_kms_helper coretemp drm kvm_intel kvm psmouse serio_raw evdev i2c_i801 button processor i2c_algo_bit lpc_ich i2c_core mfd_core thermal_sys ext4 crc16 mbcache jbd2 btrfs xor raid6_pq crc32c libcrc32c raid1 md_mod sd_mod crct10dif_generic crc_t10dif crct10dif_common hid_generic usbhid hid sg sr_mod cdrom ata_generic firewire_ohci firewire_core crc_itu_t pata_it8213 ahci libahci libata scsi_mod ehci_pci uhci_hcd ehci_hcd r8169 mii usbcore usb_common
Apr  6 11:43:38 debworm kernel: [103549.360352] CPU: 1 PID: 5608 Comm: jbd2/rbd1-8 Tainted: P           O 3.13-1-amd64 #1 Debian 3.13.7-1
Apr  6 11:43:38 debworm kernel: [103549.360391] Hardware name: Gigabyte Technology Co., Ltd. EP45-DS4P/EP45-DS4P, BIOS F9A 04/21/2009
Apr  6 11:43:38 debworm kernel: [103549.360429] task: ffff8801d3133800 ti: ffff880096bf6000 task.ti: ffff880096bf6000
Apr  6 11:43:38 debworm kernel: [103549.360458] RIP: 0010:[<ffffffffa1158662>]  [<ffffffffa1158662>] rbd_obj_request_destroy+0x142/0x150 [rbd]
Apr  6 11:43:38 debworm kernel: [103549.360502] RSP: 0018:ffff880096bf7b18  EFLAGS: 00010286
Apr  6 11:43:38 debworm kernel: [103549.360520] RAX: 000000000000006e RBX: ffff88021239d680 RCX: 0000000000000006
Apr  6 11:43:38 debworm kernel: [103549.360546] RDX: 0000000000000007 RSI: 0000000000000046 RDI: 0000000000000246
Apr  6 11:43:38 debworm kernel: [103549.360573] RBP: ffff8801f3fd4f38 R08: 000000000000000a R09: 000000000000a34e
Apr  6 11:43:38 debworm kernel: [103549.360601] R10: 0000000000000000 R11: ffff880096bf787e R12: ffff88021239d5d8
Apr  6 11:43:38 debworm kernel: [103549.360628] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88021239d5d8
Apr  6 11:43:38 debworm kernel: [103549.360660] FS:  0000000000000000(0000) GS:ffff88022fc80000(0000) knlGS:0000000000000000
Apr  6 11:43:38 debworm kernel: [103549.360688] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  6 11:43:38 debworm kernel: [103549.360708] CR2: 00007fa667dc4000 CR3: 00000000a2883000 CR4: 00000000000007e0
Apr  6 11:43:38 debworm kernel: [103549.360732] Stack:
Apr  6 11:43:38 debworm kernel: [103549.360740]  0000000000000000 ffff8801f3fd4f38 000000180c140000 ffffffffa1159bb1
Apr  6 11:43:38 debworm kernel: [103549.360772]  ffff880223321b80 0000000000002000 ffff88022646e9c8 0000100000000000
Apr  6 11:43:38 debworm kernel: [103549.360803]  0000000000140000 000000180c140000 0000000000000246 0000000100000246
Apr  6 11:43:38 debworm kernel: [103549.360835] Call Trace:
Apr  6 11:43:38 debworm kernel: [103549.360847]  [<ffffffffa1159bb1>] ? rbd_img_request_fill+0x621/0x8b0 [rbd]
Apr  6 11:43:38 debworm kernel: [103549.360872]  [<ffffffffa115b47d>] ? rbd_request_fn+0x1dd/0x2b0 [rbd]
Apr  6 11:43:38 debworm kernel: [103549.360894]  [<ffffffff812485ea>] ? __blk_run_queue+0x2a/0x40
Apr  6 11:43:38 debworm kernel: [103549.360915]  [<ffffffff81248690>] ? queue_unplugged+0x20/0x90
Apr  6 11:43:38 debworm kernel: [103549.360938]  [<ffffffff8124b95f>] ? blk_flush_plug_list+0x1ff/0x240
Apr  6 11:43:38 debworm kernel: [103549.360960]  [<ffffffff8124bcfb>] ? blk_finish_plug+0xb/0x30
Apr  6 11:43:38 debworm kernel: [103549.360987]  [<ffffffffa0299a07>] ? jbd2_journal_commit_transaction+0xd67/0x17d0 [jbd2]
Apr  6 11:43:38 debworm kernel: [103549.361016]  [<ffffffff81019895>] ? sched_clock+0x5/0x10
Apr  6 11:43:38 debworm kernel: [103549.361035]  [<ffffffff8108b8d5>] ? sched_clock_local+0x15/0x80
Apr  6 11:43:38 debworm kernel: [103549.361060]  [<ffffffffa029e0af>] ? kjournald2+0xaf/0x240 [jbd2]
Apr  6 11:43:38 debworm kernel: [103549.361081]  [<ffffffff81096f40>] ? prepare_to_wait_event+0xf0/0xf0
Apr  6 11:43:38 debworm kernel: [103549.361107]  [<ffffffffa029e000>] ? commit_timeout+0x10/0x10 [jbd2]
Apr  6 11:43:38 debworm kernel: [103549.361129]  [<ffffffff8107bb41>] ? kthread+0xc1/0xe0
Apr  6 11:43:38 debworm kernel: [103549.361148]  [<ffffffff8107ba80>] ? kthread_create_on_node+0x180/0x180
Apr  6 11:43:38 debworm kernel: [103549.361173]  [<ffffffff814ae44c>] ? ret_from_fork+0x7c/0xb0
Apr  6 11:43:38 debworm kernel: [103549.361193]  [<ffffffff8107ba80>] ? kthread_create_on_node+0x180/0x180
Apr  6 11:43:38 debworm kernel: [103549.361214] Code: a1 31 c0 e8 14 69 34 e0 0f 0b 48 c7 c1 38 e8 15 a1 ba 4b 07 00 00 48 c7 c6 e0 f8 15 a1 48 c7 c7 d0 e6 15 a1 31 c0 e8 f1 68 34 e0 <0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 85 ff 74 23 8b 87 
Apr  6 11:43:38 debworm kernel: [103549.361423]  RSP <ffff880096bf7b18>
Apr  6 11:43:38 debworm kernel: [103549.368551] ---[ end trace 76340b50a326be77 ]---

This is a catastrophic error with reboot as the only mean to recover.

Actions
Copy link

Also available in: Atom PDF