Actions
Bug #8000
closedSLAB: Unable to allocate memory on node 0
% Done:
0%
Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I'm getting the following kernel errors with ext4 on rbd:
Apr 6 11:43:38 debworm kernel: [103549.359234] jbd2/rbd1-8: page allocation failure: order:1, mode:0x204020 Apr 6 11:43:38 debworm kernel: [103549.359239] CPU: 1 PID: 5608 Comm: jbd2/rbd1-8 Tainted: P O 3.13-1-amd64 #1 Debian 3.13.7-1 Apr 6 11:43:38 debworm kernel: [103549.359240] Hardware name: Gigabyte Technology Co., Ltd. EP45-DS4P/EP45-DS4P, BIOS F9A 04/21/2009 Apr 6 11:43:38 debworm kernel: [103549.359242] ffff880096bf78c8 ffffffff814a1327 0000000000204020 ffffffff8111b503 Apr 6 11:43:38 debworm kernel: [103549.359245] 0000000000000000 0000000000000000 ffff88022fff6b00 000000012fff5000 Apr 6 11:43:38 debworm kernel: [103549.359247] ffff88022fff8fb0 0000000200000046 ffff88022fff6b08 0000000000000001 Apr 6 11:43:38 debworm kernel: [103549.359250] Call Trace: Apr 6 11:43:38 debworm kernel: [103549.359257] [<ffffffff814a1327>] ? dump_stack+0x41/0x51 Apr 6 11:43:38 debworm kernel: [103549.359260] [<ffffffff8111b503>] ? warn_alloc_failed+0xe3/0x130 Apr 6 11:43:38 debworm kernel: [103549.359262] [<ffffffff8111f5ea>] ? __alloc_pages_nodemask+0x81a/0x9d0 Apr 6 11:43:38 debworm kernel: [103549.359265] [<ffffffff811607ef>] ? kmem_getpages+0x5f/0x140 Apr 6 11:43:38 debworm kernel: [103549.359268] [<ffffffff81162007>] ? fallback_alloc+0x1c7/0x200 Apr 6 11:43:38 debworm kernel: [103549.359270] [<ffffffff811625ad>] ? kmem_cache_alloc+0x11d/0x430 Apr 6 11:43:38 debworm kernel: [103549.359281] [<ffffffffa108cc0e>] ? ceph_osdc_alloc_request+0x4e/0x2d0 [libceph] Apr 6 11:43:38 debworm kernel: [103549.359284] [<ffffffffa11571de>] ? rbd_osd_req_create+0x5e/0x1c0 [rbd] Apr 6 11:43:38 debworm kernel: [103549.359287] [<ffffffffa1159803>] ? rbd_img_request_fill+0x273/0x8b0 [rbd] Apr 6 11:43:38 debworm kernel: [103549.359290] [<ffffffffa115b47d>] ? rbd_request_fn+0x1dd/0x2b0 [rbd] Apr 6 11:43:38 debworm kernel: [103549.359293] [<ffffffff812485ea>] ? __blk_run_queue+0x2a/0x40 Apr 6 11:43:38 debworm kernel: [103549.359295] [<ffffffff81248690>] ? queue_unplugged+0x20/0x90 Apr 6 11:43:38 debworm kernel: [103549.359298] [<ffffffff8124b95f>] ? blk_flush_plug_list+0x1ff/0x240 Apr 6 11:43:38 debworm kernel: [103549.359300] [<ffffffff8124bcfb>] ? blk_finish_plug+0xb/0x30 Apr 6 11:43:38 debworm kernel: [103549.359318] [<ffffffffa0299a07>] ? jbd2_journal_commit_transaction+0xd67/0x17d0 [jbd2] Apr 6 11:43:38 debworm kernel: [103549.359321] [<ffffffff81019895>] ? sched_clock+0x5/0x10 Apr 6 11:43:38 debworm kernel: [103549.359324] [<ffffffff8108b8d5>] ? sched_clock_local+0x15/0x80 Apr 6 11:43:38 debworm kernel: [103549.359330] [<ffffffffa029e0af>] ? kjournald2+0xaf/0x240 [jbd2] Apr 6 11:43:38 debworm kernel: [103549.359333] [<ffffffff81096f40>] ? prepare_to_wait_event+0xf0/0xf0 Apr 6 11:43:38 debworm kernel: [103549.359338] [<ffffffffa029e000>] ? commit_timeout+0x10/0x10 [jbd2] Apr 6 11:43:38 debworm kernel: [103549.359341] [<ffffffff8107bb41>] ? kthread+0xc1/0xe0 Apr 6 11:43:38 debworm kernel: [103549.359343] [<ffffffff8107ba80>] ? kthread_create_on_node+0x180/0x180 Apr 6 11:43:38 debworm kernel: [103549.359346] [<ffffffff814ae44c>] ? ret_from_fork+0x7c/0xb0 Apr 6 11:43:38 debworm kernel: [103549.359348] [<ffffffff8107ba80>] ? kthread_create_on_node+0x180/0x180 Apr 6 11:43:38 debworm kernel: [103549.359349] Mem-Info: Apr 6 11:43:38 debworm kernel: [103549.359350] Node 0 DMA per-cpu: Apr 6 11:43:38 debworm kernel: [103549.359352] CPU 0: hi: 0, btch: 1 usd: 0 Apr 6 11:43:38 debworm kernel: [103549.359353] CPU 1: hi: 0, btch: 1 usd: 0 Apr 6 11:43:38 debworm kernel: [103549.359355] CPU 2: hi: 0, btch: 1 usd: 0 Apr 6 11:43:38 debworm kernel: [103549.359356] CPU 3: hi: 0, btch: 1 usd: 0 Apr 6 11:43:38 debworm kernel: [103549.359357] Node 0 DMA32 per-cpu: Apr 6 11:43:38 debworm kernel: [103549.359359] CPU 0: hi: 186, btch: 31 usd: 0 Apr 6 11:43:38 debworm kernel: [103549.359360] CPU 1: hi: 186, btch: 31 usd: 0 Apr 6 11:43:38 debworm kernel: [103549.359361] CPU 2: hi: 186, btch: 31 usd: 0 Apr 6 11:43:38 debworm kernel: [103549.359363] CPU 3: hi: 186, btch: 31 usd: 0 Apr 6 11:43:38 debworm kernel: [103549.359364] Node 0 Normal per-cpu: Apr 6 11:43:38 debworm kernel: [103549.359365] CPU 0: hi: 186, btch: 31 usd: 8 Apr 6 11:43:38 debworm kernel: [103549.359367] CPU 1: hi: 186, btch: 31 usd: 213 Apr 6 11:43:38 debworm kernel: [103549.359368] CPU 2: hi: 186, btch: 31 usd: 198 Apr 6 11:43:38 debworm kernel: [103549.359369] CPU 3: hi: 186, btch: 31 usd: 185 Apr 6 11:43:38 debworm kernel: [103549.359372] active_anon:869609 inactive_anon:216123 isolated_anon:0 Apr 6 11:43:38 debworm kernel: [103549.359372] active_file:380431 inactive_file:414833 isolated_file:0 Apr 6 11:43:38 debworm kernel: [103549.359372] unevictable:0 dirty:13334 writeback:0 unstable:0 Apr 6 11:43:38 debworm kernel: [103549.359372] free:96767 slab_reclaimable:25267 slab_unreclaimable:13377 Apr 6 11:43:38 debworm kernel: [103549.359372] mapped:50681 shmem:177434 pagetables:14845 bounce:0 Apr 6 11:43:38 debworm kernel: [103549.359372] free_cma:0 Apr 6 11:43:38 debworm kernel: [103549.359375] Node 0 DMA free:15904kB min:252kB low:312kB high:376kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Apr 6 11:43:38 debworm kernel: [103549.359380] lowmem_reserve[]: 0 3236 7984 7984 Apr 6 11:43:38 debworm kernel: [103549.359382] Node 0 DMA32 free:135116kB min:53040kB low:66300kB high:79560kB active_anon:1271424kB inactive_anon:317936kB active_file:819608kB inactive_file:732796kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3390144kB managed:3316452kB mlocked:0kB dirty:392kB writeback:0kB mapped:98084kB shmem:287868kB slab_reclaimable:664kB slab_unreclaimable:100kB kernel_stack:2264kB pagetables:23540kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Apr 6 11:43:38 debworm kernel: [103549.359386] lowmem_reserve[]: 0 0 4748 4748 Apr 6 11:43:38 debworm kernel: [103549.359389] Node 0 Normal free:236048kB min:77772kB low:97212kB high:116656kB active_anon:2207012kB inactive_anon:546556kB active_file:702116kB inactive_file:926536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:4980736kB managed:4862776kB mlocked:4kB dirty:52944kB writeback:0kB mapped:104640kB shmem:421868kB slab_reclaimable:100404kB slab_unreclaimable:53408kB kernel_stack:4048kB pagetables:35840kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Apr 6 11:43:38 debworm kernel: [103549.359393] lowmem_reserve[]: 0 0 0 0 Apr 6 11:43:38 debworm kernel: [103549.359395] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15904kB Apr 6 11:43:38 debworm kernel: [103549.359405] Node 0 DMA32: 32135*4kB (UM) 310*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 135116kB Apr 6 11:43:38 debworm kernel: [103549.359412] Node 0 Normal: 57959*4kB (UE) 6*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 235980kB Apr 6 11:43:38 debworm kernel: [103549.359420] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Apr 6 11:43:38 debworm kernel: [103549.359421] 973008 total pagecache pages Apr 6 11:43:38 debworm kernel: [103549.359423] 287 pages in swap cache Apr 6 11:43:38 debworm kernel: [103549.359424] Swap cache stats: add 26110, delete 25823, find 332/417 Apr 6 11:43:38 debworm kernel: [103549.359425] Free swap = 12474668kB Apr 6 11:43:38 debworm kernel: [103549.359426] Total swap = 12574716kB Apr 6 11:43:38 debworm kernel: [103549.359428] SLAB: Unable to allocate memory on node 0 (gfp=0x4020) Apr 6 11:43:38 debworm kernel: [103549.359429] cache: ceph_osd_request, object size: 864, order: 1 Apr 6 11:43:38 debworm kernel: [103549.359431] node 0: slabs: 2/2, objs: 18/18, free: 0 Apr 6 11:43:38 debworm kernel: [103549.359457] ------------[ cut here ]------------ Apr 6 11:43:38 debworm kernel: [103549.359505] invalid opcode: 0000 [#1] SMP Apr 6 11:43:38 debworm kernel: [103549.359522] Modules linked in: nls_utf8 nls_cp437 usb_storage ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs rbd ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_powersave snd_hrtimer cpufreq_stats cpufreq_conservative cpufreq_userspace pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) binfmt_misc cbc ceph libceph rpcsec_gss_krb5 nfsv4 dns_resolver nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc bonding xts gf128mul fuse it87 hwmon_vid parport_pc ppdev lp parport loop dm_crypt dm_mod fglrx(PO) snd_hda_codec_hdmi snd_usb_audio snd_hda_codec_realtek snd_usbmidi_lib snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core videodev media iTCO_wdt iTCO_vendor_support snd_timer snd acpi_cpufreq pcspkr radeon soundcore ttm drm_kms_helper coretemp drm kvm_intel kvm psmouse serio_raw evdev i2c_i801 button processor i2c_algo_bit lpc_ich i2c_core mfd_core thermal_sys ext4 crc16 mbcache jbd2 btrfs xor raid6_pq crc32c libcrc32c raid1 md_mod sd_mod crct10dif_generic crc_t10dif crct10dif_common hid_generic usbhid hid sg sr_mod cdrom ata_generic firewire_ohci firewire_core crc_itu_t pata_it8213 ahci libahci libata scsi_mod ehci_pci uhci_hcd ehci_hcd r8169 mii usbcore usb_common Apr 6 11:43:38 debworm kernel: [103549.360352] CPU: 1 PID: 5608 Comm: jbd2/rbd1-8 Tainted: P O 3.13-1-amd64 #1 Debian 3.13.7-1 Apr 6 11:43:38 debworm kernel: [103549.360391] Hardware name: Gigabyte Technology Co., Ltd. EP45-DS4P/EP45-DS4P, BIOS F9A 04/21/2009 Apr 6 11:43:38 debworm kernel: [103549.360429] task: ffff8801d3133800 ti: ffff880096bf6000 task.ti: ffff880096bf6000 Apr 6 11:43:38 debworm kernel: [103549.360458] RIP: 0010:[<ffffffffa1158662>] [<ffffffffa1158662>] rbd_obj_request_destroy+0x142/0x150 [rbd] Apr 6 11:43:38 debworm kernel: [103549.360502] RSP: 0018:ffff880096bf7b18 EFLAGS: 00010286 Apr 6 11:43:38 debworm kernel: [103549.360520] RAX: 000000000000006e RBX: ffff88021239d680 RCX: 0000000000000006 Apr 6 11:43:38 debworm kernel: [103549.360546] RDX: 0000000000000007 RSI: 0000000000000046 RDI: 0000000000000246 Apr 6 11:43:38 debworm kernel: [103549.360573] RBP: ffff8801f3fd4f38 R08: 000000000000000a R09: 000000000000a34e Apr 6 11:43:38 debworm kernel: [103549.360601] R10: 0000000000000000 R11: ffff880096bf787e R12: ffff88021239d5d8 Apr 6 11:43:38 debworm kernel: [103549.360628] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88021239d5d8 Apr 6 11:43:38 debworm kernel: [103549.360660] FS: 0000000000000000(0000) GS:ffff88022fc80000(0000) knlGS:0000000000000000 Apr 6 11:43:38 debworm kernel: [103549.360688] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Apr 6 11:43:38 debworm kernel: [103549.360708] CR2: 00007fa667dc4000 CR3: 00000000a2883000 CR4: 00000000000007e0 Apr 6 11:43:38 debworm kernel: [103549.360732] Stack: Apr 6 11:43:38 debworm kernel: [103549.360740] 0000000000000000 ffff8801f3fd4f38 000000180c140000 ffffffffa1159bb1 Apr 6 11:43:38 debworm kernel: [103549.360772] ffff880223321b80 0000000000002000 ffff88022646e9c8 0000100000000000 Apr 6 11:43:38 debworm kernel: [103549.360803] 0000000000140000 000000180c140000 0000000000000246 0000000100000246 Apr 6 11:43:38 debworm kernel: [103549.360835] Call Trace: Apr 6 11:43:38 debworm kernel: [103549.360847] [<ffffffffa1159bb1>] ? rbd_img_request_fill+0x621/0x8b0 [rbd] Apr 6 11:43:38 debworm kernel: [103549.360872] [<ffffffffa115b47d>] ? rbd_request_fn+0x1dd/0x2b0 [rbd] Apr 6 11:43:38 debworm kernel: [103549.360894] [<ffffffff812485ea>] ? __blk_run_queue+0x2a/0x40 Apr 6 11:43:38 debworm kernel: [103549.360915] [<ffffffff81248690>] ? queue_unplugged+0x20/0x90 Apr 6 11:43:38 debworm kernel: [103549.360938] [<ffffffff8124b95f>] ? blk_flush_plug_list+0x1ff/0x240 Apr 6 11:43:38 debworm kernel: [103549.360960] [<ffffffff8124bcfb>] ? blk_finish_plug+0xb/0x30 Apr 6 11:43:38 debworm kernel: [103549.360987] [<ffffffffa0299a07>] ? jbd2_journal_commit_transaction+0xd67/0x17d0 [jbd2] Apr 6 11:43:38 debworm kernel: [103549.361016] [<ffffffff81019895>] ? sched_clock+0x5/0x10 Apr 6 11:43:38 debworm kernel: [103549.361035] [<ffffffff8108b8d5>] ? sched_clock_local+0x15/0x80 Apr 6 11:43:38 debworm kernel: [103549.361060] [<ffffffffa029e0af>] ? kjournald2+0xaf/0x240 [jbd2] Apr 6 11:43:38 debworm kernel: [103549.361081] [<ffffffff81096f40>] ? prepare_to_wait_event+0xf0/0xf0 Apr 6 11:43:38 debworm kernel: [103549.361107] [<ffffffffa029e000>] ? commit_timeout+0x10/0x10 [jbd2] Apr 6 11:43:38 debworm kernel: [103549.361129] [<ffffffff8107bb41>] ? kthread+0xc1/0xe0 Apr 6 11:43:38 debworm kernel: [103549.361148] [<ffffffff8107ba80>] ? kthread_create_on_node+0x180/0x180 Apr 6 11:43:38 debworm kernel: [103549.361173] [<ffffffff814ae44c>] ? ret_from_fork+0x7c/0xb0 Apr 6 11:43:38 debworm kernel: [103549.361193] [<ffffffff8107ba80>] ? kthread_create_on_node+0x180/0x180 Apr 6 11:43:38 debworm kernel: [103549.361214] Code: a1 31 c0 e8 14 69 34 e0 0f 0b 48 c7 c1 38 e8 15 a1 ba 4b 07 00 00 48 c7 c6 e0 f8 15 a1 48 c7 c7 d0 e6 15 a1 31 c0 e8 f1 68 34 e0 <0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 85 ff 74 23 8b 87 Apr 6 11:43:38 debworm kernel: [103549.361423] RSP <ffff880096bf7b18> Apr 6 11:43:38 debworm kernel: [103549.368551] ---[ end trace 76340b50a326be77 ]---
This is a catastrophic error with reboot as the only mean to recover.
Actions