Project

General

Profile

Actions

Bug #11615

open

Memory Spike and page fault in slab allocator

Added by Adam Tygart almost 9 years ago. Updated almost 9 years ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Category:
fs/ceph
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

Attached is a backtrace and a memory graph from a kernel panic related to multiple simultaneous rsyncs of files into a cephfs volume.

Server info: Ceph 0.94.1 on CentOS 7.1

Client info: Gentoo, kernel 3.19.5


Files

Screen Shot 2015-05-13 at 11.30.53.png (345 KB) Screen Shot 2015-05-13 at 11.30.53.png Backtrace Adam Tygart, 05/13/2015 06:35 PM
Screen Shot 2015-05-13 at 11.48.21.png (33.9 KB) Screen Shot 2015-05-13 at 11.48.21.png graph of memory spike Adam Tygart, 05/13/2015 06:35 PM
Actions #1

Updated by Ilya Dryomov almost 9 years ago

Sorry I didn't see this earlier.

The panic is not in ceph_kvmalloc() but in the guts of slab memory allocator. It's hard to make conclusions from the backtrace alone, the best I can tell is it died from a page fault in mm/slub.c

244 static inline void *get_freepointer(struct kmem_cache *s, void *object)
245 {
246         return *(void **)(object + s->offset); <<--
247 }

on a invalid *object - 0x6232ae8. This value came from

2416         preempt_enable();
2417 
2418         object = c->freelist; <<--
2419         page = c->page;
2420         if (unlikely(!object || !node_match(page, node))) {

which suggests a busted struct kmem_cache_cpu. How exactly it got busted is an open question and it probably has to do with the sudden memory usage spike shown on your graph. More questions then answers for sure, if you can reproduce this on a latest kernel we can take a deeper look - a lot of changes went into cephfs caps-related bits since 3.19.5.

Actions #2

Updated by Ilya Dryomov almost 9 years ago

  • Subject changed from Memory Spike and Panic in ceph_kvmalloc to Memory Spike and page fault in slab allocator
  • Status changed from New to Need More Info
Actions

Also available in: Atom PDF