Project

General

Profile

Bug #48763

mds memory leak

Added by super xor about 1 year ago. Updated 12 months ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I have a possible memory leak in 14.2.10 mds. MDS suddenly uses 107GB Ram, before around 70.
Once mds starts its eating almost 3GB per second. Stabilizes at 107, pretty much stay exactly at 85.00% of total. Weird hard limit?

ceph.conf limits are much lower

mds_cache_memory_limit = 24111111100
mds_cache_size = 108146002

ceph 4155 22.3 84.9 112370780 111950096 ? Ssl Jan05 256:29 /usr/bin/ceph-mds -f --cluster ceph --id m1-1045558 --setuser ceph --setgroup ceph

--------------mds---------------- --mds_cache--- ------mds_log------ -mds_mem- ----mds_server----- mds_ -----objecter------ purg 
req  rlat fwd  inos caps exi  imi |stry recy recd|subm evts segs repl|ino  dn  |hcr  hcs  hsr  cre |sess|actv rd   wr   rdwr|purg|
 13    0    0  7.6M 4.8M   0    0 | 79    0    0 |  0  105k 130    0 |7.6M 7.6M| 13    0    0    0 | 79 |  1   12    0    0 |  0 
1.0k  50    0  7.6M 4.8M   0    0 | 94    0    0 | 28  105k 129    0 |7.6M 7.6M|1.0k   3    0    0 | 79 |  0  839   13    0 |  5 
1.4k   0    0  7.6M 4.8M   0    0 | 78    0    0 |190  105k 129    0 |7.6M 7.6M|1.4k   9    0    0 | 79 |  0  957  104    0 | 44 

{
    "mempool": {
        "by_pool": {
            "bloom_filter": {
                "items": 52597138,
                "bytes": 52597138
            },
            "bluestore_alloc": {
                "items": 0,
                "bytes": 0
            },
            "bluestore_cache_data": {
                "items": 0,
                "bytes": 0
            },
            "bluestore_cache_onode": {
                "items": 0,
                "bytes": 0
            },
            "bluestore_cache_other": {
                "items": 0,
                "bytes": 0
            },
            "bluestore_fsck": {
                "items": 0,
                "bytes": 0
            },
            "bluestore_txc": {
                "items": 0,
                "bytes": 0
            },
            "bluestore_writing_deferred": {
                "items": 0,
                "bytes": 0
            },
            "bluestore_writing": {
                "items": 0,
                "bytes": 0
            },
            "bluefs": {
                "items": 0,
                "bytes": 0
            },
            "buffer_anon": {
                "items": 1388,
                "bytes": 1379487
            },
            "buffer_meta": {
                "items": 0,
                "bytes": 0
            },
            "osd": {
                "items": 0,
                "bytes": 0
            },
            "osd_mapbl": {
                "items": 0,
                "bytes": 0
            },
            "osd_pglog": {
                "items": 0,
                "bytes": 0
            },
            "osdmap": {
                "items": 6622,
                "bytes": 249024
            },
            "osdmap_mapping": {
                "items": 0,
                "bytes": 0
            },
            "pgmap": {
                "items": 0,
                "bytes": 0
            },
            "mds_co": {
                "items": 372987505,
                "bytes": 22913939386
            },
            "unittest_1": {
                "items": 0,
                "bytes": 0
            },
            "unittest_2": {
                "items": 0,
                "bytes": 0
            }
        },
        "total": {
            "items": 425592653,
            "bytes": 22968165035
        }
    }
}

mds-ram-usage.png View (57.3 KB) Yongseok Oh, 01/18/2021 04:55 AM

mds cache.png View (124 KB) wei qiaomiao, 02/04/2021 09:03 AM

History

#1 Updated by super xor about 1 year ago

cat /proc/meminfo
MemTotal:       131716852 kB
MemFree:        12847564 kB
MemAvailable:   112647708 kB
Buffers:          779124 kB
Cached:          2895456 kB
SwapCached:            0 kB
Active:         18636676 kB
Inactive:       99499472 kB
Active(anon):   17382264 kB
Inactive(anon):     1984 kB
Active(file):    1254412 kB
Inactive(file): 99497488 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:              2544 kB
Writeback:             0 kB
AnonPages:      114461560 kB
Mapped:           103096 kB
Shmem:              2412 kB
KReclaimable:     204148 kB
Slab:             269608 kB
SReclaimable:     204148 kB
SUnreclaim:        65460 kB
KernelStack:       15776 kB
PageTables:       235300 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    65858424 kB
Committed_AS:   121151636 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:             4288 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      411492 kB
DirectMap2M:    59043840 kB
DirectMap1G:    76546048 kB

#2 Updated by Patrick Donnelly about 1 year ago

  • Status changed from New to Need More Info

Can you set `debug mds = 10` during the event so we can get an idea what the MDS is doing during this time, assuming you can reproduce it. It clearly was staying within its cache limit so the memory usage is happening elsewhere.

#3 Updated by Yongseok Oh about 1 year ago

A similar situation has been found in our CephFS cluster configued with Nautilus 14.2.11. The RAM usage suddenly reaches 116GiB roughly. In the end, the MDS daemon has been crashed. There are a vast of "nicely exporting to mds.0" on the same directory between 7 to 10. It seems that the exporting task may incur this issue.

2021-01-17 11:22:14.955 7fc32cb45700  0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.101101* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985125 cv=985123/985123 state=1610874881|complete f(v278892 m2021-01-17 07:16:58.436354 8140=8140+0) n(v6520 rc2021-01-17 07:16:58.436354 b2810889 8140=8140+0) hs=8140+0,ss=0+0 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 exportbound=0 sticky=0 replicated=0 dirty=1 waiter=0 authpin=0 tempexporting=0 0x5664e731ed00]
2021-01-17 11:22:29.967 7fc32cb45700  0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.110001* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985061 cv=985059/985059 state=1610874881|complete f(v278892 m2021-01-16 19:15:37.275888 7985=7985+0) n(v6520 rc2021-01-16 19:15:37.275888 b2769103 7985=7985+0) hs=7985+0,ss=0+0 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 exportbound=0 sticky=0 replicated=0 dirty=1 waiter=0 authpin=0 tempexporting=0 0x5664e72d2500]
2021-01-17 11:22:29.967 7fc32cb45700  0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.110010* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985086 cv=985069/985069 state=1610874881|complete f(v278892 m2021-01-17 11:11:30.170399 8226=8226+0) n(v6520 rc2021-01-17 11:11:30.170399 b2866215 8226=8226+0) hs=8226+1,ss=0+0 dirty=2 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 exportbound=0 sticky=0 replicated=0 dirty=1 waiter=0 authpin=0 tempexporting=0 0x5664dd22d900]
2021-01-17 11:22:44.955 7fc32cb45700  0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.110101* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985146 cv=985131/985131 state=1610874881|complete f(v278892 m2021-01-17 11:11:30.284272 7993=7993+0) n(v6520 rc2021-01-17 11:11:30.284272 b2757949 7993=7993+0) hs=7993+1,ss=0+0 dirty=1 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 exportbound=0 sticky=0 replicated=0 dirty=1 waiter=0 authpin=0 tempexporting=0 0x5664e72d5700]
2021-01-17 11:22:59.946 7fc32cb45700  0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.111001* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985163 cv=985158/985158 state=1610874881|complete f(v278892 m2021-01-17 11:11:30.095218 7973=7973+0) n(v6520 rc2021-01-17 11:11:30.095218 b2759472 7973=7973+0) hs=7973+0,ss=0+0 dirty=1 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 exportbound=0 sticky=0 replicated=0 dirty=1 waiter=0 authpin=0 tempexporting=0 0x566509097900]
2021-01-17 11:23:09.958 7fc32cb45700  0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.111011* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985103 cv=985077/985077 state=1610874881|complete f(v278892 m2021-01-17 11:11:30.095218 8089=8089+0) n(v6520 rc2021-01-17 11:11:30.095218 b2791516 8089=8089+0) hs=8089+2,ss=0+0 dirty=2 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 exportbound=0 sticky=0 replicated=0 dirty=1 waiter=0 authpin=0 tempexporting=0 0x566509098300]
2021-01-17 11:23:24.946 7fc32cb45700  0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.111111* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985190 cv=985185/985185 state=1610874881|complete f(v278892 m2021-01-17 11:11:30.265332 7922=7922+0) n(v6520 rc2021-01-17 11:11:30.265332 b2729155 7922=7922+0) hs=7922+0,ss=0+0 dirty=1 | child=1 sticky=0 replicated=0 dirty=1 authpin=0 tempexporting=0 0x566520294800]

 in thread 7fc32cb45700 thread_name:ms_dispatch

 1: (()+0xddc174) [0x564971585174]
 2: (()+0xf5f0) [0x7fc333c965f0]
 3: (MDSCacheObject::MDSCacheObject()+0x1b) [0x5649713b9b23]
 4: (CDentry::CDentry(std::basic_string_view<char, std::char_traits<char> >, unsigned int, snapid_t, snapid_t)+0x44) [0x5649713b9d5c]
 5: (CDir::add_null_dentry(std::basic_string_view<char, std::char_traits<char> >, snapid_t, snapid_t)+0x10c) [0x56497139a6a2]
 6: (Migrator::decode_import_dir(ceph::buffer::v14_2_0::list::iterator_impl<true>&, int, CDir*, EImportStart*, LogSegment*, std::map<CInode*, std::map<client_t, Capability::Export, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Export> > >, std::less<CInode*>, std::allocator<std::pair<CInode* const, std::map<client_t, Capability::Export, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Export> > > > > >&, std::list<ScatterLock*, std::allocator<ScatterLock*> >&)+0x645) [0x56497134869d]
 7: (Migrator::handle_export_dir(boost::intrusive_ptr<MExportDir const> const&)+0x72c) [0x564971343478]
 8: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x275) [0x56497132afcf]
 9: (MDSRank::handle_deferrable_message(boost::intrusive_ptr<Message const> const&)+0x51c) [0x5649710086c2]
 10: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x379) [0x56497100744f]
 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0xdd) [0x564971006feb]
 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1f3) [0x564970fddd85]
 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xe9) [0x7fc336fe1eed]
 14: (DispatchQueue::entry()+0x5ec) [0x7fc336fe0a8e]
 15: (DispatchQueue::DispatchThread::entry()+0x1c) [0x7fc33714b0e8]
 16: (Thread::entry_wrapper()+0x78) [0x7fc336dbae58]
 17: (Thread::_entry_func(void*)+0x18) [0x7fc336dbadd6]
 18: (()+0x7e65) [0x7fc333c8ee65]
 19: (clone()+0x6d) [0x7fc33293c88d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

#4 Updated by wei qiaomiao 12 months ago

A similar mds memory leak has been found in our CephFS cluster with ceph version 12.2.12。
Cache status and dump_mempools command display mds only uses about 4GB memory, but tcmalloc and top display mds use about 27GB memory. It is suspected that MDS has a memory leak out of mempool。
ceph.conf

mds_cache_memory_limit = 4294967296

The results of MDS cache status and heap stats are shown in the attachment。

Also available in: Atom PDF