Actions
Bug #48763
openmds memory leak
Status:
Need More Info
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I have a possible memory leak in 14.2.10 mds. MDS suddenly uses 107GB Ram, before around 70.
Once mds starts its eating almost 3GB per second. Stabilizes at 107, pretty much stay exactly at 85.00% of total. Weird hard limit?
ceph.conf limits are much lower
mds_cache_memory_limit = 24111111100 mds_cache_size = 108146002
ceph 4155 22.3 84.9 112370780 111950096 ? Ssl Jan05 256:29 /usr/bin/ceph-mds -f --cluster ceph --id m1-1045558 --setuser ceph --setgroup ceph
--------------mds---------------- --mds_cache--- ------mds_log------ -mds_mem- ----mds_server----- mds_ -----objecter------ purg req rlat fwd inos caps exi imi |stry recy recd|subm evts segs repl|ino dn |hcr hcs hsr cre |sess|actv rd wr rdwr|purg| 13 0 0 7.6M 4.8M 0 0 | 79 0 0 | 0 105k 130 0 |7.6M 7.6M| 13 0 0 0 | 79 | 1 12 0 0 | 0 1.0k 50 0 7.6M 4.8M 0 0 | 94 0 0 | 28 105k 129 0 |7.6M 7.6M|1.0k 3 0 0 | 79 | 0 839 13 0 | 5 1.4k 0 0 7.6M 4.8M 0 0 | 78 0 0 |190 105k 129 0 |7.6M 7.6M|1.4k 9 0 0 | 79 | 0 957 104 0 | 44
{
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 52597138,
"bytes": 52597138
},
"bluestore_alloc": {
"items": 0,
"bytes": 0
},
"bluestore_cache_data": {
"items": 0,
"bytes": 0
},
"bluestore_cache_onode": {
"items": 0,
"bytes": 0
},
"bluestore_cache_other": {
"items": 0,
"bytes": 0
},
"bluestore_fsck": {
"items": 0,
"bytes": 0
},
"bluestore_txc": {
"items": 0,
"bytes": 0
},
"bluestore_writing_deferred": {
"items": 0,
"bytes": 0
},
"bluestore_writing": {
"items": 0,
"bytes": 0
},
"bluefs": {
"items": 0,
"bytes": 0
},
"buffer_anon": {
"items": 1388,
"bytes": 1379487
},
"buffer_meta": {
"items": 0,
"bytes": 0
},
"osd": {
"items": 0,
"bytes": 0
},
"osd_mapbl": {
"items": 0,
"bytes": 0
},
"osd_pglog": {
"items": 0,
"bytes": 0
},
"osdmap": {
"items": 6622,
"bytes": 249024
},
"osdmap_mapping": {
"items": 0,
"bytes": 0
},
"pgmap": {
"items": 0,
"bytes": 0
},
"mds_co": {
"items": 372987505,
"bytes": 22913939386
},
"unittest_1": {
"items": 0,
"bytes": 0
},
"unittest_2": {
"items": 0,
"bytes": 0
}
},
"total": {
"items": 425592653,
"bytes": 22968165035
}
}
}
Files
Updated by Anonymous over 3 years ago
cat /proc/meminfo MemTotal: 131716852 kB MemFree: 12847564 kB MemAvailable: 112647708 kB Buffers: 779124 kB Cached: 2895456 kB SwapCached: 0 kB Active: 18636676 kB Inactive: 99499472 kB Active(anon): 17382264 kB Inactive(anon): 1984 kB Active(file): 1254412 kB Inactive(file): 99497488 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 2544 kB Writeback: 0 kB AnonPages: 114461560 kB Mapped: 103096 kB Shmem: 2412 kB KReclaimable: 204148 kB Slab: 269608 kB SReclaimable: 204148 kB SUnreclaim: 65460 kB KernelStack: 15776 kB PageTables: 235300 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 65858424 kB Committed_AS: 121151636 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB Percpu: 4288 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 411492 kB DirectMap2M: 59043840 kB DirectMap1G: 76546048 kB
Updated by Patrick Donnelly over 3 years ago
- Status changed from New to Need More Info
Can you set `debug mds = 10` during the event so we can get an idea what the MDS is doing during this time, assuming you can reproduce it. It clearly was staying within its cache limit so the memory usage is happening elsewhere.
Updated by Yongseok Oh over 3 years ago
- File mds-ram-usage.png mds-ram-usage.png added
A similar situation has been found in our CephFS cluster configued with Nautilus 14.2.11. The RAM usage suddenly reaches 116GiB roughly. In the end, the MDS daemon has been crashed. There are a vast of "nicely exporting to mds.0" on the same directory between 7 to 10. It seems that the exporting task may incur this issue.
2021-01-17 11:22:14.955 7fc32cb45700 0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.101101* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985125 cv=985123/985123 state=1610874881|complete f(v278892 m2021-01-17 07:16:58.436354 8140=8140+0) n(v6520 rc2021-01-17 07:16:58.436354 b2810889 8140=8140+0) hs=8140+0,ss=0+0 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 exportbound=0 sticky=0 replicated=0 dirty=1 waiter=0 authpin=0 tempexporting=0 0x5664e731ed00]
2021-01-17 11:22:29.967 7fc32cb45700 0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.110001* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985061 cv=985059/985059 state=1610874881|complete f(v278892 m2021-01-16 19:15:37.275888 7985=7985+0) n(v6520 rc2021-01-16 19:15:37.275888 b2769103 7985=7985+0) hs=7985+0,ss=0+0 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 exportbound=0 sticky=0 replicated=0 dirty=1 waiter=0 authpin=0 tempexporting=0 0x5664e72d2500]
2021-01-17 11:22:29.967 7fc32cb45700 0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.110010* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985086 cv=985069/985069 state=1610874881|complete f(v278892 m2021-01-17 11:11:30.170399 8226=8226+0) n(v6520 rc2021-01-17 11:11:30.170399 b2866215 8226=8226+0) hs=8226+1,ss=0+0 dirty=2 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 exportbound=0 sticky=0 replicated=0 dirty=1 waiter=0 authpin=0 tempexporting=0 0x5664dd22d900]
2021-01-17 11:22:44.955 7fc32cb45700 0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.110101* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985146 cv=985131/985131 state=1610874881|complete f(v278892 m2021-01-17 11:11:30.284272 7993=7993+0) n(v6520 rc2021-01-17 11:11:30.284272 b2757949 7993=7993+0) hs=7993+1,ss=0+0 dirty=1 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 exportbound=0 sticky=0 replicated=0 dirty=1 waiter=0 authpin=0 tempexporting=0 0x5664e72d5700]
2021-01-17 11:22:59.946 7fc32cb45700 0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.111001* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985163 cv=985158/985158 state=1610874881|complete f(v278892 m2021-01-17 11:11:30.095218 7973=7973+0) n(v6520 rc2021-01-17 11:11:30.095218 b2759472 7973=7973+0) hs=7973+0,ss=0+0 dirty=1 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 exportbound=0 sticky=0 replicated=0 dirty=1 waiter=0 authpin=0 tempexporting=0 0x566509097900]
2021-01-17 11:23:09.958 7fc32cb45700 0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.111011* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985103 cv=985077/985077 state=1610874881|complete f(v278892 m2021-01-17 11:11:30.095218 8089=8089+0) n(v6520 rc2021-01-17 11:11:30.095218 b2791516 8089=8089+0) hs=8089+2,ss=0+0 dirty=2 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 exportbound=0 sticky=0 replicated=0 dirty=1 waiter=0 authpin=0 tempexporting=0 0x566509098300]
2021-01-17 11:23:24.946 7fc32cb45700 0 mds.1.migrator nicely exporting to mds.0 [dir 0x2000804a727.111111* /volumes/_nogroup/8d5d4732-e309-4fb5-9ddd-7e159a8af18d/share/ftp/transfer/identification_result/done/ [2,head] auth v=985190 cv=985185/985185 state=1610874881|complete f(v278892 m2021-01-17 11:11:30.265332 7922=7922+0) n(v6520 rc2021-01-17 11:11:30.265332 b2729155 7922=7922+0) hs=7922+0,ss=0+0 dirty=1 | child=1 sticky=0 replicated=0 dirty=1 authpin=0 tempexporting=0 0x566520294800]
in thread 7fc32cb45700 thread_name:ms_dispatch
1: (()+0xddc174) [0x564971585174]
2: (()+0xf5f0) [0x7fc333c965f0]
3: (MDSCacheObject::MDSCacheObject()+0x1b) [0x5649713b9b23]
4: (CDentry::CDentry(std::basic_string_view<char, std::char_traits<char> >, unsigned int, snapid_t, snapid_t)+0x44) [0x5649713b9d5c]
5: (CDir::add_null_dentry(std::basic_string_view<char, std::char_traits<char> >, snapid_t, snapid_t)+0x10c) [0x56497139a6a2]
6: (Migrator::decode_import_dir(ceph::buffer::v14_2_0::list::iterator_impl<true>&, int, CDir*, EImportStart*, LogSegment*, std::map<CInode*, std::map<client_t, Capability::Export, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Export> > >, std::less<CInode*>, std::allocator<std::pair<CInode* const, std::map<client_t, Capability::Export, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Export> > > > > >&, std::list<ScatterLock*, std::allocator<ScatterLock*> >&)+0x645) [0x56497134869d]
7: (Migrator::handle_export_dir(boost::intrusive_ptr<MExportDir const> const&)+0x72c) [0x564971343478]
8: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x275) [0x56497132afcf]
9: (MDSRank::handle_deferrable_message(boost::intrusive_ptr<Message const> const&)+0x51c) [0x5649710086c2]
10: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x379) [0x56497100744f]
11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0xdd) [0x564971006feb]
12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1f3) [0x564970fddd85]
13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xe9) [0x7fc336fe1eed]
14: (DispatchQueue::entry()+0x5ec) [0x7fc336fe0a8e]
15: (DispatchQueue::DispatchThread::entry()+0x1c) [0x7fc33714b0e8]
16: (Thread::entry_wrapper()+0x78) [0x7fc336dbae58]
17: (Thread::_entry_func(void*)+0x18) [0x7fc336dbadd6]
18: (()+0x7e65) [0x7fc333c8ee65]
19: (clone()+0x6d) [0x7fc33293c88d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Updated by wei qiaomiao over 3 years ago
- File mds cache.png mds cache.png added
A similar mds memory leak has been found in our CephFS cluster with ceph version 12.2.12。
Cache status and dump_mempools command display mds only uses about 4GB memory, but tcmalloc and top display mds use about 27GB memory. It is suspected that MDS has a memory leak out of mempool。
ceph.conf
mds_cache_memory_limit = 4294967296
The results of MDS cache status and heap stats are shown in the attachment。
Actions