Project

General

Profile

Actions
Copy link

Bug #40951

closed

(think this is bug) Why CephFS metadata pool usage is only growing?

Added by Konstantin Shalygin almost 5 years ago. Updated almost 5 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v12.2.12
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Can't find a clue: singe MDS fs without actually usage, but metadata is only growing (screenshot attached). MDS with default settings, `mds_cache_memory_limit` is only changed to "25769803776". May be meta (log?) trimmed later and just not yet reach some threshold?

fs_meta pool is filestore OSD.

# ceph fs status
cephfs - 9 clients
======
+------+--------+-----------+---------------+-------+-------+
| Rank | State  |    MDS    |    Activity   |  dns  |  inos |
+------+--------+-----------+---------------+-------+-------+
|  0   | active | ceph-mds0 | Reqs:    0 /s |  154  |   38  |
+------+--------+-----------+---------------+-------+-------+
+--------------+----------+-------+-------+
|     Pool     |   type   |  used | avail |
+--------------+----------+-------+-------+
|   fs_meta    | metadata |  122M | 10.9T |
|   fs_data    |   data   | 23.0G | 67.8T |
| fs_data_nvme |   data   |    0  | 10.9T |
+--------------+----------+-------+-------+

+-------------+
| Standby MDS |
+-------------+
+-------------+
MDS version: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)

MDS permissions:

[mds.ceph-mds0]
        key = <snip></snap>
        caps mds = "allow *" 
        caps mgr = "profile mds" 
        caps mon = "profile mds" 
        caps osd = "allow *"

kernel client permissions:

# kernel client 3.10.0-957.12.2.el7.x86_64
client.mds_ovirt_iso
        key: <snip></snap>
        caps: [mds] allow rw path=/iso
        caps: [mon] allow r
        caps: [osd] allow rwx pool=fs_data, allow rwx pool=fs_data_nvme

# samba vfs luminous 12.2.12 client
client.mds_samba
        key: <snip></snap>
        caps: [mds] allow rwp
        caps: [mon] allow r
        caps: [osd] allow rwx pool=fs_data, allow rw pool=fs_data_nvme

fs dump:

dumped fsmap epoch 17
e17
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2}
legacy client fscid: 1

Filesystem 'cephfs' (1)
fs_name cephfs
epoch   17
flags   c
created 2019-07-09 15:38:01.614140
modified        2019-07-24 11:47:34.048711
tableserver     0
root    0
session_timeout 60
session_autoclose       300
max_file_size   1099511627776
last_failure    0
last_failure_osd_epoch  56984
compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2}
max_mds 1
in      0
up      {0=377152890}
failed
damaged
stopped
data_pools      [21,23]
metadata_pool   22
inline_data     disabled
balancer
standby_count_wanted    0
377152890:      172.16.16.5:6800/2469778625 'ceph-mds0' mds.0.14 up:active seq 5

fs data pool on this week:

fs meta pool on this week:

Files

Download all files
fs_data.png (84.8 KB) fs_data.png Konstantin Shalygin, 07/25/2019 09:26 AM
fs_meta.png (107 KB) fs_meta.png Konstantin Shalygin, 07/25/2019 09:26 AM
another_cluster.jpg (43.1 KB) another_cluster.jpg Konstantin Shalygin, 07/25/2019 09:38 AM
fs_data_26.07.19.png (78.5 KB) fs_data_26.07.19.png Konstantin Shalygin, 07/26/2019 03:23 AM
fs_meta_26.07.19.png (154 KB) fs_meta_26.07.19.png Konstantin Shalygin, 07/26/2019 03:23 AM
fs_meta_26.07.19_test.png (69.1 KB) fs_meta_26.07.19_test.png Konstantin Shalygin, 07/26/2019 06:17 AM
fs_data_26.07.19_test.png (68.4 KB) fs_data_26.07.19_test.png Konstantin Shalygin, 07/26/2019 06:17 AM
cachedump (986 KB) cachedump 26.07.2019 MDS cache dump Konstantin Shalygin, 07/26/2019 06:42 AM
cache_drop.png (74 KB) cache_drop.png Konstantin Shalygin, 07/26/2019 09:25 AM
fs_meta_2019-07-29.png (29 KB) fs_meta_2019-07-29.png Konstantin Shalygin, 07/29/2019 03:55 AM
Actions
Copy link

Also available in: Atom PDF