Project

General

Profile

Actions

Feature #43215

closed

kclient: send usage metrics to MDS

Added by Patrick Donnelly over 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
% Done:

100%

Source:
Development
Tags:
Backport:
Reviewed:
Affected Versions:

Description

kernel client version of #36253


Subtasks 2 (0 open2 closed)

Feature #44533: kclient: add min/max perf latency supportResolvedXiubo Li

Actions
Feature #44534: kclient: add standard deviation support for the read/write/metadata latencyResolvedXiubo Li

Actions

Related issues 4 (0 open4 closed)

Related to CephFS - Feature #36253: cephfs: clients should send usage metadata to MDSs for administration/monitoringResolvedVenky Shankar

Actions
Related to CephFS - Feature #24285: mgr: add module which displays current usage of file system (`fs top`)ResolvedVenky Shankar

Actions
Related to CephFS - Feature #43423: mds: collect and show the dentry lease metricResolvedXiubo Li

Actions
Related to CephFS - Feature #43435: kclient:send client provided metric flags in client metadataResolvedXiubo Li

Actions
Actions #1

Updated by Patrick Donnelly over 4 years ago

  • Related to Feature #36253: cephfs: clients should send usage metadata to MDSs for administration/monitoring added
Actions #2

Updated by Patrick Donnelly over 4 years ago

  • Related to Feature #24285: mgr: add module which displays current usage of file system (`fs top`) added
Actions #3

Updated by Xiubo Li over 4 years ago

  • Status changed from New to In Progress
Actions #4

Updated by Xiubo Li over 4 years ago

For both dentry lease and caps perf metric we will only count the hit/miss info triggered from the vfs calls, for the cases like request reply handling and perodically ceph_trim_dentries() we will ignore them.

Currently only the debugfs is support and next will fulfill sending the mertic data to MDS.

The output will be:

# cat /sys/kernel/debug/ceph/536b8f9c-0d78-46bc-bbe9-c2ceb61c82a0.client4275/metric 
item              hit              miss
---------------------------------------
d_lease           19               0
i_caps            168              1

Actions #5

Updated by Xiubo Li over 4 years ago

Will add the entry count and cap count support:

    The output will be:

    item            total           hit             miss
    ----------------------------------------------------
    d_lease         11              141             7
    i_caps          10              28995740        13
Actions #6

Updated by Xiubo Li over 4 years ago

  • Related to Feature #43423: mds: collect and show the dentry lease metric added
Actions #7

Updated by Xiubo Li over 4 years ago

The output will be:


# cat /sys/kernel/debug/ceph/02a220c4-e4c6-43cd-9a84-993a020d2ffb.client4273/metrics
item          total           miss            hit
-------------------------------------------------
d_lease       125             120             2184

session       caps            miss            hit
-------------------------------------------------
0             63              692             7100
1             58              0               113
2             66              0               109

# ./bin/ceph fs perf stats | python -m json.tool
{
    "client_metadata": {
        "client.4273": {
            "IP": "v1:192.168.195.165",
            "hostname": "fedora1",
            "mount_point": "N/A",
            "root": "/" 
        }
    },
    "counters": [
        "cap_hit" 
    ],
    "global_counters": [
        "read_latency",
        "write_latency",
        "metadata_latency",
        "dentry_lease_hit" 
    ],
    "global_metrics": {
        "client.4273": [
            [
                0,
                0
            ],
            [
                0,
                0
            ],
            [
                0,
                0
            ],
            [
                2184,
                120
            ]
        ]
    },
    "metrics": {
        "delayed_ranks": [],
        "mds.0": {
            "client.4273": [
                [
                    7100,
                    692
                ]
            ]
        },
        "mds.1": {
            "client.4273": [
                [
                    113,
                    0
                ]
            ]
        },
        "mds.2": {
            "client.4273": [
                [
                    109,
                    0
                ]
            ]
        }
    }
}

Actions #8

Updated by Xiubo Li over 4 years ago

  • Status changed from In Progress to Fix Under Review
Actions #9

Updated by Xiubo Li over 4 years ago

  • Related to Feature #43435: kclient:send client provided metric flags in client metadata added
Actions #10

Updated by Xiubo Li over 4 years ago

  • Related to Bug #24823: mds: deadlock when setting config value via admin socket added
Actions #11

Updated by Xiubo Li over 4 years ago

  • Related to deleted (Bug #24823: mds: deadlock when setting config value via admin socket)
Actions #12

Updated by Xiubo Li over 4 years ago

  • Related to Feature #43435: kclient:send client provided metric flags in client metadata added
Actions #13

Updated by Xiubo Li over 4 years ago

  • Related to deleted (Feature #43435: kclient:send client provided metric flags in client metadata)
Actions #14

Updated by Xiubo Li over 4 years ago

Add read/write latency support:

#  ./bin/ceph fs perf stats | python -m json.tool
{
    "client_metadata": {
        "client.4836": {
            "IP": "v1:192.168.195.165",
            "hostname": "fedora1",
            "mount_point": "N/A",
            "root": "/" 
        }
    },
    "counters": [
        "cap_hit" 
    ],
    "global_counters": [
        "read_latency",
        "write_latency",
        "metadata_latency",
        "dentry_lease_hit" 
    ],
    "global_metrics": {
        "client.4836": [
            [
                0,
                15000000
            ],
            [
                13,
                77000000
            ],
            [
                0,
                0
            ],
            [
                6,
                0
            ]
        ]
    },
    "metrics": {
        "delayed_ranks": [],
        "mds.0": {
            "client.4836": [
                [
                    1222,
                    9
                ]
            ]
        },
        "mds.1": {
            "client.4836": [
                [
                    4,
                    0
                ]
            ]
        }
    }
}

Actions #15

Updated by Patrick Donnelly about 4 years ago

  • Target version changed from v15.0.0 to v16.0.0
Actions #16

Updated by Xiubo Li about 4 years ago

For the patch series [1] it only support the debugfs showing, will add sending the metric to ceph stuff after the depending ceph PR [2] get merged.

[1]: https://patchwork.kernel.org/project/ceph-devel/list/?series=253159
[2]: https://github.com/ceph/ceph/pull/26004

Thanks

Actions #18

Updated by Xiubo Li almost 4 years ago

  • Project changed from CephFS to Linux kernel client
  • Category deleted (Performance/Resource Usage)
  • Status changed from Fix Under Review to Resolved
Actions

Also available in: Atom PDF