Project

General

Profile

Feature #43215

kclient: send usage metrics to MDS

Added by Patrick Donnelly 8 months ago. Updated 13 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
% Done:

100%

Source:
Development
Tags:
Backport:
Reviewed:
Affected Versions:

Description

kernel client version of #36253


Subtasks

Feature #44533: kclient: add min/max perf latency supportResolvedXiubo Li

Feature #44534: kclient: add standard deviation support for the read/write/metadata latencyResolvedXiubo Li


Related issues

Related to fs - Feature #36253: cephfs: clients should send usage metadata to MDSs for administration/monitoring Resolved
Related to fs - Feature #24285: mgr: add module which displays current usage of file system (`fs top`) Fix Under Review
Related to fs - Feature #43423: mds: collect and show the dentry lease metric Fix Under Review
Related to fs - Feature #43435: kclient:send client provided metric flags in client metadata Fix Under Review

History

#1 Updated by Patrick Donnelly 8 months ago

  • Related to Feature #36253: cephfs: clients should send usage metadata to MDSs for administration/monitoring added

#2 Updated by Patrick Donnelly 8 months ago

  • Related to Feature #24285: mgr: add module which displays current usage of file system (`fs top`) added

#3 Updated by Xiubo Li 8 months ago

  • Status changed from New to In Progress

#4 Updated by Xiubo Li 8 months ago

For both dentry lease and caps perf metric we will only count the hit/miss info triggered from the vfs calls, for the cases like request reply handling and perodically ceph_trim_dentries() we will ignore them.

Currently only the debugfs is support and next will fulfill sending the mertic data to MDS.

The output will be:

# cat /sys/kernel/debug/ceph/536b8f9c-0d78-46bc-bbe9-c2ceb61c82a0.client4275/metric 
item              hit              miss
---------------------------------------
d_lease           19               0
i_caps            168              1

#5 Updated by Xiubo Li 8 months ago

Will add the entry count and cap count support:

    The output will be:

    item            total           hit             miss
    ----------------------------------------------------
    d_lease         11              141             7
    i_caps          10              28995740        13

#6 Updated by Xiubo Li 7 months ago

  • Related to Feature #43423: mds: collect and show the dentry lease metric added

#7 Updated by Xiubo Li 7 months ago

The output will be:


# cat /sys/kernel/debug/ceph/02a220c4-e4c6-43cd-9a84-993a020d2ffb.client4273/metrics
item          total           miss            hit
-------------------------------------------------
d_lease       125             120             2184

session       caps            miss            hit
-------------------------------------------------
0             63              692             7100
1             58              0               113
2             66              0               109

# ./bin/ceph fs perf stats | python -m json.tool
{
    "client_metadata": {
        "client.4273": {
            "IP": "v1:192.168.195.165",
            "hostname": "fedora1",
            "mount_point": "N/A",
            "root": "/" 
        }
    },
    "counters": [
        "cap_hit" 
    ],
    "global_counters": [
        "read_latency",
        "write_latency",
        "metadata_latency",
        "dentry_lease_hit" 
    ],
    "global_metrics": {
        "client.4273": [
            [
                0,
                0
            ],
            [
                0,
                0
            ],
            [
                0,
                0
            ],
            [
                2184,
                120
            ]
        ]
    },
    "metrics": {
        "delayed_ranks": [],
        "mds.0": {
            "client.4273": [
                [
                    7100,
                    692
                ]
            ]
        },
        "mds.1": {
            "client.4273": [
                [
                    113,
                    0
                ]
            ]
        },
        "mds.2": {
            "client.4273": [
                [
                    109,
                    0
                ]
            ]
        }
    }
}

#8 Updated by Xiubo Li 7 months ago

  • Status changed from In Progress to Fix Under Review

#9 Updated by Xiubo Li 7 months ago

  • Related to Feature #43435: kclient:send client provided metric flags in client metadata added

#10 Updated by Xiubo Li 7 months ago

  • Related to Bug #24823: mds: deadlock when setting config value via admin socket added

#11 Updated by Xiubo Li 7 months ago

  • Related to deleted (Bug #24823: mds: deadlock when setting config value via admin socket)

#12 Updated by Xiubo Li 7 months ago

  • Related to Feature #43435: kclient:send client provided metric flags in client metadata added

#13 Updated by Xiubo Li 7 months ago

  • Related to deleted (Feature #43435: kclient:send client provided metric flags in client metadata)

#14 Updated by Xiubo Li 7 months ago

Add read/write latency support:

#  ./bin/ceph fs perf stats | python -m json.tool
{
    "client_metadata": {
        "client.4836": {
            "IP": "v1:192.168.195.165",
            "hostname": "fedora1",
            "mount_point": "N/A",
            "root": "/" 
        }
    },
    "counters": [
        "cap_hit" 
    ],
    "global_counters": [
        "read_latency",
        "write_latency",
        "metadata_latency",
        "dentry_lease_hit" 
    ],
    "global_metrics": {
        "client.4836": [
            [
                0,
                15000000
            ],
            [
                13,
                77000000
            ],
            [
                0,
                0
            ],
            [
                6,
                0
            ]
        ]
    },
    "metrics": {
        "delayed_ranks": [],
        "mds.0": {
            "client.4836": [
                [
                    1222,
                    9
                ]
            ]
        },
        "mds.1": {
            "client.4836": [
                [
                    4,
                    0
                ]
            ]
        }
    }
}

#15 Updated by Patrick Donnelly 6 months ago

  • Target version changed from v15.0.0 to v16.0.0

#16 Updated by Xiubo Li 5 months ago

For the patch series [1] it only support the debugfs showing, will add sending the metric to ceph stuff after the depending ceph PR [2] get merged.

[1]: https://patchwork.kernel.org/project/ceph-devel/list/?series=253159
[2]: https://github.com/ceph/ceph/pull/26004

Thanks

#18 Updated by Xiubo Li 13 days ago

  • Project changed from fs to Linux kernel client
  • Category deleted (Performance/Resource Usage)
  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF