Ceph » CephFS

Not sure if required but I wanted to add some more information, while running cephfs-top the mgr module crashes all the time:

storage01:~ # ceph health detail
HEALTH_WARN 23 mgr modules have recently crashed
[WRN] RECENT_MGR_MODULE_CRASH: 23 mgr modules have recently crashed
    mgr module stats crashed in daemon mgr.storage01.ygsvte on host storage01 at 2023-05-04T11:36:48.948046Z
    mgr module stats crashed in daemon mgr.storage01.ygsvte on host storage01 at 2023-05-04T11:35:33.897963Z
    mgr module stats crashed in daemon mgr.storage01.ygsvte on host storage01 at 2023-05-04T11:36:13.927182Z
    mgr module stats crashed in daemon mgr.storage01.ygsvte on host storage01 at 2023-05-04T11:35:13.908107Z

But I get output from cephfs-top so it kind of works but leaves all these crashes.

Eugen Block wrote:

Not sure if required but I wanted to add some more information, while running cephfs-top the mgr module crashes all the time:

[...]

But I get output from cephfs-top so it kind of works but leaves all these crashes.

I don't see these crashes in `ceph health detail` with the latest cephfs-top code. Can you confirm if it's related crash? Could you please attach the mgr logs?

This is kind of strange, when I initially wanted to test cephfs-top I chose a different virtual ceph cluster which already was running ceph version 17.2.6 (images from quay.io) on openSUSE Leap 15.4, but I got some curses error messages. Then I chose to use above one-node cluster where I got cephfs-top to work but with mentioned stack trace. Now in the other cluster I managed to mitigate the curses error and now cephfs-top works there as well, but without breaking the mgr module. Both systems are Leap 15.4 with ceph version 17.2.6:

# machine with stack trace
storage01:~ # ceph versions
{
[...]
    "overall": {
        "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 9
    }
}

storage01:~ # grep NAME /etc/os-release 
NAME="openSUSE Leap" 
PRETTY_NAME="openSUSE Leap 15.4" 
CPE_NAME="cpe:/o:opensuse:leap:15.4" 

storage01:~ # rpm -qa | grep cephfs-top
cephfs-top-17.2.6.248+gad656d572cb-lp154.2.1.noarch

# machine without stack trace
nautilus:~ # ceph versions
{
[...]
    "overall": {
        "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 21
    }
}

nautilus:~ # grep NAME /etc/os-release 
NAME="openSUSE Leap" 
PRETTY_NAME="openSUSE Leap 15.4" 
CPE_NAME="cpe:/o:opensuse:leap:15.4" 

nautilus:~ # rpm -qa | grep cephfs-top
cephfs-top-17.2.6.248+gad656d572cb-lp154.2.1.noarch

So they are practically identical, only the nautilus machine has been installed with Leap 15.1 and was upgraded to Quincy while the other was installed with Leap 15.3 and Pacific. Any idea what the difference could be here?

Eugen Block wrote:

This is kind of strange, when I initially wanted to test cephfs-top I chose a different virtual ceph cluster which already was running ceph version 17.2.6 (images from quay.io) on openSUSE Leap 15.4, but I got some curses error messages. Then I chose to use above one-node cluster where I got cephfs-top to work but with mentioned stack trace. Now in the other cluster I managed to mitigate the curses error and now cephfs-top works there as well, but without breaking the mgr module. Both systems are Leap 15.4 with ceph version 17.2.6:

[...]

So they are practically identical, only the nautilus machine has been installed with Leap 15.1 and was upgraded to Quincy while the other was installed with Leap 15.3 and Pacific. Any idea what the difference could be here?

Leap 15.1 uses kernel v4.12. You need Kernel v5.14 to have the complete set of 'perf stats' patches.
Could you please send the output of 'session ls' to check?

Both VMs use the same kernel version (they are not running 15.1 anymore, both have been upgraded to 15.4 on the way to quincy):

storage01:~ # uname -r
5.14.21-150400.24.60-default

nautilus:~ # uname -r
5.14.21-150400.24.60-default

The session ls output would only show mounted clients, but both these storage nodes do not have the cephfs mounted. Or do you suspect that older clients mounting the cephfs could cause the stack trace? Could you please clarify? In the cluster with a stack trace there are older kernel clients, for example:

        "client_metadata": {
            "client_features": {
                "feature_bits": "0x0000000000003bff" 
            },
            "metric_spec": {
                "metric_flags": {
                    "feature_bits": "0x" 
                }
            },
            "entity_id": "nova-mount",
            "hostname": "compute01",
            "kernel_version": "5.3.18-lp152.106-default",
            "root": "/openstack-cluster/nova-instances" 

Eugen Block wrote:

Both VMs use the same kernel version (they are not running 15.1 anymore, both have been upgraded to 15.4 on the way to quincy):

storage01:~ # uname -r
5.14.21-150400.24.60-default

nautilus:~ # uname -r
5.14.21-150400.24.60-default

So is the above mentioned issue (in Description) exist in your cluster?

The session ls output would only show mounted clients, but both these storage nodes do not have the cephfs mounted. Or do you suspect that older clients mounting the cephfs could cause the stack trace? Could you please clarify? In the cluster with a stack trace there are older kernel clients, for example:
[...]

I need the output from the older cluster, where you hit the issue.