Ceph » CephFS

On IRC Patrick said:

19:00 < batrick> MDSMap epoch received
19:00 < batrick> mds failover in process
19:00 < batrick> would be a good start

We already have this dout() message already when we get a new map:

        dout("check_new_map new %u old %u\n",
             newmap->m_epoch, oldmap->m_epoch);

By mds failover, do you mean the point where the client gives up on an MDS and migrates to another? Or do you mean some sort of export/import activity?

Jeff Layton wrote:

We already have this dout() message already when we get a new map:

[...]

By mds failover, do you mean the point where the client gives up on an MDS and migrates to another? Or do you mean some sort of export/import activity?

If an MDS is recovering (e.g. up:reconnect/up:replay/etc.). The client could note that and perhaps note how many requests it (will) need to replay, caps with that rank, etc.

I can see where to add such a message, but I'm not that familiar with all of the different MDS states. Which ones, specifically, should we notify the user of? You mentioned:

up:replay
up:reconnect

Jeff Layton wrote:

I can see where to add such a message, but I'm not that familiar with all of the different MDS states. Which ones, specifically, should we notify the user of? You mentioned:
[...]
...but there are a bunch of others. What about up and out states, or up:resolve, up:rejoin, etc.?

All state transitions for a rank should be reported. (IMO)

I'm not convinced it's beneficial to spam the kernel's ring buffer with these messages. Ceph is already a bit too chatty with this sort of thing as it is...

Since you said you want this for debugging purposes, do you need anything beyond the dout() message that already reports this?

                dout("check_new_map mds%d state %s%s -> %s%s (session %s)\n",
                     i, ceph_mds_state_name(oldstate),
                     ceph_mdsmap_is_laggy(oldmap, i) ? " (laggy)" : "",
                     ceph_mds_state_name(newstate),
                     ceph_mdsmap_is_laggy(newmap, i) ? " (laggy)" : "",
                     ceph_session_state_name(s->s_state));

If you want this mainly for teuthology testing, what may be best is to create a new script that gets run on the kclient before a test that can selectively enable certain debug messages like this one.

Would that be acceptable?

Jeff Layton wrote:

I'm not convinced it's beneficial to spam the kernel's ring buffer with these messages. Ceph is already a bit too chatty with this sort of thing as it is...

Since you said you want this for debugging purposes, do you need anything beyond the dout() message that already reports this?

[...]

If you want this mainly for teuthology testing, what may be best is to create a new script that gets run on the kclient before a test that can selectively enable certain debug messages like this one.

How do we do that?

Would that be acceptable?

Probably.

Basically, we'd just need to do something like this in a script before the test runs:

#!/bin/sh
modprobe ceph
echo 'module ceph func check_new_map +p' > /sys/kernel/debug/dynamic_debug/control

That would turn on all of the dout() messages in the check_new_map() function. We can get more specific than that using line numbers too, but that gets to be a bit more fragile in the face of kernel changes.

We could extend this idea to something more general too:

https://www.kernel.org/doc/html/latest/admin-guide/dynamic-debug-howto.html

Ilya, do you have anything like this already for rbd testing?

Oh, actually, we can use the format directive, so something like this would probably also work:

#!/bin/sh
modprobe ceph
echo 'module ceph format "check_new_map mds" +p' > /sys/kernel/debug/dynamic_debug/control

No, we don't enable douts anywhere in the krbd suite.