Ceph » CephFS

Jos Collin wrote:

@Venky,

When we disable mirroring using 'ceph mgr module disable mirroring' and then enable using 'ceph fs mirror enable <primary_fs_name>', the 'fs mirror status' command doesn't output the "state: failed" status. The mirroring seems failed, but the "state: failed" doesn't show.

It doesn't hit 'FSMirror::init', so that m_init_failed is set to True.

So, some change likely broke the part where m_init_failed gets set or there is a delayed check for m_init_failed. I would start with the latter and hash that out first.

Venky Shankar wrote:

Jos Collin wrote:

@Venky,

When we disable mirroring using 'ceph mgr module disable mirroring' and then enable using 'ceph fs mirror enable <primary_fs_name>', the 'fs mirror status' command doesn't output the "state: failed" status. The mirroring seems failed, but the "state: failed" doesn't show.

It doesn't hit 'FSMirror::init', so that m_init_failed is set to True.

So, some change likely broke the part where m_init_failed gets set or there is a delayed check for m_init_failed. I would start with the latter and hash that out first.

It doesn't seem to be a delayed check for m_init_failed. The debug always shows m_init_failed = false, even after waiting for a few minutes:

 460  void FSMirror::mirror_status(Formatter *f) {
 461    std::scoped_lock locker(m_lock);
!462    f->open_object_section("status");
 463    if (m_init_failed) {
 464      f->dump_string("state", "failed");
 465    } else if (is_blocklisted(locker)) {
 466      f->dump_string("state", "blocklisted");
 467    } else {
 468      // dump rados addr for blocklist test
 469      f->dump_string("rados_inst", m_addrs);
 470      f->open_object_section("peers");
 471      for ([[maybe_unused]] auto &[peer, peer_replayer] : m_peer_replayers) {
 472        peer.dump(f);
 473      }
─── Variables ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
arg this = 0x5601b6cfe000: {m_cct = 0x5601b1e9a000,m_filesystem = {fscid = 1,fs_name = "a"},m_pool_id …, f = 0x5601b6d28e00: {_vptr.Formatter = 0x7fb7cf694b30 <vtable for ceph::JSONFormatter+16>}
loc locker = {_M_device = @0x5601b6cfe060}
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
>>> p m_init_failed
$4 = false

The command:

./bin/ceph --admin-daemon asok/client.admin.342404.asok fs mirror status a@1

works, but it doesn't show the 'state'. So this should be a problem with failed to set m_init_failed value. Checking that.

Jos Collin wrote:

Venky Shankar wrote:

Jos Collin wrote:

@Venky,

When we disable mirroring using 'ceph mgr module disable mirroring' and then enable using 'ceph fs mirror enable <primary_fs_name>', the 'fs mirror status' command doesn't output the "state: failed" status. The mirroring seems failed, but the "state: failed" doesn't show.

It doesn't hit 'FSMirror::init', so that m_init_failed is set to True.

So, some change likely broke the part where m_init_failed gets set or there is a delayed check for m_init_failed. I would start with the latter and hash that out first.

It doesn't seem to be a delayed check for m_init_failed. The debug always shows m_init_failed = false, even after waiting for a few minutes:
[...]

The command:
[...]
works, but it doesn't show the 'state'. So this should be a problem with failed to set m_init_failed value. Checking that.

Do you have the run link? I'll have a look now.

Venky Shankar wrote:

Jos Collin wrote:

Venky Shankar wrote:

Jos Collin wrote:

@Venky,

When we disable mirroring using 'ceph mgr module disable mirroring' and then enable using 'ceph fs mirror enable <primary_fs_name>', the 'fs mirror status' command doesn't output the "state: failed" status. The mirroring seems failed, but the "state: failed" doesn't show.

It doesn't hit 'FSMirror::init', so that m_init_failed is set to True.

So, some change likely broke the part where m_init_failed gets set or there is a delayed check for m_init_failed. I would start with the latter and hash that out first.

It doesn't seem to be a delayed check for m_init_failed. The debug always shows m_init_failed = false, even after waiting for a few minutes:
[...]

The command:
[...]
works, but it doesn't show the 'state'. So this should be a problem with failed to set m_init_failed value. Checking that.

Do you have the run link? I'll have a look now.

The above finding is from the vstart cluster reproduced.
I was also looking at https://pulpito.ceph.com/rishabh-2023-06-23_17:37:30-fs-wip-rishabh-improvements-authmon-distro-default-smithi/7313862.

Jos Collin wrote:

Venky Shankar wrote:

Jos Collin wrote:

Venky Shankar wrote:

Jos Collin wrote:

@Venky,

When we disable mirroring using 'ceph mgr module disable mirroring' and then enable using 'ceph fs mirror enable <primary_fs_name>', the 'fs mirror status' command doesn't output the "state: failed" status. The mirroring seems failed, but the "state: failed" doesn't show.

It doesn't hit 'FSMirror::init', so that m_init_failed is set to True.

So, some change likely broke the part where m_init_failed gets set or there is a delayed check for m_init_failed. I would start with the latter and hash that out first.

It doesn't seem to be a delayed check for m_init_failed. The debug always shows m_init_failed = false, even after waiting for a few minutes:
[...]

The command:
[...]
works, but it doesn't show the 'state'. So this should be a problem with failed to set m_init_failed value. Checking that.

Do you have the run link? I'll have a look now.

The above finding is from the vstart cluster reproduced.
I was also looking at https://pulpito.ceph.com/rishabh-2023-06-23_17:37:30-fs-wip-rishabh-improvements-authmon-distro-default-smithi/7313862.

No problem. Just upload the cephfs-mirror debug logs from the vstart cluster when running this test.