Ceph » CephFS

please try newer kernel

The "sen2agriprod" server actually runs on centOS7 (kernel 3.10.0) which is in the recommended platforms.

If you think that I have to update I will give a try on the other servers (this is not the official opensuse way, but ok if mandatory)

Thanks

this ticket is now rather old. do you mind, if I just close it?

Do you restart the mds on sen2agriprod? Or just you restart all mds? We have the similar case, loosing all the mds's metadata(use the command `ceph mds metadata` and get just name pair), Then we restart all the mds, the cluster work's again and the command `ceph mds metaddata` return the correct metadata. Before the case, I just enable the prometheus module, which result mgr respam, and mgr ask mon to give its mds metadata, which failing. That's why command 'ceph fs status' failed, it doesn`t have mds metadata. I guess before I enable prometheus module, the cluster already have this problem.

FYI - restarting the MDS fixes the issue.

It seems the MDS can miss sending beacon in up:boot state. This state encodes the MDS metadata and includes that in the beacon message to the monitor. In MDSDaemon::handle_mds_map():

  if (whoami == MDS_RANK_NONE) {
    // We do not hold a rank:                                                                                                                                                                                                                                                                                               
    dout(10) <<  __func__ << ": handling map in rankless mode" << dendl;

    if (new_state == DS::STATE_STANDBY) {
      /* Note: STATE_BOOT is never an actual state in the FSMap. The Monitors                                                                                                                                                                                                                                               
       * generally mark a new MDS as STANDBY (although it's possible to                                                                                                                                                                                                                                                     
       * immediately be assigned a rank).                                                                                                                                                                                                                                                                                   
       */
      if (old_state == DS::STATE_NULL) {
        dout(1) << "Monitors have assigned me to become a standby." << dendl;
        beacon.set_want_state(*mdsmap, new_state);
      } else if (old_state == DS::STATE_STANDBY) {
        dout(5) << "I am still standby" << dendl;
      }

The MDS can switch from STATE_NULL to STATE_STANDBY from the above code block. When this happens the MDS doesn't go through STATE_BOOT (up:boot). What's unknown is under what circumstances this path can be hit.

Venky Shankar wrote:

It seems the MDS can miss sending beacon in up:boot state. This state encodes the MDS metadata and includes that in the beacon message to the monitor. In MDSDaemon::handle_mds_map():

[...]

The MDS can switch from STATE_NULL to STATE_STANDBY from the above code block. When this happens the MDS doesn't go through STATE_BOOT (up:boot). What's unknown is under what circumstances this path can be hit.

STATE_NULL comes from the previous mdsmap ("oldmap" in the code). That is, the MDS is not in the map at all. Once the mons receive the first beacon with want_state STATE_BOOT, the mds is added as a standby (or immediately promoted if a rank is available). At no point is the MDS in the mdsmap with state STATE_BOOT.

So, I don't think that's the cause of the bug here.

The MDS is identified using a nonce as well as an IP in the map, right? After the containerized OSDs managed to clobber their bluestores, I wonder about multiple MDS instances running in close enough proximity to confuse things, but I don't think that could break this...

Was discussion about this tracker with Patrick - there are separate paxos proposals for fsmap update and the metadata update. If the metadata update fails the MDS still transitions to the next state. Theoretical case atm. Requires test with some paxos failure injection to conclude.

#59318 may also be related somehow but I'm not sure.

There is a another case that may cause mds metadata lost.

The root cause is that peon's pending_metadata (in memory) may be inconsistent with mon's db (in disk). When a peon turns into leader, and at the same time a standby mds stops, the new leader may flush wrong mds metadata into db.

It can be reproduced like this:

A Cluster with 3 mon and 3 mds (one active, other two standby), 6 osd.
    step 1. stop two standby mds;
    step 2. restart all mon; (make pending_medata consistent with db)
    step 3. start other two mds
    step 4. stop leader mon
    step 5. run "ceph mds metadata" command to check mds metadata
    step 6. stop one standby mds
    step 7. run "ceph mds metadata" command to check mds metadata again