Project

General

Profile

Documentation #54551

docs.ceph.com/en/pacific/cephfs/add-remove-mds/#adding-an-mds cannot work

Added by Thomas Roth 9 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Category:
Correctness/Safety
Target version:
% Done:

0%

Tags:
Backport:
quincy, pacific
Reviewed:
Affected Versions:
Labels (FS):
task(easy)
Pull request ID:

Description

The docu ADDING AN MDS starts by Create an mds data point /var/lib/ceph/mds/ceph-${id} (according to Google, this is the only place in the net where the term data point appears).
This path seems to be wrong:

In a Pacific installation (rhel 8.5), there are many directories in /var/lib/ceph, but only /var/lib/ceph/<fsid> is populated.
On the running MDS (created by ceph fs volume create <CephFS-Name>) , /var/lib/ceph/<fsid>/mds.<CephFS-Name>.<Mon-Name>.ltfjam contains config, keyring and unit.* files.

On a potential additional MDS, such a directory could be created manually, of course. The next step in the docu, ceph auth get-or-create mds.${id}... would put the keyring into that directory,
but of course none of the other files.

Checking what actually gets started on the running MDS by (in the said installation) systemctl status ceph-<fsid>@mds.<CephFS-Name>.<Hostname>.whdfrs,
that is /bin/bash /var/lib/ceph/<fsid>/%i/unit.run, it is quite a bit podman rm and podman run.
All of the missing files could of course be scp'd by hand, but that is not how it should work?

In addition, docs.ceph.com/en/latest/cephfs/add-remove-mds/# after a lengthy description of provisioning hardware for MDSes proceeds straight to the faulty ADDING AN MDS, but does not explain how to
specify the hardware aka the hostname/IP . It seems this information cannot be found anywhere on docs.ceph.com.

(Counter example to the last point: If I wanted to add another monitor to the cluster, I would do @ceph orch host add <Hostname> <IP>)


Related issues

Copied to CephFS - Backport #55238: pacific: docs.ceph.com/en/pacific/cephfs/add-remove-mds/#adding-an-mds cannot work Resolved
Copied to CephFS - Backport #55239: quincy: docs.ceph.com/en/pacific/cephfs/add-remove-mds/#adding-an-mds cannot work Resolved

History

#1 Updated by Venky Shankar 8 months ago

  • Category set to Correctness/Safety
  • Assignee set to Dhairya Parmar
  • Target version set to v18.0.0
  • Backport set to quincy, pacific
  • Labels (FS) task(easy) added

#2 Updated by Venky Shankar 8 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 45639

Thomas,

Does the documentation changes makes things clear - https://ceph--45639.org.readthedocs.build/en/45639/cephfs/add-remove-mds/

Cheers,
Venky

#3 Updated by Thomas Roth 8 months ago

Hi Venky,

yes and no - the change makes it clear that a directory has to be created.
But is there a canonical path, a path that should be the same in all (Pacific) installations?

Because as I wrote, /var/lib/ceph/mds/ceph-${id} seems not to work. Instead, ceph (ceph-adm) has everything in /var/lib/ceph/ceph-${id}/.
In the particular case of one of our MDS, it is var/lib/ceph/ceph-${id}/mds.${cephfs-name}.${mds-hostname}.sohlr.

This difference in directory levels will be important especially for automated setups.

Regards,
Thomas

#4 Updated by Thomas Roth 8 months ago

Concerning my question on how to specify the host for the MDS, Google helped:

- Add 2 hosts by ceph orch host add <hostname1> <IP1>, ceph orch host add <hostname2> <IP2>
- ceph orch apply mds catweazle --placement="2 <hostname1> <hostname2>"

The latter command is in the docu, albeit under the title CONVERTING AN EXISTING CLUSTER TO CEPHADM.

Since I started afresh using cephadm from the start, it did not occur to me to look there...

#5 Updated by Dhairya Parmar 8 months ago

Thomas Roth wrote:

Hi Venky,

yes and no - the change makes it clear that a directory has to be created.
But is there a canonical path, a path that should be the same in all (Pacific) installations?

Because as I wrote, /var/lib/ceph/mds/ceph-${id} seems not to work. Instead, ceph (ceph-adm) has everything in /var/lib/ceph/ceph-${id}/.
In the particular case of one of our MDS, it is var/lib/ceph/ceph-${id}/mds.${cephfs-name}.${mds-hostname}.sohlr.

This difference in directory levels will be important especially for automated setups.

Regards,
Thomas

Hi Thomas,

ceph-adm/ceph orchestrator will always use the path(and also the directory level) similar to what you shared i.e. /var/lib/ceph/ceph-${id}/mds.${cephfs-name}.${mds-hostname}.sohlr and it will be always followed. The doc https://docs.ceph.com/en/pacific/cephfs/add-remove-mds/#adding-an-mds aims at setting up MDS manually, that's why you see a different directory level in that case. ceph-adm/ceph orchestrator automates a lot of things and is much better and easy to use, so it is recommended to use those methods to set up the cluster and thus also the MDS.

#6 Updated by Dhairya Parmar 8 months ago

Thomas Roth wrote:

Concerning my question on how to specify the host for the MDS, Google helped:

- Add 2 hosts by ceph orch host add <hostname1> <IP1>, ceph orch host add <hostname2> <IP2>
- ceph orch apply mds catweazle --placement="2 <hostname1> <hostname2>"

The latter command is in the docu, albeit under the title CONVERTING AN EXISTING CLUSTER TO CEPHADM.

Since I started afresh using cephadm from the start, it did not occur to me to look there...

Ceph orchestrator is a great tool, You can also refer to this doc - https://docs.ceph.com/en/latest/mgr/orchestrator/#stateless-services-mds-rgw-nfs-rbd-mirror-iscsi, hope it helps.

#7 Updated by Venky Shankar 8 months ago

  • Status changed from Fix Under Review to Pending Backport

#8 Updated by Backport Bot 8 months ago

  • Copied to Backport #55238: pacific: docs.ceph.com/en/pacific/cephfs/add-remove-mds/#adding-an-mds cannot work added

#9 Updated by Backport Bot 8 months ago

  • Copied to Backport #55239: quincy: docs.ceph.com/en/pacific/cephfs/add-remove-mds/#adding-an-mds cannot work added

#10 Updated by Dhairya Parmar 4 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF