Documentation #54551
closeddocs.ceph.com/en/pacific/cephfs/add-remove-mds/#adding-an-mds cannot work
0%
Description
The docu ADDING AN MDS starts by Create an mds data point /var/lib/ceph/mds/ceph-${id} (according to Google, this is the only place in the net where the term data point appears).
This path seems to be wrong:
In a Pacific installation (rhel 8.5), there are many directories in /var/lib/ceph, but only /var/lib/ceph/<fsid> is populated.
On the running MDS (created by ceph fs volume create <CephFS-Name>) , /var/lib/ceph/<fsid>/mds.<CephFS-Name>.<Mon-Name>.ltfjam contains config, keyring and unit.* files.
On a potential additional MDS, such a directory could be created manually, of course. The next step in the docu, ceph auth get-or-create mds.${id}... would put the keyring into that directory,
but of course none of the other files.
Checking what actually gets started on the running MDS by (in the said installation) systemctl status ceph-<fsid>@mds.<CephFS-Name>.<Hostname>.whdfrs,
that is /bin/bash /var/lib/ceph/<fsid>/%i/unit.run, it is quite a bit podman rm and podman run.
All of the missing files could of course be scp'd by hand, but that is not how it should work?
In addition, docs.ceph.com/en/latest/cephfs/add-remove-mds/# after a lengthy description of provisioning hardware for MDSes proceeds straight to the faulty ADDING AN MDS, but does not explain how to
specify the hardware aka the hostname/IP . It seems this information cannot be found anywhere on docs.ceph.com.
(Counter example to the last point: If I wanted to add another monitor to the cluster, I would do @ceph orch host add <Hostname> <IP>)
Updated by Venky Shankar about 2 years ago
- Category set to Correctness/Safety
- Assignee set to Dhairya Parmar
- Target version set to v18.0.0
- Backport set to quincy, pacific
- Labels (FS) task(easy) added
Updated by Venky Shankar about 2 years ago
- Status changed from New to Fix Under Review
- Pull request ID set to 45639
Thomas,
Does the documentation changes makes things clear - https://ceph--45639.org.readthedocs.build/en/45639/cephfs/add-remove-mds/
Cheers,
Venky
Updated by Thomas Roth about 2 years ago
Hi Venky,
yes and no - the change makes it clear that a directory has to be created.
But is there a canonical path, a path that should be the same in all (Pacific) installations?
Because as I wrote, /var/lib/ceph/mds/ceph-${id} seems not to work. Instead, ceph (ceph-adm) has everything in /var/lib/ceph/ceph-${id}/.
In the particular case of one of our MDS, it is var/lib/ceph/ceph-${id}/mds.${cephfs-name}.${mds-hostname}.sohlr.
This difference in directory levels will be important especially for automated setups.
Regards,
Thomas
Updated by Thomas Roth about 2 years ago
Concerning my question on how to specify the host for the MDS, Google helped:
- Add 2 hosts by ceph orch host add <hostname1> <IP1>, ceph orch host add <hostname2> <IP2>
- ceph orch apply mds catweazle --placement="2 <hostname1> <hostname2>"
The latter command is in the docu, albeit under the title CONVERTING AN EXISTING CLUSTER TO CEPHADM.
Since I started afresh using cephadm from the start, it did not occur to me to look there...
Updated by Dhairya Parmar about 2 years ago
Thomas Roth wrote:
Hi Venky,
yes and no - the change makes it clear that a directory has to be created.
But is there a canonical path, a path that should be the same in all (Pacific) installations?Because as I wrote, /var/lib/ceph/mds/ceph-${id} seems not to work. Instead, ceph (ceph-adm) has everything in /var/lib/ceph/ceph-${id}/.
In the particular case of one of our MDS, it is var/lib/ceph/ceph-${id}/mds.${cephfs-name}.${mds-hostname}.sohlr.This difference in directory levels will be important especially for automated setups.
Regards,
Thomas
Hi Thomas,
ceph-adm/ceph orchestrator will always use the path(and also the directory level) similar to what you shared i.e. /var/lib/ceph/ceph-${id}/mds.${cephfs-name}.${mds-hostname}.sohlr and it will be always followed. The doc https://docs.ceph.com/en/pacific/cephfs/add-remove-mds/#adding-an-mds aims at setting up MDS manually, that's why you see a different directory level in that case. ceph-adm/ceph orchestrator automates a lot of things and is much better and easy to use, so it is recommended to use those methods to set up the cluster and thus also the MDS.
Updated by Dhairya Parmar about 2 years ago
Thomas Roth wrote:
Concerning my question on how to specify the host for the MDS, Google helped:
- Add 2 hosts by
ceph orch host add <hostname1> <IP1>,ceph orch host add <hostname2> <IP2>
-ceph orch apply mds catweazle --placement="2 <hostname1> <hostname2>"The latter command is in the docu, albeit under the title CONVERTING AN EXISTING CLUSTER TO CEPHADM.
Since I started afresh using cephadm from the start, it did not occur to me to look there...
Ceph orchestrator is a great tool, You can also refer to this doc - https://docs.ceph.com/en/latest/mgr/orchestrator/#stateless-services-mds-rgw-nfs-rbd-mirror-iscsi, hope it helps.
Updated by Venky Shankar about 2 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot about 2 years ago
- Copied to Backport #55238: pacific: docs.ceph.com/en/pacific/cephfs/add-remove-mds/#adding-an-mds cannot work added
Updated by Backport Bot about 2 years ago
- Copied to Backport #55239: quincy: docs.ceph.com/en/pacific/cephfs/add-remove-mds/#adding-an-mds cannot work added
Updated by Dhairya Parmar over 1 year ago
- Status changed from Pending Backport to Resolved
Updated by Zac Dover 2 months ago
I've made the notes a little clearer here: https://github.com/ceph/ceph/pull/55637
I've tried to help the reader distinguish between cases in which they will want to use an automated tool like cephadm to deploy MDSes and cases in which they will want to manually deploy MDSes. As the text stood, I thought that it was a little ambiguous and could potentially confuse a first-time reader unfamiliar with CephFS.