Project

General

Profile

Bug #45834

cephadm: "fs volume create cephfs" overwrites existing placement specification

Added by Jan Fajerski 9 months ago. Updated about 15 hours ago.

Status:
Triaged
Priority:
Urgent
Category:
Administration/Usability
Target version:
% Done:

0%

Source:
Development
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
mgr/mds_autoscaler, mgr/volumes
Labels (FS):
Pull request ID:
Crash signature:

Description

The orchestrator behaves unexpectedly with apply mds. Consider the following:

I have a ceph cluster running and want cephfs

ceph orch apply mds cephfs 3

gives me 3 MDS daemons but no cephfs. The "cephfs" argument doesn't seem to do anything.

Then I created an fs vie

ceph fs volume create cephfs

. This reduces the number of MDS daemons to 2.

A subsequent call to

ceph orch apply mds cephfs 3

recreates the initially desired state.

History

#1 Updated by Sebastian Wagner 9 months ago

  • Description updated (diff)

#2 Updated by Sebastian Wagner 9 months ago

  • Tags set to ux

#3 Updated by Sebastian Wagner 9 months ago

Easy solution would be to validate the existence of this CephFS, before starting the MDS daemons.

#4 Updated by Sebastian Wagner 7 months ago

  • Tracker changed from Bug to Documentation

#5 Updated by Sebastian Wagner 11 days ago

  • Tracker changed from Documentation to Bug
  • Project changed from Orchestrator to CephFS
  • Subject changed from cephadm: ceph orch apply mds <fsname> <n> to cephadm: "fs volume create cephfs" overwrites existing placement specification
  • Description updated (diff)
  • Regression set to No
  • Severity set to 3 - minor
  • Component(FS) mgr/volumes added

The problem is caused by volumes.fs.fs_util.create_mds replacing the existing placement spec with None and thus is overwriting it with the default.

The orchestrator API is declarative. We should probably avoid applying an MDS service here, if it already exists.

#6 Updated by Patrick Donnelly 8 days ago

  • Category set to Administration/Usability
  • Priority changed from Normal to Urgent
  • Target version set to v17.0.0
  • Source set to Development
  • Tags deleted (ux)
  • Backport set to pacific
  • Component(FS) mgr/mds_autoscaler added

#7 Updated by Patrick Donnelly 1 day ago

  • Status changed from New to Triaged
  • Assignee set to Milind Changire

#8 Updated by Milind Changire about 15 hours ago

Jan,
Could you attach the fs dump just after creating 'cephfs'
eg.
$ ceph fs dump --format=json-pretty

I suspect that after creating the 'cephfs' file-system, max_mds is set to 1 and standby_count_wanted is also set to 1. So, in total there would be 2 mds instances running. Which is what you get to see in your case.

I'm not sure how the orch commands affect the fsmap when "ceph orch apply mds cephfs 3" is executed after the 'cephfs' fs is created. But the MDS Autoscaler only responds to fsmap changes to max_mds and standby_count_wanted tunables.

The very first "ceph orch apply mds cephfs 3" could be a no-op, as there wasn't any fs named 'cephfs' existing when that command was executed. Sebastian could clarify it further.

The mds count as 2 could be the default behavior on new fs creation. Need to confirm it with your "fs dump" output.

Also available in: Atom PDF