Project

General

Profile

Support #11923

MDS init script starts multiple instances when MDS is referenced in ceph.conf

Added by Brian Andrus almost 9 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Tags:
Reviewed:
Affected Versions:
Component(FS):
Labels (FS):
Pull request ID:

Description

MDS component init script does not seem to be able to properly differentiate between auto-detected instances and instances stated explicitly in the ceph.conf.

If an MDS is defined in ceph.conf, it is seemingly started twice.

root      4042  0.0  0.0 115212  1472 ?        Ss   13:46   0:00 /usr/bin/bash -c ulimit -n 32768; /usr/bin/ceph-mds -i <NAME> --pid-file /var/run/ceph/mds.<NAME>.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root      4047  0.0  0.0 159016  6928 ?        Sl   13:46   0:00 /usr/bin/ceph-mds -i <NAME> --pid-file /var/run/ceph/mds.<NAME>.pid -c /etc/ceph/ceph.conf --cluster ceph -f

If the statement is removed from ceph.conf, the daemon starts as expected (with only one process).

As with OSD and MON daemons, the init script should not start more than one process, even if the MDS is explicitly stated in ceph.conf (for convenient stop/start and other administrative purposes).

ceph.conf View - ceph.conf (1.8 KB) Vimal A.R, 07/02/2015 11:16 AM

History

#1 Updated by Sage Weil almost 9 years ago

Can you attach the ceph.conf and an ls -al of the instance directory? This will happen if the section is in ceph.conf with host = foo and there is a sysvinit or upstart file in the instance dir (by design).

#2 Updated by Vimal A.R over 8 years ago

Hello Sage,

The listing of /var/lib/ceph/mds is as following:

~~~
  1. ls -l /var/lib/ceph/mds/
    total 0
    drwxr-xr-x 2 root root 46 Jun 23 19:04 <mds_name>
  1. ls l /var/lib/ceph/mds/<mds_name>/
    total 4
    -rw-r--r-
    1 root root 0 Jun 23 19:04 done
    rw-r--r- 1 root root 59 Jun 23 19:04 keyring
    rw-r--r- 1 root root 0 Jun 23 19:04 sysvinit
    ~~~

The ceph.conf is attached with the tracker.

Thank you,

Vimal

#3 Updated by Greg Farnum over 8 years ago

How was this ceph.conf and MDS set up? If you remove the "sysvinit" file from the MDS directory it shouldn't be started up automatically on boot via that mechanism, but I think something must have been manually adjusted to get into this situation.

#4 Updated by Vimal A.R over 8 years ago

Greg,

I'd like to put in a slight correction in the 'ps' output. even it may be the same as what Brian said in the initial description. The actual 'ps' output is a bit different than what was added initially, ie.. it has four processes (Two processes having a child each).

# ps -ef | grep mds
root      4577     1  0 08:49 ?        00:00:00 /usr/bin/bash -c ulimit -n 32768; /usr/bin/ceph-mds -i <NAME> --pid-file /var/run/ceph/mds.<NAME>.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root      4583  4577  0 08:49 ?        00:00:00 /usr/bin/ceph-mds -i <NAME> --pid-file /var/run/ceph/mds.<NAME>.pid -c /etc/ceph/ceph.conf --cluster ceph -f

root      4764     1  0 08:49 ?        00:00:00 /usr/bin/bash -c ulimit -n 32768; /usr/bin/ceph-mds -i <NAME> --pid-file /var/run/ceph/mds.<NAME>.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root      4769  4764  0 08:49 ?        00:00:00 /usr/bin/ceph-mds -i <NAME> --pid-file /var/run/ceph/mds.<NAME>.pid -c /etc/ceph/ceph.conf --cluster ceph -f

If I understand correct, the sysvinit file is responsible to start the MDS process by default. And any additional mention of an MDS in ceph.conf will start another corresponding process? At the time of the above 'ps' output, the sysvinit file is present (which is default) as well as the MDS entry in ceph.conf. Since the new ps output is slightly different from the one in the initial description, can we confirm that the above 'ps' output is indeed expected in this case?

I'm not sure if an MDS proces starts with "/usr/bin/bash -c ulimit -n 32768"? Is this normal or was this what Greg pointed to be the abnormality?

What would be suggested method to prevent this situation? Is it removing the 'sysvinit' file or removing the MDS entry in ceph.conf?

Thank you,

Vimal

#5 Updated by Greg Farnum over 8 years ago

  • Tracker changed from Bug to Support
  • Status changed from New to Resolved

Yes, in a default configuration you don't list the daemons and sysvinit starts them based on what folders exist in /var/lib/ceph containing a "sysvinit" file.

I'm not sure exactly what the standard ps output would look like, but nothing there surprises me. The ulimit increase is to allow for large numbers of outgoing sockets to OSDs and large numbers of incoming client connections.

I would recommend removing the explicit MDS enumeration from ceph.conf, but I don't know why you did that in the first place. If you need the explicit enumeration you could also remove the sysvinit file from the /var/lib/ceph/mds/<name> folder. Either should work equally well, but keeping the sysvinit file and removing the explicit enumeration is more standard.

I've also created a ticket to try and prevent the double-start in order to be friendlier.

Also available in: Atom PDF