Support #11923
closed
MDS init script starts multiple instances when MDS is referenced in ceph.conf
Added by Brian Andrus almost 9 years ago.
Updated almost 9 years ago.
Description
MDS component init script does not seem to be able to properly differentiate between auto-detected instances and instances stated explicitly in the ceph.conf.
If an MDS is defined in ceph.conf, it is seemingly started twice.
root 4042 0.0 0.0 115212 1472 ? Ss 13:46 0:00 /usr/bin/bash -c ulimit -n 32768; /usr/bin/ceph-mds -i <NAME> --pid-file /var/run/ceph/mds.<NAME>.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root 4047 0.0 0.0 159016 6928 ? Sl 13:46 0:00 /usr/bin/ceph-mds -i <NAME> --pid-file /var/run/ceph/mds.<NAME>.pid -c /etc/ceph/ceph.conf --cluster ceph -f
If the statement is removed from ceph.conf, the daemon starts as expected (with only one process).
As with OSD and MON daemons, the init script should not start more than one process, even if the MDS is explicitly stated in ceph.conf (for convenient stop/start and other administrative purposes).
Files
Can you attach the ceph.conf and an ls -al of the instance directory? This will happen if the section is in ceph.conf with host = foo and there is a sysvinit or upstart file in the instance dir (by design).
Hello Sage,
The listing of /var/lib/ceph/mds is as following:
~~~
- ls -l /var/lib/ceph/mds/
total 0
drwxr-xr-x 2 root root 46 Jun 23 19:04 <mds_name>
- ls
l /var/lib/ceph/mds/<mds_name>/
total 4
-rw-r--r- 1 root root 0 Jun 23 19:04 done
rw-r--r- 1 root root 59 Jun 23 19:04 keyring
rw-r--r- 1 root root 0 Jun 23 19:04 sysvinit
~~~
The ceph.conf is attached with the tracker.
Thank you,
Vimal
How was this ceph.conf and MDS set up? If you remove the "sysvinit" file from the MDS directory it shouldn't be started up automatically on boot via that mechanism, but I think something must have been manually adjusted to get into this situation.
Greg,
I'd like to put in a slight correction in the 'ps' output. even it may be the same as what Brian said in the initial description. The actual 'ps' output is a bit different than what was added initially, ie.. it has four processes (Two processes having a child each).
# ps -ef | grep mds
root 4577 1 0 08:49 ? 00:00:00 /usr/bin/bash -c ulimit -n 32768; /usr/bin/ceph-mds -i <NAME> --pid-file /var/run/ceph/mds.<NAME>.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root 4583 4577 0 08:49 ? 00:00:00 /usr/bin/ceph-mds -i <NAME> --pid-file /var/run/ceph/mds.<NAME>.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root 4764 1 0 08:49 ? 00:00:00 /usr/bin/bash -c ulimit -n 32768; /usr/bin/ceph-mds -i <NAME> --pid-file /var/run/ceph/mds.<NAME>.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root 4769 4764 0 08:49 ? 00:00:00 /usr/bin/ceph-mds -i <NAME> --pid-file /var/run/ceph/mds.<NAME>.pid -c /etc/ceph/ceph.conf --cluster ceph -f
If I understand correct, the sysvinit file is responsible to start the MDS process by default. And any additional mention of an MDS in ceph.conf will start another corresponding process? At the time of the above 'ps' output, the sysvinit file is present (which is default) as well as the MDS entry in ceph.conf. Since the new ps output is slightly different from the one in the initial description, can we confirm that the above 'ps' output is indeed expected in this case?
I'm not sure if an MDS proces starts with "/usr/bin/bash -c ulimit -n 32768"? Is this normal or was this what Greg pointed to be the abnormality?
What would be suggested method to prevent this situation? Is it removing the 'sysvinit' file or removing the MDS entry in ceph.conf?
Thank you,
Vimal
- Tracker changed from Bug to Support
- Status changed from New to Resolved
Yes, in a default configuration you don't list the daemons and sysvinit starts them based on what folders exist in /var/lib/ceph containing a "sysvinit" file.
I'm not sure exactly what the standard ps output would look like, but nothing there surprises me. The ulimit increase is to allow for large numbers of outgoing sockets to OSDs and large numbers of incoming client connections.
I would recommend removing the explicit MDS enumeration from ceph.conf, but I don't know why you did that in the first place. If you need the explicit enumeration you could also remove the sysvinit file from the /var/lib/ceph/mds/<name> folder. Either should work equally well, but keeping the sysvinit file and removing the explicit enumeration is more standard.
I've also created a ticket to try and prevent the double-start in order to be friendlier.
Also available in: Atom
PDF