Bug #14263
closedmon:Re join the Monitor alone to form a cluster when it's rank=0
0%
Description
Monitor environment:
ceph240\ceph243\ceph244
[root@ceph240 ~]# ceph mon_status -f json-pretty
{
"name": "ceph244",
"rank": 2,
"state": "peon",
"election_epoch": 7402,
"quorum": [
0,
1,
2
],
"outside_quorum": [],
"extra_probe_peers": [],
"sync_provider": [],
"monmap": {
"epoch": 106,
"fsid": "bd9e23c5-da96-4908-a335-455445480e54",
"modified": "2016-01-05 19:22:31.516818",
"created": "0.000000",
"mons": [
{
"rank": 0,
"name": "ceph240",
"addr": "100.100.100.240:6789\/0"
},
{
"rank": 1,
"name": "ceph243",
"addr": "100.100.100.243:6789\/0"
},
{
"rank": 2,
"name": "ceph244",
"addr": "100.100.100.244:6789\/0"
}
]
}
}
Setp 1:
Delete ceph240, its rank = 0
ceph mon remove ceph240
[root@ceph240 ~]# ceph mon_status -f json-pretty
{
"name": "ceph244",
"rank": 1,
"state": "peon",
"election_epoch": 7404,
"quorum": [
0,
1
],
"outside_quorum": [],
"extra_probe_peers": [],
"sync_provider": [],
"monmap": {
"epoch": 107,
"fsid": "bd9e23c5-da96-4908-a335-455445480e54",
"modified": "2016-01-06 16:14:59.740930",
"created": "0.000000",
"mons": [
{
"rank": 0,
"name": "ceph243",
"addr": "100.100.100.243:6789\/0"
},
{
"rank": 1,
"name": "ceph244",
"addr": "100.100.100.244:6789\/0"
}
]
}
}
Setp 2:
Gets the key and map information of the cluster
Command
(1)ceph auth get mon. -o /tmp/key-filename
(2)ceph mon getmap -o /tmp/map-filename
(3)rm -rf /var/lib/ceph/mon/ceph-ceph240/*
(4)ceph-mon -i ceph240 --mkfs --monmap /tmp/map-filename --keyring /tmp/key-filename
(5)ceph mon add ceph240 100.100.100.240:6789
(6)ceph.conf'info:
*note:Mon_initial_members must be equal to ceph240, otherwise there will not be a description of the problem* [root@ceph240 ~]# more /etc/ceph/ceph.conf [global] fsid = bd9e23c5-da96-4908-a335-455445480e54 public_network = 100.100.100.0/24 cluster_network = 111.111.111.0/24 max_core_file_size = unlimited *mon_initial_members = ceph240* mon_host = 100.100.100.240:6789,100.100.100.243:6789,100.100.100.244:6789, osd_journal_size = 10240 osd_pool_default_size = 3 debug_osd=20/20 debug_mon = 20/20 debug_paxos = 20/20 [mon.ceph240] host = ceph240 [client] debug_client = 0/0 debug_objecter = 0/0 debug_rbd = 0/0 debug_rados = 0/0 log file=/var/log/ceph/rbd.log
Setp 3:
See if cluster ceph240 joined successfully
*join successfully*
[root@ceph240 ~]# ceph mon_status -f json-pretty
{
"name": "ceph244",
"rank": 2,
"state": "peon",
"election_epoch": 7406,
"quorum": [
1,
2
],
"outside_quorum": [],
"extra_probe_peers": [],
"sync_provider": [],
"monmap": {
"epoch": 108,
"fsid": "bd9e23c5-da96-4908-a335-455445480e54",
"modified": "2016-01-06 16:33:21.905559",
"created": "0.000000",
"mons": [
{
"rank": 0,
"name": "ceph240",
"addr": "100.100.100.240:6789\/0"
},
{
"rank": 1,
"name": "ceph243",
"addr": "100.100.100.243:6789\/0"
},
{
"rank": 2,
"name": "ceph244",
"addr": "100.100.100.244:6789\/0"
}
]
}
}
Setp 4:
start ceph240
Command:
service ceph start mon.ceph240
Use cluster command queries appear the following error: [root@ceph240 ~]# ceph mon_status -f json-pretty 2016-01-06 16:35:18.756279 7fd957313700 0 librados: client.admin authentication error (1) Operation not permitted Error connecting to cluster: PermissionError
ceph240 log: 2016-01-06 16:34:32.192786 7fb861dbc880 1 mon.ceph240@0(probing) e0 win_standalone_election 2016-01-06 16:34:32.192801 7fb861dbc880 1 mon.ceph240@0(probing).elector(1) init, last seen epoch 1 2016-01-06 16:34:32.192806 7fb861dbc880 10 mon.ceph240@0(probing).elector(1) bump_epoch 1 to 2 2016-01-06 16:34:32.193464 7fb861dbc880 10 mon.ceph240@0(probing) e0 join_election
Cause analysis:
ceph240 in the restart will add a separate cluster,when mon_initial_members equal to ceph240 and ceph240's rank equal to 0
Monitor::bootstrap()
{
......
// singleton monitor?
if (monmap->size() == 1 && rank == 0) {// only mon
win_standalone_election();
return;
}
......
Updated by Kefu Chai over 8 years ago
- Status changed from New to Fix Under Review
Updated by Greg Farnum over 8 years ago
- Status changed from Fix Under Review to New
I don't think this has been diagnosed correctly. ceph240 has epoch 1 and bumps it to 2, which makes me think the monmap didn't get injected properly. Maybe there's some other interesting bug but I don't think the stuff here demonstrates it's the presence of ceph240 in mon_initial_members. :)
Updated by huanwen ren over 8 years ago
another solution
https://github.com/ceph/ceph/pull/7230
Updated by Joao Eduardo Luis over 8 years ago
- Category set to Monitor
- Status changed from New to Rejected
The only way for this actually happen is for the user to explicitly create the necessary circumstances for this to happen. And those required a bit of effort to nail just right as the original bug report's steps didn't work for me.
If all the steps in the original bug report are followed, the monitor refuses to start because it is not in the monmap. Please note that in the original steps you are first obtaining the cluster monmap ('ceph get monmap -o /tmp/monmap'), adding the monitor to the cluster ('ceph mon add ...') and using the monmap in /tmp/monmap to mkfs the monitor. This results in the monitor only knowing about the other two monitors, and not being itself present in the monmap it just refuses to start.
The proper steps to reproduce this condition is to actually not obtaining the monmap and instead relying solely on 'mon host' and 'mon initial members'. However, one first needs to remove the other monitors from 'mon initial members'. If 'mon initial members' contains all the monitors in the cluster, it works flawlessly. If you intentionally remove the other two monitors, the monitor will generate an initial monmap that only contains itself -- thus assuming it's the only monitor in the cluster.
I'm closing this as I don't see evidence that there's actually a bug instead of just user error.
Updated by huanwen ren over 8 years ago
@Joao Luis
Thank you very much
We have automated deployment Monitor is a join the cluster, it will be in ceph.conf mon initial members equal to the scene of a mon hostname is rank=0 node. delete the mon to join again, in does not change the ceph.conf, In addition to the smallest node of the rank node can be normal to join
My aim is to solve the problem,under the premise of without changing the ceph. Conf
whether there is a better way?
Updated by Joao Eduardo Luis over 8 years ago
I'm not sure what you mean, but assuming your aim is to add a new monitor without changing ceph.conf, especially if ceph.conf does not contain a properly configured 'mon initial members', then your best bet is to:
ceph add <id> <addr>
ceph get monmap -o /tmp/foo
ceph-mon -i <id> --mkfs --monmap /tmp/foo [etc...]
The monitor really knows where the other monitors are. If it can't know from ceph.conf, because your ceph.conf is not correctly populated, then you need to have the monitor in the monmap PRIOR to creating the new monitor with mkfs AND use the monmap where the monitor exists.
This should help you.
Updated by huanwen ren over 8 years ago
@Joao Luis
thank for you attention
this way for cluster has existed more than two mon is(And mon is normal), but if there are only a mon, so after the CEPH add <id> <addr>, is not by CEPH get monmap the-o /tmp/foo get map information, because the cluster of mon is not available
Updated by Joao Eduardo Luis over 8 years ago
if you only have one monitor, the simplest way is to rely on 'mon host' and 'mon initial members' only. No need to run 'ceph mon getmap' prior to adding it.
For instance, let's say you have just the one monitor. Now you want to add a second monitor.
The existing monitor's mon_status looks something like:
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{
"name": "lxc-master-ceph-a",
"rank": 0,
"state": "leader",
"election_epoch": 28,
"quorum": [
0
],
"outside_quorum": [],
"extra_probe_peers": [],
"sync_provider": [],
"monmap": {
"epoch": 8,
"fsid": "4a679bed-5145-4563-99d6-f0667502458a",
"modified": "2016-01-18 13:38:41.630246",
"created": "0.000000",
"mons": [
{
"rank": 0,
"name": "lxc-master-ceph-a",
"addr": "10.0.3.116:6789\/0"
}
]
}
}
And you want to add a new monitor to this cluster.
Just make sure you have something like this on your soon-to-be new monitor:
mon host = 10.0.3.116:6789,10.0.3.34:6789 mon initial members = lxc-master-ceph-a,lxc-master-ceph-b
and then
ceph-mon -i lxc-master-ceph-b --mkfs --keyring /path/to/keyring && ceph-mon -i lxc-master-ceph-b
The monitor will find its way into the quorum. This will only happen the first time it boots too.
You don't need to get the existing monmap or do any other foo to get it working.
If you don't want to add that to the configuration file, you alternatively specify it via the cli:
ceph-mon -i lxc-master-ceph-b --mkfs --keyring /path/to/keyring --mon-host '10.0.3.116:6789,10.0.3.34:6789' --mon-initial-members 'lxc-master-ceph-a,lxc-master-ceph-b'
Hope this helps.