Project

General

Profile

Actions
Copy link

Feature #21198

open

Monitors don't handle incomplete network splits

Added by zhiang li over 6 years ago. Updated over 6 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Administration/Usability
Target version:
Ceph - v10.2.10
% Done:

0%

Source:
other
Tags:
Backport:
Reviewed:
Affected Versions:
Ceph - v10.2.5
Component(RADOS):
Monitor
Pull request ID:

Description

the network between monitors(the minimum rank and the maximum rank) disconnect, the node of the maximum rank always keep electing and the mon can't work,fox example the command of "ceph -s" can't work.
the network topology as below:

1(175)
      / \
(176)2   0(174)

the log of the 176 node as below, the other logs in the attachment

2017-08-22 18:07:27.800813 7fcdae7b4700 5 mon.node176@2(electing).elector(11791) election timer expired
2017-08-22 18:07:27.800841 7fcdae7b4700 5 mon.node176@2(electing).elector(11791) start -- can i be leader?
2017-08-22 18:07:27.800922 7fcdae7b4700 1 mon.node176@2(electing).elector(11791) init, last seen epoch 11791
2017-08-22 18:07:27.805078 7fcdadfb3700 5 mon.node176@2(electing).elector(11791) handle_propose from mon.1
2017-08-22 18:07:27.805081 7fcdadfb3700 10 mon.node176@2(electing).elector(11791) handle_propose required features 9025616074506240, peer features 576460752032874495
2017-08-22 18:07:27.805083 7fcdadfb3700 10 mon.node176@2(electing).elector(11791) bump_epoch 11791 to 11793
2017-08-22 18:07:27.806013 7fcdadfb3700 10 mon.node176@2(electing) e5 join_election
2017-08-22 18:07:27.806028 7fcdadfb3700 10 mon.node176@2(electing) e5 _reset
2017-08-22 18:07:27.806034 7fcdadfb3700 10 mon.node176@2(electing) e5 cancel_probe_timeout (none scheduled)
2017-08-22 18:07:27.806038 7fcdadfb3700 10 mon.node176@2(electing) e5 timecheck_finish
2017-08-22 18:07:27.806041 7fcdadfb3700 10 mon.node176@2(electing) e5 scrub_event_cancel
2017-08-22 18:07:27.806044 7fcdadfb3700 10 mon.node176@2(electing) e5 scrub_reset
2017-08-22 18:07:27.806053 7fcdadfb3700 5 mon.node176@2(electing).elector(11793) defer to 1
2017-08-22 18:07:33.806178 7fcdae7b4700 5 mon.node176@2(electing).elector(11793) election timer expired
2017-08-22 18:07:33.806202 7fcdae7b4700 5 mon.node176@2(electing).elector(11793) start -- can i be leader?
2017-08-22 18:07:33.806247 7fcdae7b4700 1 mon.node176@2(electing).elector(11793) init, last seen epoch 11793
2017-08-22 18:07:33.810660 7fcdadfb3700 5 mon.node176@2(electing).elector(11793) handle_propose from mon.1
2017-08-22 18:07:33.810662 7fcdadfb3700 10 mon.node176@2(electing).elector(11793) handle_propose required features 9025616074506240, peer features 576460752032874495
2017-08-22 18:07:33.810665 7fcdadfb3700 10 mon.node176@2(electing).elector(11793) bump_epoch 11793 to 11795
2017-08-22 18:07:33.811652 7fcdadfb3700 10 mon.node176@2(electing) e5 join_election
2017-08-22 18:07:33.811666 7fcdadfb3700 10 mon.node176@2(electing) e5 _reset
2017-08-22 18:07:33.811668 7fcdadfb3700 10 mon.node176@2(electing) e5 cancel_probe_timeout (none scheduled)
2017-08-22 18:07:33.811670 7fcdadfb3700 10 mon.node176@2(electing) e5 timecheck_finish
2017-08-22 18:07:33.811673 7fcdadfb3700 10 mon.node176@2(electing) e5 scrub_event_cancel
2017-08-22 18:07:33.811674 7fcdadfb3700 10 mon.node176@2(electing) e5 scrub_reset
2017-08-22 18:07:33.811681 7fcdadfb3700 5 mon.node176@2(electing).elector(11795) defer to 1
2017-08-22 18:07:39.811807 7fcdae7b4700 5 mon.node176@2(electing).elector(11795) election timer expired
2017-08-22 18:07:39.811841 7fcdae7b4700 5 mon.node176@2(electing).elector(11795) start -- can i be leader?
2017-08-22 18:07:39.811925 7fcdae7b4700 1 mon.node176@2(electing).elector(11795) init, last seen epoch 11795
2017-08-22 18:07:39.816982 7fcdadfb3700 5 mon.node176@2(electing).elector(11795) handle_propose from mon.1

Files

Download all files
174_1.log-20170824.gz (7.54 KB) 174_1.log-20170824.gz 174 zhiang li, 08/31/2017 09:23 AM
175_2.log-20170824.gz (9.08 KB) 175_2.log-20170824.gz 175 zhiang li, 08/31/2017 09:23 AM
176_1.log-20170824.gz (819 Bytes) 176_1.log-20170824.gz 176 zhiang li, 08/31/2017 09:23 AM
Actions
Copy link
 #2

Updated by Greg Farnum over 6 years ago

  • Tracker changed from Bug to Feature
  • Subject changed from the network between monitors(the minimum rank and the maximum rank) disconnect, the node of the maximum rank always keep electing and the mon can't work to Monitors don't handle incomplete network splits
  • Category set to Administration/Usability

Yep. If you have a network partition that can be crossed by some monitors but not others, we're screwed.

Resolving it would be a big project to try and detect these situations and work around them by declaring some monitors dead.

Actions
Copy link

Also available in: Atom PDF