Project

General

Profile

Actions
Copy link

Bug #18203

closed

Additional Ceph Monitors failed verifying authorize reply

Added by Kyle Traff over 7 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Using ceph-deploy, I add two additional ceph monitors to my ceph cluster:

ceph-deploy mon add server2
ceph-deploy mon add server3

To which I get the following feedback:

[server2][DEBUG ] ************************************************************************
[server2][DEBUG ] status for monitor: mon.server2
[server2][DEBUG ] {
[server2][DEBUG ] "election_epoch": 0,
[server2][DEBUG ] "extra_probe_peers": [],
[server2][DEBUG ] "monmap": {
[server2][DEBUG ] "created": "2016-12-07 15:00:41.747419",
[server2][DEBUG ] "epoch": 0,
[server2][DEBUG ] "fsid": "e6324592-220d-4aba-a764-4ae811e9140a",
[server2][DEBUG ] "modified": "2016-12-07 15:00:41.747419",
[server2][DEBUG ] "mons": [
[server2][DEBUG ] {
[server2][DEBUG ] "addr": "129.82.20.125:6789/0",
[server2][DEBUG ] "name": "derams1",
[server2][DEBUG ] "rank": 0
[server2][DEBUG ] }
[server2][DEBUG ] ]
[server2][DEBUG ] },
[server2][DEBUG ] "name": "server2",
[server2][DEBUG ] "outside_quorum": [],
[server2][DEBUG ] "quorum": [],
[server2][DEBUG ] "rank": -1,
[server2][DEBUG ] "state": "probing",
[server2][DEBUG ] "sync_provider": []
[server2][DEBUG ] }
[server2][DEBUG ] ************************************************************************
[server2][INFO ] monitor: mon.server2 is currently at the state of probing

(See full output - https://gist.github.com/ktraff/f056731855eb35d9fc814d7292796b9d)

However, the additional monitors never connect. Instead, I see these messages in the log files:

From admin node:

2016-12-08 16:07:46.238289 7fd6aed0d700 0 mon.derams1@0(leader) e1 ms_verify_authorizer bad authorizer from mon server2:6789/0
2016-12-08 16:07:46.238299 7fd6aed0d700 0 -- 129.82.20.125:6789/0 >> server2:6789/0 pipe(0x55fd15b9a800 sd=8 :6789 s=0 pgs=0 cs=0 l=0 c=0x55fd15da7c00).accept: got bad authorizer

From server2:

2016-12-08 16:06:39.023640 7f44c3ea1700 0 cephx: verify_reply couldn't decrypt with error: error decoding block for decryption
2016-12-08 16:06:39.023652 7f44c3ea1700 0 -- server2:6789/0 >> admin:6789/0 pipe(0x555cc3f1e000 sd=13 :35008 s=1 pgs=0 cs=0 l=0 c=0x555cc3d10a80).failed verifying authorize reply
2016-12-08 16:06:39.934802 7f44bef29700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2016-12-08 16:06:39.935031 7f44bef29700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2016-12-08 16:06:41.105398 7f44bef29700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2016-12-08 16:06:41.105597 7f44bef29700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished

Actions
Copy link
 #1

Updated by Sage Weil over 7 years ago

  • Status changed from New to Closed

It sounds like the mon. key wasn't installed properly on the new mons. try copying it into your ceph-deploy dir or directly to the dest monitor.

Actions
Copy link

Also available in: Atom PDF