Project

General

Profile

Bug #19629

mgr: set_config from python module crashes mgr (assertion failure due to access denied)

Added by Tim Serong 6 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
04/14/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

Tried `ceph tell mgr enable_auth false` (enable_auth being implemented by the rest module). The ceph CLI became unresponsive (never returned to the shell). Meanwhile, mgr crashed with:

    -8> 2017-04-14 15:34:45.938295 7f00c778a700  4 mgr[rest] handle_command: {
  "prefix": "enable_auth", 
  "val": "false" 
}
    -7> 2017-04-14 15:34:45.938336 7f00c778a700  1 lockdep using id 41
    -6> 2017-04-14 15:34:45.938369 7f00c778a700 10 monclient: _send_command 5 [{"prefix":"config-key put","key":"mgr.rest.enable_auth","val":"false"}]
    -5> 2017-04-14 15:34:45.938376 7f00c778a700 10 monclient: _send_mon_message to mon.b at 127.0.0.1:40447/0
    -4> 2017-04-14 15:34:45.938381 7f00c778a700  1 -- 127.0.0.1:0/3288703021 --> 127.0.0.1:40447/0 -- mon_command({"prefix":"config-key put","key":"mgr.rest.enable_auth","val":"false"} v 0) v1 -- 0x55ea04db41c0 con 0
    -3> 2017-04-14 15:34:45.938756 7f00caf91700  1 -- 127.0.0.1:0/3288703021 <== mon.1 127.0.0.1:40447/0 25 ==== mon_command_ack([{"prefix":"config-key put","key":"mgr.rest.enable_auth","val":"false"}]=-13 access denied v0) v1 ==== 117+0+0 (2523711155 0 0) 0x55ea04db41c0 con 0x55ea02d59800
    -2> 2017-04-14 15:34:45.938773 7f00caf91700 10 monclient: handle_mon_command_ack 5 [{"prefix":"config-key put","key":"mgr.rest.enable_auth","val":"false"}]
    -1> 2017-04-14 15:34:45.938778 7f00caf91700 10 monclient: _finish_command 5 = -13 access denied
     0> 2017-04-14 15:34:45.940929 7f00c778a700 -1 /home/tserong/src/github/SUSE/ceph/src/mgr/PyModules.cc: In function 'void PyModules::set_config(const string&, const string&, const string&)' thread 7f00c778a700 time 2017-04-14 15:34:45.938831
/home/tserong/src/github/SUSE/ceph/src/mgr/PyModules.cc: 546: FAILED assert(set_cmd.r == 0)

Why is the mon giving mgr "access denied" when it tries to set a config key?

We should consider getting rid of that assert too...

History

#1 Updated by Sage Weil 6 months ago

  • Priority changed from Normal to Urgent

#2 Updated by Tim Serong 6 months ago

The permission denied problem goes away if you give mgr full access to the mon. For example, in a vstart environment:

# bin/ceph auth caps mgr.x mon 'allow *' mds 'allow *' osd 'allow *'
updated caps for mgr.x
# bin/init-ceph restart mgr.x
# bin/ceph tell mgr enable_auth false
(...works fine, doesn't crash anymore...)

Originally it was "mon 'allow profile mgr'", now it's "mon 'allow *'". Is that too permissive?

Note that mgr must be restarted to pick up the auth caps change. Is that correct behaviour, or should it notice the changed caps by magic?

#3 Updated by Tim Serong 6 months ago

  • Status changed from New to In Progress
  • Assignee set to Tim Serong

#4 Updated by Tim Serong 6 months ago

  • Status changed from In Progress to Need Review

https://github.com/ceph/ceph/pull/14706

This PR doesn't help with mgr's auth caps, it just makes sure it logs the failure rather than crashing horribly.

#5 Updated by Sage Weil 6 months ago

  • Status changed from Need Review to Resolved

Also available in: Atom PDF