Bug #23406: Attempt to set dashboard login credentials causes ceph-mgr to crash in Python 3-only environment - Dashboard - Ceph

Actions

Copy link

Bug #23406

closed

Attempt to set dashboard login credentials causes ceph-mgr to crash in Python 3-only environment

Added by Nathan Cutler about 6 years ago. Updated about 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Ricardo Dias

Category:

General

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Environment: SLE-15 (Python 3-only)

Currently running mimic_dev2 but the issue is presumed to be present on master as well

First, stop the ceph-mgr@.... systemd unit.

Second, apply the following fix for #23404

    @staticmethod
    def password_hash(password, salt_password=None):
        if not salt_password:
            salt_password = bcrypt.gensalt()
        if sys.version_info > (3, 0):
-           return bcrypt.hashpw(password, salt_password)
+           return bcrypt.hashpw(password.encode('utf8'), salt_password)
        return bcrypt.hashpw(password.encode('utf8'), salt_password)

Third, start ceph-mgr from the command line

ceph-mgr -f --cluster ceph --id target149202189062 --setuser ceph --setgroup ceph &

Fourth, enable the dashboard module:

ceph mgr module enable dashboard

Fifth, attempt to set login credentials:

ceph dashboard set-login-credentials admin admin

At this point, the ceph-mgr process crashes like so:

   -29> 2018-03-19 15:02:39.711 7f36e2057700  1 -- 149.202.189.62:6800/39198 <== client.14442 149.202.189.62:0/481764425 1 ==== command(tid 0: {"prefix": "dashboard set-login-credentials", "username": "admin", "password": "admin", "target": ["mgr", ""]}) v1 ==== 134+0+0 (2557707952 0 0) 0x559b990c3c00 con 0x559b9b170700
   -28> 2018-03-19 15:02:39.715 7f36e2057700  4 mgr.server handle_command decoded 4
   -27> 2018-03-19 15:02:39.715 7f36e2057700  4 mgr.server handle_command prefix=dashboard set-login-credentials
   -26> 2018-03-19 15:02:39.715 7f36e2057700  0 log_channel(audit) log [DBG] : from='client.14442 149.202.189.62:0/481764425' entity='client.admin' cmd=[{"prefix": "dashboard set-login-credentials", "username": "admin", "password": "admin", "target": ["mgr", ""]}]: dispatch
   -25> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'balancer status'
   -24> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'balancer mode'
   -23> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'balancer on'
   -22> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'balancer off'
   -21> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'balancer eval'
   -20> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'balancer eval-verbose'
   -19> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'balancer optimize'
   -18> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'balancer show'
   -17> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'balancer rm'
   -16> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'balancer reset'
   -15> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'balancer dump'
   -14> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'balancer execute'
   -13> 2018-03-19 15:02:39.715 7f36e2057700  1 mgr.server handle_command pyc_prefix: 'dashboard set-login-credentials'
   -12> 2018-03-19 15:02:39.715 7f36e2057700  4 mgr.server handle_command passing through 4
   -11> 2018-03-19 15:02:39.715 7f36e2858700 10 monclient: _send_command 6 [{"prefix":"config-key set","key":"mgr/dashboard_v2/username","val":"admin"}]
   -10> 2018-03-19 15:02:39.715 7f36e2858700 10 monclient: _send_mon_message to mon.target149202189062 at 149.202.189.62:6789/0
    -9> 2018-03-19 15:02:39.715 7f36e2858700  1 -- 149.202.189.62:0/39198 --> 149.202.189.62:6789/0 -- mon_command({"prefix":"config-key set","key":"mgr/dashboard_v2/username","val":"admin"} v 0) v1 -- 0x559b9af8eb40 con 0
    -8> 2018-03-19 15:02:39.719 7f36f5a61700  5 -- 149.202.189.62:0/39198 >> 149.202.189.62:6789/0 conn(0x559b9af96700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=362 cs=1 l=1). rx mon.0 seq 21 0x559b9af8eb40 mon_command_ack([{"prefix":"config-key set","key":"mgr/dashboard_v2/username","val":"admin"}]=0 set mgr/dashboard_v2/username v0) v1
    -7> 2018-03-19 15:02:39.719 7f36f17d1700  1 -- 149.202.189.62:0/39198 <== mon.0 149.202.189.62:6789/0 21 ==== mon_command_ack([{"prefix":"config-key set","key":"mgr/dashboard_v2/username","val":"admin"}]=0 set mgr/dashboard_v2/username v0) v1 ==== 138+0+0 (1064844682 0 0) 0x559b9af8eb40 con 0x559b9af96700
    -6> 2018-03-19 15:02:39.719 7f36f17d1700 10 monclient: handle_mon_command_ack 6 [{"prefix":"config-key set","key":"mgr/dashboard_v2/username","val":"admin"}]
    -5> 2018-03-19 15:02:39.719 7f36f17d1700 10 monclient: _finish_command 6 = 0 set mgr/dashboard_v2/username
    -4> 2018-03-19 15:02:40.079 7f36f5a61700  5 -- 149.202.189.62:0/39198 >> 149.202.189.62:6789/0 conn(0x559b9af96700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=362 cs=1 l=1). rx mon.0 seq 22 0x559b9b17c000 log(2 entries from seq 967 at 2018-03-19 15:02:39.718734) v1
    -3> 2018-03-19 15:02:40.079 7f36f17d1700  1 -- 149.202.189.62:0/39198 <== mon.0 149.202.189.62:6789/0 22 ==== log(2 entries from seq 967 at 2018-03-19 15:02:39.718734) v1 ==== 580+0+0 (1039645374 0 0) 0x559b9b17c000 con 0x559b9af96700
    -2> 2018-03-19 15:02:40.079 7f36f17d1700  4 mgr ms_dispatch active log(2 entries from seq 967 at 2018-03-19 15:02:39.718734) v1
    -1> 2018-03-19 15:02:40.079 7f36f17d1700  4 mgr ms_dispatch log(2 entries from seq 967 at 2018-03-19 15:02:39.718734) v1
     0> 2018-03-19 15:02:40.095 7f36e2858700 -1 *** Caught signal (Aborted) **
 in thread 7f36e2858700 thread_name:mgr-fin

 ceph version 13.0.1-3023-g71b0480e48 (71b0480e485b79d18f4476f6245c95bf48271c21) mimic (dev)
 1: (()+0x24f620) [0x559b9801c620]
 2: (()+0x122e0) [0x7f36f963a2e0]
 3: (gsignal()+0x110) [0x7f36f858d0e0]
 4: (abort()+0x151) [0x7f36f858e6c1]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x125) [0x7f36f8f0c205]
 6: (()+0x96ff6) [0x7f36f8f09ff6]
 7: (()+0x97041) [0x7f36f8f0a041]
 8: (()+0x97283) [0x7f36f8f0a283]
 9: (()+0x2587f2) [0x559b980257f2]
 10: (handle_pyerror[abi:cxx11]()+0x575) [0x559b97f0dc55]
 11: (ActivePyModule::handle_command(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*)+0x398) [0x559b97f249b8]
 12: (()+0xfd66c) [0x559b97eca66c]
 13: (FunctionContext::finish(int)+0x2c) [0x559b97edf93c]
 14: (Context::complete(int)+0x9) [0x559b97edb9c9]
 15: (Finisher::finisher_thread_entry()+0x1a7) [0x7f36fb91e197]
 16: (()+0x7559) [0x7f36f962f559]
 17: (clone()+0x3f) [0x7f36f864f82f]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-mgr.target149202189062.log
--- end dump of recent events ---

Full ceph-mgr log attached.

Files

ceph-mgr.target149202189062.log (996 KB) ceph-mgr.target149202189062.log

Nathan Cutler, 03/19/2018 03:18 PM

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Nathan Cutler about 6 years ago

Related to Bug #23404: dashboard module does not work in Python 3-only environment added

Actions

Copy link

Updated by Ricardo Dias about 6 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Ricardo Dias about 6 years ago

The current `handle_pyerror` function implementation relies in the `traceback.format_exception_only` python function to format the exception object. The problem is that this python function might also raise an exception.

Actions

Copy link