Bug #24118: mds: crash when using `config set` on tracked configs - CephFS - Ceph

Actions

Copy link

Bug #24118

closed

mds: crash when using `config set` on tracked configs

Added by Patrick Donnelly almost 6 years ago. Updated almost 6 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Sage Weil

Category:

Correctness/Safety

Target version:

Ceph - v13.2.0

% Done:

Source:

Community (user)

Tags:

Backport:

mimic

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

MDS

Labels (FS):

crash

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

These configs: https://github.com/ceph/ceph/blob/7dbba9e54282e0a4c3000eb0c1a66e346c7eab98/src/mds/MDSDaemon.cc#L362-L380

Trying to set one with the new-style `ceph config set` causes a dependency cycle assertion:

$ bin/ceph config set mds.c mds_max_purge_ops 15

2018-05-14 13:43:19.778 7fe13ced3700 10 monclient: handle_config config(13 keys) v1
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals no callback set
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored debug_mds
2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals debug_mds = 15/15 (no change)
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored debug_mgrc
2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals debug_mgrc = 20/20 (no change)
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored debug_monc
2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals debug_monc = 20/20 (no change)
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored debug_ms
2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals debug_ms = 1/1 (no change)
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored mds_debug_auth_pins
2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals mds_debug_auth_pins = true (no change)
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored mds_debug_frag
2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals mds_debug_frag = true (no change)
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored mds_debug_scatterstat
2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals mds_debug_scatterstat = true (no change)
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored mds_debug_subtrees
2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals mds_debug_subtrees = true (no change)
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored mds_max_purge_ops
2018-05-14 13:43:19.778 7fe13ced3700 10 set_mon_vals mds_max_purge_ops = 15
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored mds_verify_scatter
2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals mds_verify_scatter = true (no change)
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored mon_pg_warn_min_per_osd
2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals mon_pg_warn_min_per_osd = 3 (no change)
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored osd_pool_default_min_size
2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals osd_pool_default_min_size = 1 (no change)
2018-05-14 13:43:19.778 7fe13ced3700  4 set_mon_vals callback ignored osd_pool_default_size
2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals osd_pool_default_size = 3 (no change)
2018-05-14 13:43:19.782 7fe13ced3700  0
------------------------------------
existing dependency MDSDaemon::mds_lock (15) -> MonClient::monc_lock (13) at:

2018-05-14 13:43:19.782 7fe13ced3700  0 new dependency MonClient::monc_lock (13) -> MDSDaemon::mds_lock (15) creates a cycle at
 ceph version 13.1.0-307-gb871031 (b87103157e77d7ae3b3530b366a5d69455e3fe96) nautilus (dev)
 1: (MDSDaemon::handle_conf_change(md_config_t const*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+0x50) [0x8a2d2ddd20]
 2: (md_config_t::_apply_changes(std::ostream*)+0x503) [0x7fe1449d2e43]
 3: (md_config_t::set_mon_vals(CephContext*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)+0x543) [0x7fe1449d3503]
 4: (MonClient::handle_config(MConfig*)+0x155) [0x7fe14473fa55]
 5: (MonClient::ms_dispatch(Message*)+0x32b) [0x7fe144749f6b]
 6: (DispatchQueue::entry()+0xb5a) [0x7fe14476b06a]
 7: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fe14480af9d]
 8: (()+0x76ba) [0x7fe143fd06ba]
 9: (clone()+0x6d) [0x7fe14325a41d]

2018-05-14 13:43:19.782 7fe13ced3700  0 btw, i am holding these locks:
2018-05-14 13:43:19.782 7fe13ced3700  0   MonClient::monc_lock (13)
2018-05-14 13:43:19.782 7fe13ced3700  0

2018-05-14 13:43:19.786 7fe13ced3700 -1 *** Caught signal (Aborted) **
 in thread 7fe13ced3700 thread_name:ms_dispatch

 ceph version 13.1.0-307-gb871031 (b87103157e77d7ae3b3530b366a5d69455e3fe96) nautilus (dev)
 1: (()+0x3dbb10) [0x8a2d5bdb10]
 2: (()+0x11390) [0x7fe143fda390]
 3: (gsignal()+0x38) [0x7fe143188428]
 4: (abort()+0x16a) [0x7fe14318a02a]
 5: (lockdep_will_lock(char const*, int, bool)+0x9fe) [0x7fe1449df51e]
 6: (Mutex::Lock(bool)+0x1b9) [0x7fe1446c4299]
 7: (MDSDaemon::handle_conf_change(md_config_t const*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+0x50) [0x8a2d2ddd20]
 8: (md_config_t::_apply_changes(std::ostream*)+0x503) [0x7fe1449d2e43]
 9: (md_config_t::set_mon_vals(CephContext*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)+0x543) [0x7fe1449d3503]
 10: (MonClient::handle_config(MConfig*)+0x155) [0x7fe14473fa55]
 11: (MonClient::ms_dispatch(Message*)+0x32b) [0x7fe144749f6b]
 12: (DispatchQueue::entry()+0xb5a) [0x7fe14476b06a]
 13: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fe14480af9d]
 14: (()+0x76ba) [0x7fe143fd06ba]
 15: (clone()+0x6d) [0x7fe14325a41d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Related issues 1 (0 open — 1 closed)