Actions
Bug #24118
closedmds: crash when using `config set` on tracked configs
Status:
Resolved
Priority:
Urgent
Assignee:
Category:
Correctness/Safety
Target version:
% Done:
0%
Source:
Community (user)
Tags:
Backport:
mimic
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
crash
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
These configs: https://github.com/ceph/ceph/blob/7dbba9e54282e0a4c3000eb0c1a66e346c7eab98/src/mds/MDSDaemon.cc#L362-L380
Trying to set one with the new-style `ceph config set` causes a dependency cycle assertion:
$ bin/ceph config set mds.c mds_max_purge_ops 15
2018-05-14 13:43:19.778 7fe13ced3700 10 monclient: handle_config config(13 keys) v1 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals no callback set 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored debug_mds 2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals debug_mds = 15/15 (no change) 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored debug_mgrc 2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals debug_mgrc = 20/20 (no change) 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored debug_monc 2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals debug_monc = 20/20 (no change) 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored debug_ms 2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals debug_ms = 1/1 (no change) 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored mds_debug_auth_pins 2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals mds_debug_auth_pins = true (no change) 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored mds_debug_frag 2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals mds_debug_frag = true (no change) 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored mds_debug_scatterstat 2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals mds_debug_scatterstat = true (no change) 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored mds_debug_subtrees 2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals mds_debug_subtrees = true (no change) 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored mds_max_purge_ops 2018-05-14 13:43:19.778 7fe13ced3700 10 set_mon_vals mds_max_purge_ops = 15 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored mds_verify_scatter 2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals mds_verify_scatter = true (no change) 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored mon_pg_warn_min_per_osd 2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals mon_pg_warn_min_per_osd = 3 (no change) 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored osd_pool_default_min_size 2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals osd_pool_default_min_size = 1 (no change) 2018-05-14 13:43:19.778 7fe13ced3700 4 set_mon_vals callback ignored osd_pool_default_size 2018-05-14 13:43:19.778 7fe13ced3700 20 set_mon_vals osd_pool_default_size = 3 (no change) 2018-05-14 13:43:19.782 7fe13ced3700 0 ------------------------------------ existing dependency MDSDaemon::mds_lock (15) -> MonClient::monc_lock (13) at: 2018-05-14 13:43:19.782 7fe13ced3700 0 new dependency MonClient::monc_lock (13) -> MDSDaemon::mds_lock (15) creates a cycle at ceph version 13.1.0-307-gb871031 (b87103157e77d7ae3b3530b366a5d69455e3fe96) nautilus (dev) 1: (MDSDaemon::handle_conf_change(md_config_t const*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+0x50) [0x8a2d2ddd20] 2: (md_config_t::_apply_changes(std::ostream*)+0x503) [0x7fe1449d2e43] 3: (md_config_t::set_mon_vals(CephContext*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)+0x543) [0x7fe1449d3503] 4: (MonClient::handle_config(MConfig*)+0x155) [0x7fe14473fa55] 5: (MonClient::ms_dispatch(Message*)+0x32b) [0x7fe144749f6b] 6: (DispatchQueue::entry()+0xb5a) [0x7fe14476b06a] 7: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fe14480af9d] 8: (()+0x76ba) [0x7fe143fd06ba] 9: (clone()+0x6d) [0x7fe14325a41d] 2018-05-14 13:43:19.782 7fe13ced3700 0 btw, i am holding these locks: 2018-05-14 13:43:19.782 7fe13ced3700 0 MonClient::monc_lock (13) 2018-05-14 13:43:19.782 7fe13ced3700 0 2018-05-14 13:43:19.786 7fe13ced3700 -1 *** Caught signal (Aborted) ** in thread 7fe13ced3700 thread_name:ms_dispatch ceph version 13.1.0-307-gb871031 (b87103157e77d7ae3b3530b366a5d69455e3fe96) nautilus (dev) 1: (()+0x3dbb10) [0x8a2d5bdb10] 2: (()+0x11390) [0x7fe143fda390] 3: (gsignal()+0x38) [0x7fe143188428] 4: (abort()+0x16a) [0x7fe14318a02a] 5: (lockdep_will_lock(char const*, int, bool)+0x9fe) [0x7fe1449df51e] 6: (Mutex::Lock(bool)+0x1b9) [0x7fe1446c4299] 7: (MDSDaemon::handle_conf_change(md_config_t const*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+0x50) [0x8a2d2ddd20] 8: (md_config_t::_apply_changes(std::ostream*)+0x503) [0x7fe1449d2e43] 9: (md_config_t::set_mon_vals(CephContext*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)+0x543) [0x7fe1449d3503] 10: (MonClient::handle_config(MConfig*)+0x155) [0x7fe14473fa55] 11: (MonClient::ms_dispatch(Message*)+0x32b) [0x7fe144749f6b] 12: (DispatchQueue::entry()+0xb5a) [0x7fe14476b06a] 13: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fe14480af9d] 14: (()+0x76ba) [0x7fe143fd06ba] 15: (clone()+0x6d) [0x7fe14325a41d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Updated by Sage Weil almost 6 years ago
- Status changed from New to Fix Under Review
Updated by Sage Weil almost 6 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Nathan Cutler almost 6 years ago
- Copied to Backport #24157: mimic: mds: crash when using `config set` on tracked configs added
Updated by Patrick Donnelly almost 6 years ago
- Status changed from Pending Backport to Resolved
- Assignee set to Sage Weil
Actions