Ceph » CephFS

2018-08-01T00:06:57.866 INFO:teuthology.orchestra.run.smithi006:Running: "sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'sleep 30 && ceph tell mds.* config set mds_cache_memory_limit 1'" 
2018-08-01T00:06:57.870 INFO:teuthology.run_tasks:Running task exec...
2018-08-01T00:06:57.883 INFO:teuthology.task.exec:Executing custom commands...
2018-08-01T00:06:57.883 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi006.front.sepia.ceph.com
2018-08-01T00:06:57.883 INFO:teuthology.orchestra.run.smithi006:Running: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_trim_caps'
2018-08-01T00:06:57.963 INFO:teuthology.orchestra.run.smithi006.stdout:child exits
2018-08-01T00:07:16.289 INFO:teuthology.orchestra.run.smithi006:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'
2018-08-01T00:07:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2018-08-01 00:07:28.044 7f0d1e7fc700  0 client.4224 ms_handle_reset on 172.21.15.6:6813/3168836445
2018-08-01T00:07:28.065 INFO:tasks.ceph.mds.a.smithi006.stderr:terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_get> >'
2018-08-01T00:07:28.065 INFO:tasks.ceph.mds.a.smithi006.stderr:  what():  boost::bad_get: failed value get using boost::get
2018-08-01T00:07:28.065 INFO:tasks.ceph.mds.a.smithi006.stderr:*** Caught signal (Aborted) **
2018-08-01T00:07:28.065 INFO:tasks.ceph.mds.a.smithi006.stderr: in thread 7f501d435700 thread_name:ms_dispatch
2018-08-01T00:07:28.069 INFO:tasks.ceph.mds.a.smithi006.stderr: ceph version 14.0.0-1750-g68ebd2d (68ebd2d14ff0b20f86331ea3f0c0a6a81369b4d6) nautilus (dev)
2018-08-01T00:07:28.069 INFO:tasks.ceph.mds.a.smithi006.stderr: 1: (()+0x11390) [0x7f5024b90390]
2018-08-01T00:07:28.069 INFO:tasks.ceph.mds.a.smithi006.stderr: 2: (gsignal()+0x38) [0x7f50242dd428]
2018-08-01T00:07:28.069 INFO:tasks.ceph.mds.a.smithi006.stderr: 3: (abort()+0x16a) [0x7f50242df02a]
2018-08-01T00:07:28.069 INFO:tasks.ceph.mds.a.smithi006.stderr: 4: (__gnu_cxx::__verbose_terminate_handler()+0x135) [0x7f50257650f5]
2018-08-01T00:07:28.069 INFO:tasks.ceph.mds.a.smithi006.stderr: 5: (__cxxabiv1::__terminate(void (*)())+0x6) [0x7f5025758486]
2018-08-01T00:07:28.069 INFO:tasks.ceph.mds.a.smithi006.stderr: 6: (()+0x74b4d1) [0x7f50257584d1]
2018-08-01T00:07:28.069 INFO:tasks.ceph.mds.a.smithi006.stderr: 7: (()+0x74b854) [0x7f5025758854]
2018-08-01T00:07:28.070 INFO:tasks.ceph.mds.a.smithi006.stderr: 8: (()+0x1431ed) [0x564afeaab1ed]
2018-08-01T00:07:28.070 INFO:tasks.ceph.mds.a.smithi006.stderr: 9: (unsigned long const md_config_t::get_val<unsigned long>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x85) [0x564afeb1f735]
2018-08-01T00:07:28.070 INFO:tasks.ceph.mds.a.smithi006.stderr: 10: (MDCache::handle_conf_change(ConfigProxy const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, MDSMap const&)+0x154) [0x564afeb89764]
2018-08-01T00:07:28.070 INFO:tasks.ceph.mds.a.smithi006.stderr: 11: (MDSDaemon::handle_conf_change(ConfigProxy const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+0x230) [0x564afea644d0]
2018-08-01T00:07:28.070 INFO:tasks.ceph.mds.a.smithi006.stderr: 12: (void ObserverMgr<ceph::internal::md_config_obs_impl<ConfigProxy> >::apply_changes<ConfigProxy>(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, ConfigProxy const&, std::ostream*)+0x2b3) [0x564afea72e13]
2018-08-01T00:07:28.070 INFO:tasks.ceph.mds.a.smithi006.stderr: 13: (ConfigProxy::apply_changes(std::ostream*)+0x47) [0x564afea73047]
2018-08-01T00:07:28.070 INFO:tasks.ceph.mds.a.smithi006.stderr: 14: (MDSDaemon::_handle_command(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&, MCommand*, ceph::buffer::list*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, Context**, bool*)+0xddf) [0x564afea6b97f]
2018-08-01T00:07:28.070 INFO:tasks.ceph.mds.a.smithi006.stderr: 15: (MDSDaemon::handle_command(MCommand*)+0x70f) [0x564afea6c7ff]
2018-08-01T00:07:28.070 INFO:tasks.ceph.mds.a.smithi006.stderr: 16: (MDSDaemon::handle_core_message(Message*)+0xb1) [0x564afea6cbd1]
2018-08-01T00:07:28.071 INFO:tasks.ceph.mds.a.smithi006.stderr: 17: (MDSDaemon::ms_dispatch(Message*)+0xab) [0x564afea6d26b]
2018-08-01T00:07:28.071 INFO:tasks.ceph.mds.a.smithi006.stderr: 18: (DispatchQueue::entry()+0xb92) [0x7f5025479122]
2018-08-01T00:07:28.071 INFO:tasks.ceph.mds.a.smithi006.stderr: 19: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f502550f5ad]
2018-08-01T00:07:28.071 INFO:tasks.ceph.mds.a.smithi006.stderr: 20: (()+0x76ba) [0x7f5024b866ba]
2018-08-01T00:07:28.071 INFO:tasks.ceph.mds.a.smithi006.stderr: 21: (clone()+0x6d) [0x7f50243af41d]

From: /ceph/teuthology-archive/pdonnell-2018-07-31_23:55:57-fs-wip-pdonnell-testing-20180731.215430-testing-basic-smithi/2845070/teuthology.log

Suspect cause:

commit 922bfc5f3b333f576a1444773fff6db7100dd24a HEAD (upstream/pull/22607/head)
Author: Sage Weil <sage@redhat.com>
Date:   Mon Jun 18 16:56:58 2018 -0500

    common/options: convert many TYPE_[U]INT -> TYPE_SIZE

    Note that the _cost options are in fact in units of bytes.

    Signed-off-by: Sage Weil <sage@redhat.com>
---
 src/mds/MDCache.cc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/mds/MDCache.cc b/src/mds/MDCache.cc
index c465d6cc70..99cebef66b 100644
--- a/src/mds/MDCache.cc
+++ b/src/mds/MDCache.cc
@@ -210,7 +210,7 @@ MDCache::MDCache(MDSRank *m, PurgeQueue &purge_queue_) :
   opening_root = open = false;

   cache_inode_limit = g_conf().get_val<int64_t>("mds_cache_size");
-  cache_memory_limit = g_conf().get_val<uint64_t>("mds_cache_memory_limit");
+  cache_memory_limit = g_conf().get_val<Option::size_t>("mds_cache_memory_limit");
   cache_reservation = g_conf().get_val<double>("mds_cache_reservation");
   cache_health_threshold = g_conf().get_val<double>("mds_health_cache_threshold");