Actions
Bug #48381
closedMemory leak ceph-mon in ConfigMonitor::load_config
% Done:
0%
Description
Hello everyone!
I have a memory leak on 2 clusters: QAS, PROD after update from Ceph version 13.2.5 to 15.2.5 in ceph-mon process.
One hour uptime statistics:
# ceph tell mon.service01-qas heap dump
mon.service01-qas dumping heap profile now.
------------------------------------------------
MALLOC: 222469760 ( 212.2 MiB) Bytes in use by application
MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
MALLOC: + 6777856 ( 6.5 MiB) Bytes in central cache freelist
MALLOC: + 3080704 ( 2.9 MiB) Bytes in transfer cache freelist
MALLOC: + 7320448 ( 7.0 MiB) Bytes in thread cache freelists
MALLOC: + 3014656 ( 2.9 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 242663424 ( 231.4 MiB) Actual memory used (physical + swap)
MALLOC: + 2154496 ( 2.1 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 244817920 ( 233.5 MiB) Virtual address space used
MALLOC:
MALLOC: 10291 Spans in use
MALLOC: 24 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
This is result after 11 days of proccess running:
# ceph tell mon.service01-qas heap stats
mon.service01-qas tcmalloc heap stats:------------------------------------------------
MALLOC: 3582733448 ( 3416.8 MiB) Bytes in use by application
MALLOC: + 16384 ( 0.0 MiB) Bytes in page heap freelist
MALLOC: + 64338336 ( 61.4 MiB) Bytes in central cache freelist
MALLOC: + 8880896 ( 8.5 MiB) Bytes in transfer cache freelist
MALLOC: + 41391832 ( 39.5 MiB) Bytes in thread cache freelists
MALLOC: + 28442624 ( 27.1 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 3725803520 ( 3553.2 MiB) Actual memory used (physical + swap)
MALLOC: + 7127040 ( 6.8 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 3732930560 ( 3560.0 MiB) Virtual address space used
MALLOC:
MALLOC: 437764 Spans in use
MALLOC: 25 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
# pprof --text /usr/bin/ceph-mon /var/log/ceph/mon.service01-qas.profile.1900.heap
Using local file /usr/bin/ceph-mon.
Using local file /var/log/ceph/mon.service01-qas.profile.1900.heap.
Total: 3150.8 MB
2702.7 85.8% 85.8% 3078.1 97.7% ConfigMonitor::load_config
375.5 11.9% 97.7% 375.5 11.9% std::string::_Rep::_S_create
60.4 1.9% 99.6% 60.4 1.9% rocksdb::BlockFetcher::ReadBlockContents
8.4 0.3% 99.9% 8.4 0.3% ceph::logging::Log::submit_entry
1.0 0.0% 99.9% 1.0 0.0% rocksdb::WritableFileWriter::Append
0.6 0.0% 99.9% 0.6 0.0% ceph::buffer::v15_2_0::list::refill_append_space
0.5 0.0% 99.9% 0.5 0.0% std::_Rb_tree::_M_copy
0.5 0.0% 100.0% 0.5 0.0% rocksdb_cache::BinnedLRUHandleTable::Resize
Cluster state:
# ceph status
cluster:
id: 7a2137a1-a6c6-401c-ab31-37da3ba074b8
health: HEALTH_OK
services:
mon: 4 daemons, quorum service01-qas,service02-qas,service03-qas,service04-qas (age 10d)
mgr: service04-qas(active, since 5w), standbys: service02-qas, service03-qas, service01-qas
mds: remote_files:1 {0=service04-qas=up:active} 3 up:standby
osd: 4 osds: 4 up (since 4w), 4 in (since 4w); 32 remapped pgs
task status:
scrub status:
mds.service04-qas: idle
data:
pools: 8 pools, 256 pgs
objects: 195.79k objects, 35 GiB
usage: 91 GiB used, 1.9 TiB / 2.0 TiB avail
pgs: 224 active+clean
32 active+clean+remapped
Configuration:
# ceph config show mon.service01-qas
NAME VALUE SOURCE OVERRIDES IGNORES
auth_client_required cephx file
auth_cluster_required cephx file
auth_service_required cephx file
daemonize false override
keyring $mon_data/keyring default
leveldb_block_size 65536 default
leveldb_cache_size 536870912 default
leveldb_compression false default
leveldb_log default
leveldb_write_buffer_size 33554432 default
mon_host 10.189.44.81,10.189.44.82,10.189.44.83,10.189.44.84 file
mon_initial_members service01-qas, service02-qas, service03-qas, service04-qas file
no_config_file false override
public_network 10.189.44.0/24 file
rbd_default_features 61 default
setgroup ceph cmdline
setuser ceph cmdline
Updated by Aleksey Ivanov over 3 years ago
After 26 days
# pprof --text /usr/bin/ceph-mon /var/log/ceph/mon.service01-qas.profile.4567.heap
Using local file /usr/bin/ceph-mon.
Using local file /var/log/ceph/mon.service01-qas.profile.4567.heap.
Total: 7472.4 MB
6488.8 86.8% 86.8% 7390.1 98.9% ConfigMonitor::load_config
901.6 12.1% 98.9% 901.6 12.1% std::string::_Rep::_S_create
59.7 0.8% 99.7% 59.7 0.8% rocksdb::BlockFetcher::ReadBlockContents
18.1 0.2% 99.9% 18.1 0.2% ceph::logging::Log::submit_entry
1.3 0.0% 100.0% 1.3 0.0% ceph::buffer::v15_2_0::list::refill_append_space
1.0 0.0% 100.0% 1.0 0.0% rocksdb::WritableFileWriter::Append
0.5 0.0% 100.0% 0.5 0.0% rocksdb_cache::BinnedLRUHandleTable::Resize
0.3 0.0% 100.0% 0.3 0.0% std::_Rb_tree::_M_copy
0.3 0.0% 100.0% 0.8 0.0% rocksdb_cache::BinnedLRUCacheShard::Insert
0.3 0.0% 100.0% 1.0 0.0% rocksdb::BlockBasedTable::PutDataBlockToCache
0.1 0.0% 100.0% 0.1 0.0% std::_Rb_tree::_M_emplace_unique
# ceph tell mon.service01-qas heap stats
mon.service01-qas tcmalloc heap stats:------------------------------------------------
MALLOC: 8449475032 ( 8058.0 MiB) Bytes in use by application
MALLOC: + 245760 ( 0.2 MiB) Bytes in page heap freelist
MALLOC: + 129710176 ( 123.7 MiB) Bytes in central cache freelist
MALLOC: + 5285376 ( 5.0 MiB) Bytes in transfer cache freelist
MALLOC: + 81313736 ( 77.5 MiB) Bytes in thread cache freelists
MALLOC: + 62652416 ( 59.8 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 8728682496 ( 8324.3 MiB) Actual memory used (physical + swap)
MALLOC: + 5857280 ( 5.6 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 8734539776 ( 8329.9 MiB) Virtual address space used
MALLOC:
MALLOC: 1040801 Spans in use
MALLOC: 25 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
Updated by Theofilos Mouratidis about 3 years ago
We have this bug on 2 of our clusters, we have to restart the mons once in a while so they don't go OOM
[14:17][---(production:ceph/kelly/mon*2:peon) ~]# ceph tell mon.`hostname -s` heap stats
mon.--- tcmalloc heap stats:------------------------------------------------
MALLOC: 4874807224 ( 4649.0 MiB) Bytes in use by application
MALLOC: + 20340736 ( 19.4 MiB) Bytes in page heap freelist
MALLOC: + 143053312 ( 136.4 MiB) Bytes in central cache freelist
MALLOC: + 9022976 ( 8.6 MiB) Bytes in transfer cache freelist
MALLOC: + 47257672 ( 45.1 MiB) Bytes in thread cache freelists
MALLOC: + 33554432 ( 32.0 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 5128036352 ( 4890.5 MiB) Actual memory used (physical + swap)
MALLOC: + 70123520 ( 66.9 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 5198159872 ( 4957.4 MiB) Virtual address space used
MALLOC:
MALLOC: 537698 Spans in use
MALLOC: 24 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
[14:20][--- (production:ceph/kelly/mon*2:peon) ~]# pprof --text /usr/bin/ceph-mon /var/log/ceph/mon.---.profile.5844.heap
Using local file /usr/bin/ceph-mon.
Using local file /var/log/ceph/mon.---.profile.5844.heap.
Total: 3177.7 MB
2318.9 73.0% 73.0% 2612.9 82.2% ConfigMonitor::load_config
475.8 15.0% 87.9% 475.8 15.0% rocksdb::BlockFetcher::ReadBlockContents
292.1 9.2% 97.1% 292.1 9.2% std::string::_Rep::_S_create
51.4 1.6% 98.8% 51.4 1.6% ceph::buffer::v14_2_0::list::refill_append_space
19.2 0.6% 99.4% 19.2 0.6% ceph::logging::Log::submit_entry
16.0 0.5% 99.9% 16.0 0.5% rocksdb::Arena::AllocateNewBlock
2.0 0.1% 99.9% 2.0 0.1% rocksdb_cache::BinnedLRUCacheShard::Insert
1.6 0.1% 100.0% 3.6 0.1% rocksdb::BlockBasedTable::PutDataBlockToCache
0.1 0.0% 100.0% 0.1 0.0% ceph::buffer::v14_2_0::create_aligned_in_mempool
0.1 0.0% 100.0% 0.1 0.0% rocksdb::DBImpl::SwitchMemtable
0.1 0.0% 100.0% 0.1 0.0% std::_Rb_tree::_M_emplace_hint_unique
0.1 0.0% 100.0% 0.1 0.0% std::_Rb_tree::_M_copy
0.0 0.0% 100.0% 2.0 0.1% OSDMonitor::get_version
0.0 0.0% 100.0% 0.1 0.0% AsyncConnection::AsyncConnection
0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_emplace_unique
0.0 0.0% 100.0% 0.0 0.0% std::__detail::_Map_base::operator[]
0.0 0.0% 100.0% 49.5 1.6% OSDMonitor::get_version_full
0.0 0.0% 100.0% 0.0 0.0% MgrMap::ModuleInfo::decode
0.0 0.0% 100.0% 0.0 0.0% MessageFactory::build
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::ptr_node::cloner::operator
0.0 0.0% 100.0% 0.0 0.0% LogSummary::add
0.0 0.0% 100.0% 0.1 0.0% MLog::decode_payload
0.0 0.0% 100.0% 3157.1 99.4% Monitor::_ms_dispatch
0.0 0.0% 100.0% 0.0 0.0% CryptoAES::get_key_handler
0.0 0.0% 100.0% 0.0 0.0% OSDMonitor::get_tracked_conf_keys
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::ProtocolV1
0.0 0.0% 100.0% 0.0 0.0% std::vector::_M_default_append
0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_insert_unique
0.0 0.0% 100.0% 0.0 0.0% LogEntry::LogEntry
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::list::append@5a80e0
0.0 0.0% 100.0% 0.1 0.0% AsyncMessenger::add_accept
0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_insert_
0.0 0.0% 100.0% 0.0 0.0% EventCenter::create_time_event
0.0 0.0% 100.0% 0.0 0.0% rocksdb::VersionSet::ProcessManifestWrites
0.0 0.0% 100.0% 0.1 0.0% PGMapDigest::decode
0.0 0.0% 100.0% 0.0 0.0% rocksdb::ColumnFamilyData::ConstructNewMemtable
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::ptr_node::create_hypercombined
0.0 0.0% 100.0% 0.0 0.0% LogClient::_get_mon_log_message
0.0 0.0% 100.0% 0.0 0.0% std::deque::_M_push_back_aux
0.0 0.0% 100.0% 0.1 0.0% ceph::decode
0.0 0.0% 100.0% 0.1 0.0% rocksdb::BlockBasedTable::Open
0.0 0.0% 100.0% 0.0 0.0% std::_Deque_base::_M_initialize_map
0.0 0.0% 100.0% 0.3 0.0% EventCenter::process_events
0.0 0.0% 100.0% 0.0 0.0% 0x00007ff080ea85b8
0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_emplace_equal
0.0 0.0% 100.0% 0.0 0.0% std::deque::_M_reallocate_map
0.0 0.0% 100.0% 0.0 0.0% rocksdb::ReadProperties
0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::create_connect
0.0 0.0% 100.0% 0.0 0.0% std::deque::emplace_back
0.0 0.0% 100.0% 0.0 0.0% std::vector::operator=
0.0 0.0% 100.0% 0.8 0.0% Monitor::handle_subscribe
0.0 0.0% 100.0% 0.0 0.0% rocksdb::ConcurrentArena::ConcurrentArena
0.0 0.0% 100.0% 0.0 0.0% rocksdb::VersionSet::LogAndApply
0.0 0.0% 100.0% 0.0 0.0% rocksdb::DBImpl::SchedulePendingFlush
0.0 0.0% 100.0% 0.0 0.0% CryptoKey::_set_secret
0.0 0.0% 100.0% 0.0 0.0% rocksdb::VersionStorageInfo::VersionStorageInfo
0.0 0.0% 100.0% 0.1 0.0% decode_message@4785c0
0.0 0.0% 100.0% 0.0 0.0% rocksdb::SuperVersionContext::SuperVersionContext
0.0 0.0% 100.0% 0.0 0.0% MonSessionMap::new_session
0.0 0.0% 100.0% 0.0 0.0% Protocol::Protocol
0.0 0.0% 100.0% 0.0 0.0% OSDMap::apply_incremental
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::list::append@5a6740
0.0 0.0% 100.0% 0.0 0.0% rocksdb::BlockBasedTable::ReadFilter
0.0 0.0% 100.0% 0.0 0.0% rocksdb::VersionBuilder::Rep::Apply
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::reset_recv_state
0.0 0.0% 100.0% 0.1 0.0% rocksdb::TableCache::GetTableReader
0.0 0.0% 100.0% 0.1 0.0% rocksdb::BlockBasedTable::VerifyChecksumInBlocks
0.0 0.0% 100.0% 0.0 0.0% rocksdb::LRUCacheShard::Insert
0.0 0.0% 100.0% 0.0 0.0% CephXTicketHandler::build_authorizer
0.0 0.0% 100.0% 292.0 9.2% std::basic_string::basic_string
0.0 0.0% 100.0% 0.0 0.0% OSDMonitor::start_mapping
0.0 0.0% 100.0% 0.0 0.0% SafeTimer::add_event_at
0.0 0.0% 100.0% 0.0 0.0% ceph::crypto::onwire::rxtx_t::create_handler_pair
0.0 0.0% 100.0% 0.0 0.0% rocksdb::SkipListFactory::CreateMemTableRep
0.0 0.0% 100.0% 0.0 0.0% rocksdb::Env::Default
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::handle_client_banner
0.0 0.0% 100.0% 0.0 0.0% boost::detail::function::function_obj_invoker4::invoke
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::list::append@5a7760
0.0 0.0% 100.0% 0.0 0.0% std::vector::vector
0.0 0.0% 100.0% 0.1 0.0% rocksdb::BinarySearchIndexReader::Create
0.0 0.0% 100.0% 0.0 0.0% rocksdb::FullFilterBitsBuilder::~FullFilterBitsBuilder
0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::connect
0.0 0.0% 100.0% 0.0 0.0% Monitor::forward_request_leader
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::handle_server_ident
0.0 0.0% 100.0% 0.0 0.0% OpTracker::record_history_op
0.0 0.0% 100.0% 0.0 0.0% std::__shared_ptr::reset
0.0 0.0% 100.0% 0.0 0.0% rocksdb::MutableCFOptions::MutableCFOptions
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::list::append@5a74d0
0.0 0.0% 100.0% 0.0 0.0% rocksdb::BlockBasedTable::ReadPropertiesBlock
0.0 0.0% 100.0% 0.0 0.0% PosixServerSocketImpl::accept
0.0 0.0% 100.0% 0.0 0.0% PosixWorker::connect
0.0 0.0% 100.0% 0.0 0.0% get_auth_service_handler
0.0 0.0% 100.0% 0.0 0.0% MgrClient::reconnect
0.0 0.0% 100.0% 0.0 0.0% std::vector::reserve
0.0 0.0% 100.0% 0.0 0.0% MgrMonitor::send_digests
0.0 0.0% 100.0% 0.0 0.0% Monitor::new_tick
0.0 0.0% 100.0% 0.0 0.0% Paxos::reset_lease_timeout
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::open
0.0 0.0% 100.0% 50.5 1.6% ceph::buffer::v14_2_0::list::append@5a8400
0.0 0.0% 100.0% 0.0 0.0% json_spirit::read_range_or_throw
0.0 0.0% 100.0% 0.0 0.0% rocksdb::MemTableList::InstallNewVersion
0.0 0.0% 100.0% 0.0 0.0% MgrClient::_send_stats
0.0 0.0% 100.0% 0.0 0.0% std::vector::_M_realloc_insert
0.0 0.0% 100.0% 0.0 0.0% MonmapMonitor::apply_mon_features
0.0 0.0% 100.0% 0.0 0.0% MMonPaxos::decode_payload
0.0 0.0% 100.0% 0.0 0.0% PaxosService::wait_for_writeable
0.0 0.0% 100.0% 0.0 0.0% get_auth_session_handler
0.0 0.0% 100.0% 0.0 0.0% osd_xinfo_t::decode
0.0 0.0% 100.0% 3177.7 100.0% 0x00007ff07fae896c
0.0 0.0% 100.0% 3177.7 100.0% 0x00007ff080c25ea4
0.0 0.0% 100.0% 0.0 0.0% 0x00007ff080f65dc6
0.0 0.0% 100.0% 0.0 0.0% 0x00007ff080f66ab8
0.0 0.0% 100.0% 0.0 0.0% AdminHook::call
0.0 0.0% 100.0% 0.0 0.0% AdminSocket::do_accept
0.0 0.0% 100.0% 0.0 0.0% AdminSocket::entry
0.0 0.0% 100.0% 0.0 0.0% AdminSocket::execute_command
0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::_stop
0.0 0.0% 100.0% 0.3 0.0% AsyncConnection::process
0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::send_message
0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::tick
0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::accept_conn
0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::connect_to
0.0 0.0% 100.0% 0.0 0.0% AuthMonitor::prep_auth
0.0 0.0% 100.0% 0.0 0.0% AuthMonitor::preprocess_query
0.0 0.0% 100.0% 0.0 0.0% AuthMonitor::update_from_paxos
0.0 0.0% 100.0% 0.0 0.0% C_ApplyFeatures::finish
0.0 0.0% 100.0% 0.9 0.0% C_MonContext::finish
0.0 0.0% 100.0% 0.0 0.0% C_MonOp::finish
0.0 0.0% 100.0% 19.2 0.6% CephRocksdbLogger::Logv
0.0 0.0% 100.0% 0.0 0.0% CephXAuthorizer::verify_reply
0.0 0.0% 100.0% 0.0 0.0% CephxServiceHandler::handle_request
0.0 0.0% 100.0% 0.0 0.0% ConfigMap::parse_mask
0.0 0.0% 100.0% 0.0 0.0% ConfigMonitor::check_all_subs
0.0 0.0% 100.0% 0.0 0.0% ConfigMonitor::maybe_send_config
0.0 0.0% 100.0% 0.0 0.0% ConfigMonitor::refresh_config
0.0 0.0% 100.0% 0.0 0.0% ConfigMonitor::update_from_paxos
0.0 0.0% 100.0% 0.0 0.0% ConfigProxy::set_mon_vals
0.0 0.0% 100.0% 0.9 0.0% Context::complete
0.0 0.0% 100.0% 0.0 0.0% CryptoAES::create
0.0 0.0% 100.0% 0.0 0.0% CryptoAESKeyHandler::encrypt
0.0 0.0% 100.0% 0.0 0.0% CryptoKey::decode
0.0 0.0% 100.0% 0.0 0.0% CryptoKey::set_secret
0.0 0.0% 100.0% 3157.1 99.4% DispatchQueue::DispatchThread::entry
0.0 0.0% 100.0% 3157.1 99.4% DispatchQueue::entry
0.0 0.0% 100.0% 0.0 0.0% DispatchQueue::local_delivery
0.0 0.0% 100.0% 3157.1 99.4% Dispatcher::ms_dispatch2
0.0 0.0% 100.0% 0.0 0.0% EntityName::set
0.0 0.0% 100.0% 0.0 0.0% EpollDriver::event_wait
0.0 0.0% 100.0% 0.0 0.0% EventCenter::process_time_events
0.0 0.0% 100.0% 0.0 0.0% FunctionContext::finish
0.0 0.0% 100.0% 0.0 0.0% KeyServer::_build_session_auth_info
0.0 0.0% 100.0% 0.0 0.0% KeyServer::_get_service_caps
0.0 0.0% 100.0% 0.0 0.0% KeyServer::build_session_auth_info
0.0 0.0% 100.0% 0.0 0.0% KeyServer::generate_secret
0.0 0.0% 100.0% 0.0 0.0% KeyServer::get_service_caps
0.0 0.0% 100.0% 0.0 0.0% KeyServerData::Incremental::decode
0.0 0.0% 100.0% 0.0 0.0% KeyServerData::get_caps
0.0 0.0% 100.0% 0.0 0.0% LogChannel::do_log@302a40
0.0 0.0% 100.0% 0.0 0.0% LogChannel::do_log@303080
0.0 0.0% 100.0% 0.0 0.0% LogClient::_send_to_mon
0.0 0.0% 100.0% 0.0 0.0% LogClient::queue
0.0 0.0% 100.0% 0.0 0.0% LogClientTemp::~LogClientTemp
0.0 0.0% 100.0% 0.1 0.0% LogEntry::decode
0.0 0.0% 100.0% 292.0 9.2% LogMonitor::log_channel_info::do_log_to_syslog
0.0 0.0% 100.0% 0.0 0.0% LogMonitor::preprocess_log
0.0 0.0% 100.0% 0.0 0.0% LogMonitor::preprocess_query
0.0 0.0% 100.0% 0.1 0.0% LogMonitor::update_from_paxos
0.0 0.0% 100.0% 0.0 0.0% MLog::encode_payload
0.0 0.0% 100.0% 0.0 0.0% MMgrBeacon::decode_payload
0.0 0.0% 100.0% 0.0 0.0% MMonSubscribe::decode_payload
0.0 0.0% 100.0% 0.0 0.0% MTimeCheck2::decode_payload
0.0 0.0% 100.0% 0.0 0.0% Message::encode
0.0 0.0% 100.0% 0.0 0.0% MgrClient::_send_report
0.0 0.0% 100.0% 0.0 0.0% MgrClient::handle_mgr_configure
0.0 0.0% 100.0% 0.1 0.0% MgrClient::handle_mgr_map
0.0 0.0% 100.0% 0.1 0.0% MgrClient::ms_dispatch
0.0 0.0% 100.0% 0.0 0.0% MgrClient::ms_handle_reset
0.0 0.0% 100.0% 0.0 0.0% MgrClient::update_daemon_health
0.0 0.0% 100.0% 0.0 0.0% MgrMap::ModuleOption::decode
0.0 0.0% 100.0% 0.0 0.0% MgrMonitor::preprocess_beacon
0.0 0.0% 100.0% 0.0 0.0% MgrMonitor::preprocess_query
0.0 0.0% 100.0% 0.1 0.0% MgrMonitor::prime_mgr_client
0.0 0.0% 100.0% 2613.0 82.2% MgrMonitor::update_from_paxos
0.0 0.0% 100.0% 3.0 0.1% MgrStatMonitor::update_from_paxos
0.0 0.0% 100.0% 0.0 0.0% MonCap::parse
0.0 0.0% 100.0% 0.0 0.0% MonOpRequest::mark_logmon_event
0.0 0.0% 100.0% 0.0 0.0% MonOpRequest::mark_paxos_event
0.0 0.0% 100.0% 3157.0 99.4% Monitor::dispatch_op
0.0 0.0% 100.0% 0.0 0.0% Monitor::do_admin_command
0.0 0.0% 100.0% 0.0 0.0% Monitor::get_auth_request
0.0 0.0% 100.0% 0.0 0.0% Monitor::handle_auth_done
0.0 0.0% 100.0% 0.0 0.0% Monitor::handle_command
0.0 0.0% 100.0% 0.0 0.0% Monitor::handle_route
0.0 0.0% 100.0% 0.2 0.0% Monitor::load_metadata
0.0 0.0% 100.0% 3157.1 99.4% Monitor::ms_dispatch
0.0 0.0% 100.0% 0.0 0.0% Monitor::ms_get_authorizer
0.0 0.0% 100.0% 0.0 0.0% Monitor::ms_handle_authentication
0.0 0.0% 100.0% 0.0 0.0% Monitor::no_reply
0.0 0.0% 100.0% 3140.1 98.8% Monitor::refresh_from_paxos
0.0 0.0% 100.0% 0.9 0.0% Monitor::tick
0.0 0.0% 100.0% 16.0 0.5% MonitorDBStore::apply_transaction
0.0 0.0% 100.0% 424.1 13.3% MonitorDBStore::exists
0.0 0.0% 100.0% 49.7 1.6% MonitorDBStore::get
0.0 0.0% 100.0% 1.4 0.0% MonitorDBStore::get_iterator
0.0 0.0% 100.0% 0.3 0.0% NetworkStack::create_worker
0.0 0.0% 100.0% 1.0 0.0% OSDMap::Incremental::encode
0.0 0.0% 100.0% 1.0 0.0% OSDMonitor::build_incremental
0.0 0.0% 100.0% 1.0 0.0% OSDMonitor::check_osdmap_sub
0.0 0.0% 100.0% 0.2 0.0% OSDMonitor::check_osdmap_subs
0.0 0.0% 100.0% 424.1 13.3% OSDMonitor::load_osdmap_manifest
0.0 0.0% 100.0% 1.0 0.0% OSDMonitor::reencode_incremental_map
0.0 0.0% 100.0% 1.0 0.0% OSDMonitor::send_incremental
0.0 0.0% 100.0% 0.9 0.0% OSDMonitor::tick
0.0 0.0% 100.0% 473.9 14.9% OSDMonitor::update_from_paxos
0.0 0.0% 100.0% 0.0 0.0% OpHistory::_insert_delayed
0.0 0.0% 100.0% 0.0 0.0% OpHistoryServiceThread::entry
0.0 0.0% 100.0% 0.0 0.0% ParallelPGMapper::Job::Job
0.0 0.0% 100.0% 0.0 0.0% ParallelPGMapper::queue
0.0 0.0% 100.0% 3156.2 99.3% Paxos::dispatch
0.0 0.0% 100.0% 3140.1 98.8% Paxos::do_refresh
0.0 0.0% 100.0% 8.0 0.3% Paxos::handle_begin
0.0 0.0% 100.0% 3148.2 99.1% Paxos::handle_commit
0.0 0.0% 100.0% 0.0 0.0% Paxos::handle_lease
0.0 0.0% 100.0% 8.1 0.3% Paxos::store_state
0.0 0.0% 100.0% 0.0 0.0% PaxosService::C_RetryMessage::_finish
0.0 0.0% 100.0% 0.0 0.0% PaxosService::dispatch
0.0 0.0% 100.0% 0.4 0.0% PaxosService::load_health
0.0 0.0% 100.0% 3139.5 98.8% PaxosService::refresh
0.0 0.0% 100.0% 0.0 0.0% PaxosService::wait_for_readable
0.0 0.0% 100.0% 0.0 0.0% PerfCountersCollection::with_counters
0.0 0.0% 100.0% 0.1 0.0% Processor::accept
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::accept
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::append_keepalive_or_ack
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::fault
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::handle_connect_message_1
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::handle_connect_message_2
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::handle_keepalive2
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::handle_message
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::handle_message_footer
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::read_event
0.0 0.0% 100.0% 0.0 0.0% ProtocolV1::stop
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::ProtocolV2
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::_fault
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::handle_auth_done
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::handle_frame_payload
0.0 0.0% 100.0% 0.1 0.0% ProtocolV2::handle_message
0.0 0.0% 100.0% 0.1 0.0% ProtocolV2::handle_read_frame_dispatch
0.0 0.0% 100.0% 0.1 0.0% ProtocolV2::handle_read_frame_epilogue_main
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::handle_read_frame_preamble_main
0.0 0.0% 100.0% 0.1 0.0% ProtocolV2::handle_read_frame_segment
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::post_client_banner_exchange
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::read_event
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::read_frame
0.0 0.0% 100.0% 0.1 0.0% ProtocolV2::read_frame_segment
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::ready
0.0 0.0% 100.0% 0.3 0.0% ProtocolV2::run_continuation
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::send_auth_request
0.0 0.0% 100.0% 0.0 0.0% ProtocolV2::stop
0.0 0.0% 100.0% 0.1 0.0% RocksDBStore::CompactThread::entry
0.0 0.0% 100.0% 424.7 13.4% RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound
0.0 0.0% 100.0% 0.0 0.0% RocksDBStore::RocksDBWholeSpaceIteratorImpl::raw_key_is_prefixed
0.0 0.0% 100.0% 1.4 0.0% RocksDBStore::RocksDBWholeSpaceIteratorImpl::seek_to_first@46dab0
0.0 0.0% 100.0% 0.0 0.0% RocksDBStore::RocksDBWholeSpaceIteratorImpl::value
0.0 0.0% 100.0% 0.1 0.0% RocksDBStore::compact_range
0.0 0.0% 100.0% 0.1 0.0% RocksDBStore::compact_thread_entry
0.0 0.0% 100.0% 103.6 3.3% RocksDBStore::get
0.0 0.0% 100.0% 16.0 0.5% RocksDBStore::submit_common
0.0 0.0% 100.0% 16.0 0.5% RocksDBStore::submit_transaction_sync
0.0 0.0% 100.0% 0.0 0.0% SafeTimer::add_event_after
0.0 0.0% 100.0% 0.9 0.0% SafeTimer::timer_thread
0.0 0.0% 100.0% 0.9 0.0% SafeTimerThread::entry
0.0 0.0% 100.0% 0.0 0.0% TrackedOp::mark_event
0.0 0.0% 100.0% 0.0 0.0% TrackedOp::put
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::copy
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::create
0.0 0.0% 100.0% 0.1 0.0% ceph::buffer::v14_2_0::create_aligned
0.0 0.0% 100.0% 1.0 0.0% ceph::buffer::v14_2_0::list::append_hole
0.0 0.0% 100.0% 0.1 0.0% ceph::buffer::v14_2_0::list::iterator_impl::copy
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::list::iterator_impl::copy_deep
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::list::list
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::list::obtain_contiguous_space
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::list::splice
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::list::to_str
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::v14_2_0::ptr::ptr
0.0 0.0% 100.0% 0.0 0.0% ceph::crypto::onwire::AES128GCM_OnWireRxHandler::authenticated_decrypt_update
0.0 0.0% 100.0% 0.0 0.0% ceph::decode@27f4a0
0.0 0.0% 100.0% 0.0 0.0% ceph::decode@48b490
0.0 0.0% 100.0% 0.0 0.0% ceph::encode
0.0 0.0% 100.0% 0.0 0.0% ceph::encode@48bba0
0.0 0.0% 100.0% 0.0 0.0% cephx_build_service_ticket_blob
0.0 0.0% 100.0% 0.0 0.0% cmdmap_from_json
0.0 0.0% 100.0% 0.0 0.0% creating_pgs_t::decode
0.0 0.0% 100.0% 0.0 0.0% encode_message
0.0 0.0% 100.0% 0.0 0.0% entity_addrvec_t::decode
0.0 0.0% 100.0% 0.0 0.0% handle_mon_signal
0.0 0.0% 100.0% 0.0 0.0% int decode_decrypt@2cfff0
0.0 0.0% 100.0% 0.0 0.0% int encode_encrypt
0.0 0.0% 100.0% 0.0 0.0% json_spirit::read
0.0 0.0% 100.0% 0.0 0.0% json_spirit::read_range
0.0 0.0% 100.0% 0.0 0.0% md_config_t::_expand_meta
0.0 0.0% 100.0% 0.0 0.0% md_config_t::_get_val
0.0 0.0% 100.0% 0.0 0.0% md_config_t::set_mon_vals
0.0 0.0% 100.0% 0.0 0.0% md_config_t::update_legacy_val
0.0 0.0% 100.0% 0.0 0.0% md_config_t::update_legacy_vals
0.0 0.0% 100.0% 0.0 0.0% osd_stat_t::decode
0.0 0.0% 100.0% 0.0 0.0% pow2_hist_t::decode
0.0 0.0% 100.0% 16.0 0.5% rocksdb::AllocTracker::~AllocTracker
0.0 0.0% 100.0% 16.0 0.5% rocksdb::Arena::AllocateFallback
0.0 0.0% 100.0% 0.1 0.0% rocksdb::BlockBasedTable::CreateIndexReader
0.0 0.0% 100.0% 53.2 1.7% rocksdb::BlockBasedTable::Get
0.0 0.0% 100.0% 0.0 0.0% rocksdb::BlockBasedTable::GetFilter@5ceb70
0.0 0.0% 100.0% 0.0 0.0% rocksdb::BlockBasedTable::GetFilter@5d3180
0.0 0.0% 100.0% 479.3 15.1% rocksdb::BlockBasedTable::MaybeReadBlockAndLoadToCache
0.0 0.0% 100.0% 479.3 15.1% rocksdb::BlockBasedTable::NewDataBlockIterator
0.0 0.0% 100.0% 0.1 0.0% rocksdb::BlockBasedTable::NewIndexIterator
0.0 0.0% 100.0% 0.1 0.0% rocksdb::BlockBasedTable::PrefetchIndexAndFilterBlocks
0.0 0.0% 100.0% 0.1 0.0% rocksdb::BlockBasedTableFactory::NewTableReader
0.0 0.0% 100.0% 421.8 13.3% rocksdb::BlockBasedTableIterator::FindKeyForward
0.0 0.0% 100.0% 426.2 13.4% rocksdb::BlockBasedTableIterator::InitDataBlock
0.0 0.0% 100.0% 4.3 0.1% rocksdb::BlockBasedTableIterator::Seek
0.0 0.0% 100.0% 0.0 0.0% rocksdb::ColumnFamilyData::InstallSuperVersion
0.0 0.0% 100.0% 0.1 0.0% rocksdb::CompactionIterator::CompactionProxy::~CompactionProxy
0.0 0.0% 100.0% 0.0 0.0% rocksdb::CompactionJob::FinishCompactionOutputFile
0.0 0.0% 100.0% 0.0 0.0% rocksdb::CompactionJob::Install
0.0 0.0% 100.0% 0.0 0.0% rocksdb::CompactionJob::InstallCompactionResults
0.0 0.0% 100.0% 0.0 0.0% rocksdb::CompactionJob::ProcessKeyValueCompaction
0.0 0.0% 100.0% 0.1 0.0% rocksdb::CompactionJob::Run
0.0 0.0% 100.0% 16.0 0.5% rocksdb::ConcurrentArena::AllocateAligned
0.0 0.0% 100.0% 0.0 0.0% rocksdb::CreateFile
0.0 0.0% 100.0% 53.2 1.7% rocksdb::DB::Get
0.0 0.0% 100.0% 0.1 0.0% rocksdb::DBImpl::BGWorkCompaction
0.0 0.0% 100.0% 0.1 0.0% rocksdb::DBImpl::BackgroundCallCompaction
0.0 0.0% 100.0% 0.0 0.0% rocksdb::DBImpl::BackgroundCallFlush
0.0 0.0% 100.0% 0.1 0.0% rocksdb::DBImpl::BackgroundCompaction
0.0 0.0% 100.0% 0.0 0.0% rocksdb::DBImpl::BackgroundFlush
0.0 0.0% 100.0% 0.1 0.0% rocksdb::DBImpl::CompactRange
0.0 0.0% 100.0% 19.2 0.6% rocksdb::DBImpl::DumpStats
0.0 0.0% 100.0% 0.1 0.0% rocksdb::DBImpl::FlushMemTable
0.0 0.0% 100.0% 0.0 0.0% rocksdb::DBImpl::FlushMemTableToOutputFile
0.0 0.0% 100.0% 0.0 0.0% rocksdb::DBImpl::FlushMemTablesToOutputFiles
0.0 0.0% 100.0% 53.2 1.7% rocksdb::DBImpl::Get
0.0 0.0% 100.0% 53.2 1.7% rocksdb::DBImpl::GetImpl
0.0 0.0% 100.0% 0.0 0.0% rocksdb::DBImpl::InstallSuperVersionAndScheduleWork
0.0 0.0% 100.0% 0.0 0.0% rocksdb::DBImpl::MaybeScheduleFlushOrCompaction
0.0 0.0% 100.0% 0.0 0.0% rocksdb::DBImpl::RunManualCompaction
0.0 0.0% 100.0% 16.0 0.5% rocksdb::DBImpl::Write
0.0 0.0% 100.0% 16.0 0.5% rocksdb::DBImpl::WriteImpl
0.0 0.0% 100.0% 421.8 13.3% rocksdb::DBIter::FindNextUserEntryInternal
0.0 0.0% 100.0% 426.2 13.4% rocksdb::DBIter::Seek
0.0 0.0% 100.0% 0.0 0.0% rocksdb::FlushJob::Run
0.0 0.0% 100.0% 0.0 0.0% rocksdb::JobContext::JobContext
0.0 0.0% 100.0% 19.2 0.6% rocksdb::Log
0.0 0.0% 100.0% 0.0 0.0% rocksdb::LogFileName
0.0 0.0% 100.0% 0.0 0.0% rocksdb::MakeTableFileName
0.0 0.0% 100.0% 16.0 0.5% rocksdb::MemTable::Add
0.0 0.0% 100.0% 0.0 0.0% rocksdb::MemTable::MemTable
0.0 0.0% 100.0% 16.0 0.5% rocksdb::MemTableInserter::PutCF
0.0 0.0% 100.0% 16.0 0.5% rocksdb::MemTableInserter::PutCFImpl
0.0 0.0% 100.0% 0.0 0.0% rocksdb::MemTableList::TryInstallMemtableFlushResults
0.0 0.0% 100.0% 421.8 13.3% rocksdb::MergingIterator::Next
0.0 0.0% 100.0% 4.3 0.1% rocksdb::MergingIterator::Seek
0.0 0.0% 100.0% 0.0 0.0% rocksdb::NewWritableFile
0.0 0.0% 100.0% 0.0 0.0% rocksdb::ShardedCache::Insert
0.0 0.0% 100.0% 0.0 0.0% rocksdb::Slice::ToString
0.0 0.0% 100.0% 0.0 0.0% rocksdb::SstFileManagerImpl::OnAddFile
0.0 0.0% 100.0% 0.0 0.0% rocksdb::SstFileManagerImpl::OnAddFileImpl
0.0 0.0% 100.0% 0.1 0.0% rocksdb::TableCache::FindTable
0.0 0.0% 100.0% 53.2 1.7% rocksdb::TableCache::Get
0.0 0.0% 100.0% 0.1 0.0% rocksdb::TableCache::NewIterator
0.0 0.0% 100.0% 0.0 0.0% rocksdb::TableFileName
0.0 0.0% 100.0% 0.1 0.0% rocksdb::ThreadPoolImpl::Impl::BGThread
0.0 0.0% 100.0% 0.1 0.0% rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper
0.0 0.0% 100.0% 0.0 0.0% rocksdb::ThreadPoolImpl::Impl::Submit
0.0 0.0% 100.0% 0.0 0.0% rocksdb::ThreadPoolImpl::Schedule
0.0 0.0% 100.0% 53.2 1.7% rocksdb::Version::Get
0.0 0.0% 100.0% 0.0 0.0% rocksdb::Version::PrepareApply
0.0 0.0% 100.0% 423.8 13.3% rocksdb::Version::UpdateAccumulatedStats
0.0 0.0% 100.0% 0.0 0.0% rocksdb::Version::Version
0.0 0.0% 100.0% 0.0 0.0% rocksdb::VersionBuilder::SaveTo
0.0 0.0% 100.0% 0.0 0.0% rocksdb::VersionStorageInfo::CalculateBaseBytes
0.0 0.0% 100.0% 16.0 0.5% rocksdb::WriteBatch::Iterate
0.0 0.0% 100.0% 16.0 0.5% rocksdb::WriteBatchInternal::InsertInto
0.0 0.0% 100.0% 2.0 0.1% rocksdb_cache::ShardedCache::Insert
0.0 0.0% 100.0% 0.0 0.0% std::_Hashtable::_M_insert_unique_node
0.0 0.0% 100.0% 0.0 0.0% std::_Hashtable::_M_rehash
0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_Rb_tree
0.0 0.0% 100.0% 0.1 0.0% std::_Rb_tree::operator=
0.0 0.0% 100.0% 19.6 0.6% std::_V2::generic_category
0.0 0.0% 100.0% 0.0 0.0% std::__uninitialized_copy::__uninit_copy
0.0 0.0% 100.0% 0.0 0.0% std::getline
0.0 0.0% 100.0% 0.0 0.0% std::map::operator[]
0.0 0.0% 100.0% 0.0 0.0% std::string::_M_mutate
0.0 0.0% 100.0% 0.0 0.0% std::string::_M_replace_safe
0.0 0.0% 100.0% 0.1 0.0% std::string::_Rep::_M_clone
0.0 0.0% 100.0% 0.0 0.0% std::string::_S_construct
0.0 0.0% 100.0% 0.1 0.0% std::string::append
0.0 0.0% 100.0% 0.0 0.0% std::string::assign
0.0 0.0% 100.0% 0.0 0.0% std::string::replace
0.0 0.0% 100.0% 0.1 0.0% std::string::reserve
0.0 0.0% 100.0% 19.2 0.6% std::thread::_State_impl::_M_run
0.0 0.0% 100.0% 0.0 0.0% std::vector::emplace_back
0.0 0.0% 100.0% 0.0 0.0% void decode_decrypt_enc_bl@2cf9f0
0.0 0.0% 100.0% 0.0 0.0% void encode_encrypt_enc_bl@2cd1f0
0.0 0.0% 100.0% 0.0 0.0% void encode_encrypt_enc_bl@2cd910
0.0 0.0% 100.0% 0.0 0.0% void finish_contexts
Updated by Theofilos Mouratidis about 3 years ago
Theofilos Mouratidis wrote:
We have this bug on 2 of our clusters, we have to restart the mons once in a while so they don't go OOM
[...][...]
we use ceph version 14.2.9
Updated by Theofilos Mouratidis about 3 years ago
Aleksey Ivanov, I figured out there is a piece of code that leaks with a FIXME tag on the source code. Run 'ceph config dump | grep unknown' and remove any obsolete/wrong configs, then restart your mons.
Updated by Aleksey Ivanov about 3 years ago
Thanks for helping. I found obsolete/wrong configs.
mgr unknown mgr/dashboard/RGW_API_ACCESS_KEY 5PADD1SWZP7GWYW4FXJ7 * mgr unknown mgr/dashboard/RGW_API_SECRET_KEY Zrq4kEbMd.... * mgr unknown mgr/dashboard/password $2b$12$QBwyI..... * mgr unknown mgr/dashboard/server_addr * mgr unknown mgr/dashboard/server_port 19300 * mgr unknown mgr/dashboard/ssl true * mgr unknown mgr/dashboard/username ceph-admin *
But can not delete id.
[root@service01-qas ceph]# ceph config rm global mgr/dashboard/username
[root@service01-qas ceph]# ceph config dump | grep unknown | grep username
mgr unknown mgr/dashboard/username ceph-admin *
[root@service01-qas ceph]#
In log ceph-mon.service01-qas.log:
2021-02-19T08:58:34.909+0300 7fbeb62be700 1 mon.service01-qas@0(leader).osd e68673 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 364904448 full_alloc: 364904448 kv_alloc: 285212672
2021-02-19T08:58:38.282+0300 7fbeb3ab9700 0 mon.service01-qas@0(leader) e10 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "mgr/dashboard/username"} v 0) v1
2021-02-19T08:58:38.282+0300 7fbeb3ab9700 0 log_channel(audit) log [INF] : from='client.? 10.189.44.81:0/3071651694' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "mg
r/dashboard/username"}]: dispatch
2021-02-19T08:58:39.912+0300 7fbeb62be700 1 mon.service01-qas@0(leader).osd e68673 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 364904448 full_alloc: 364904448 kv_alloc: 285212672
2021-02-19T08:58:44.914+0300 7fbeb62be700 1 mon.service01-qas@0(leader).osd e68673 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 364904448 full_alloc: 364904448 kv_alloc: 285212672
MON restart on all nodes not helping. Can you help with this?Updated by Dan van der Ster about 3 years ago
You need to remove from the mgr scope not global, e.g. `ceph config rm mgr mgr/dashboard/username`
Updated by Kefu Chai about 3 years ago
- Subject changed from Memory leak ceph-mon in ConfigMonitor::load_config to octopus: Memory leak ceph-mon in ConfigMonitor::load_config
- Status changed from New to Fix Under Review
- Assignee set to Kefu Chai
- Pull request ID set to 40298
Updated by Kefu Chai about 3 years ago
- Copied to Backport #49914: octopus: Memory leak ceph-mon in ConfigMonitor::load_config added
Updated by Kefu Chai about 3 years ago
- Status changed from Fix Under Review to Resolved
i am marking this change "pending backport" as this issue has been fixed in master. but it exists in octopus and nautilus. let's track the backport in the backport tracker issues. i just created them.
Updated by Kefu Chai about 3 years ago
- Status changed from Resolved to Pending Backport
Updated by Kefu Chai about 3 years ago
- Pull request ID changed from 40298 to 39504
Updated by Kefu Chai about 3 years ago
- Copied to Backport #49915: nautilus: Memory leak ceph-mon in ConfigMonitor::load_config added
Updated by Loïc Dachary about 3 years ago
- Subject changed from octopus: Memory leak ceph-mon in ConfigMonitor::load_config to Memory leak ceph-mon in ConfigMonitor::load_config
Updated by Aleksey Ivanov about 3 years ago
Dan van der Ster wrote:
You need to remove from the mgr scope not global, e.g. `ceph config rm mgr mgr/dashboard/username`
Looks like it works! No memory leak now.
Thanks for helping!
Updated by Loïc Dachary about 3 years ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".
Actions