Bug #882: misc leaks in librados - Messengers - Ceph

Custom queries

Bug queue
Bug triage
Crash queue
Crash triage
Feedback
My issues
Need Review
Pending backports
Product Backlog Scrub

Actions

Copy link

Bug #882

closed

misc leaks in librados

Added by Josh Durgin about 13 years ago. Updated about 5 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Josh Durgin

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Running valgrind --leak-check=yes .libs/lt-testlibrbd reveals a few leaks in the messenger:

==1310== 80 bytes in 1 blocks are definitely lost in loss record 15 of 31
==1310==    at 0x4C24DFA: operator new(unsigned long) (vg_replace_malloc.c:261)
==1310==    by 0x5148E99: MPoolOpReply::decode_payload() (MPoolOpReply.h:69)
==1310==    by 0x511E927: decode_message(ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&) (Message.cc:526)
==1310==    by 0x50CD791: SimpleMessenger::Pipe::read_message(Message**) (SimpleMessenger.cc:1952)
==1310==    by 0x50D9562: SimpleMessenger::Pipe::reader() (SimpleMessenger.cc:1574)
==1310==    by 0x50BA4DC: SimpleMessenger::Pipe::Reader::entry() (SimpleMessenger.h:196)
==1310==    by 0x595D8B9: start_thread (pthread_create.c:300)
==1310==    by 0x54C002C: clone (clone.S:112)
==1310== 2,892 (192 direct, 2,700 indirect) bytes in 6 blocks are definitely lost in loss record 30 of 31
==1310==    at 0x4C24DFA: operator new(unsigned long) (vg_replace_malloc.c:261)
==1310==    by 0x50CEA14: SimpleMessenger::Pipe::read_message(Message**) (new_allocator.h:89)
==1310==    by 0x50D9562: SimpleMessenger::Pipe::reader() (SimpleMessenger.cc:1574)
==1310==    by 0x50BA4DC: SimpleMessenger::Pipe::Reader::entry() (SimpleMessenger.h:196)
==1310==    by 0x595D8B9: start_thread (pthread_create.c:300)
==1310==    by 0x54C002C: clone (clone.S:112)

Full log is attached.

Files

valgrind.log (14.2 KB) valgrind.log

Josh Durgin, 03/11/2011 04:01 PM

History
Notes
Property changes

Actions

Copy link

Updated by Josh Durgin about 13 years ago

Subject changed from memory leaks in initialization/configuration to memory leaks in messenger
Category set to msgr
Priority changed from Normal to High

Repurposing duplicate for the messenger leaks.

Actions

Copy link

Updated by Josh Durgin about 13 years ago

Priority changed from High to Normal

The only remaining leak whose backtrace came from the messenger is:

==3861== 6,748 (448 direct, 6,300 indirect) bytes in 14 blocks are definitely lost in loss record 18 of 19
==3861==    at 0x4C24DFA: operator new(unsigned long) (vg_replace_malloc.c:261)
==3861==    by 0x4E548D8: __gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >::allocate(unsigned long, void const*) (new_allocator.h:89)
==3861==    by 0x4E53035: std::_List_base<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> >::_M_get_node() (stl_list.h:316)
==3861==    by 0x4E50FD6: std::list<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> >::_M_create_node(ceph::buffer::ptr const&) (stl_list.h:461)
==3861==    by 0x4E50EA0: std::list<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> >::_M_insert(std::_List_iterator<ceph::buffer::ptr>, ceph::buffer::ptr const&) (stl_list.h:1407)
==3861==    by 0x4E4FBC5: std::list<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> >::push_back(ceph::buffer::ptr const&) (stl_list.h:920)
==3861==    by 0x4E4D986: ceph::buffer::list::push_back(ceph::buffer::ptr const&) (buffer.h:804)
==3861==    by 0x52BF625: SimpleMessenger::Pipe::read_message(Message**) (SimpleMessenger.cc:1865)
==3861==    by 0x52BDA2B: SimpleMessenger::Pipe::reader() (SimpleMessenger.cc:1574)
==3861==    by 0x528A575: SimpleMessenger::Pipe::Reader::entry() (SimpleMessenger.h:196)
==3861==    by 0x52C6CE8: Thread::_entry_func(void*) (Thread.h:41)
==3861==    by 0x5C408B9: start_thread (pthread_create.c:300)

This doesn't increase with more I/O operations, so I'm guessing it's not too serious.

Actions

Copy link

Updated by Josh Durgin about 13 years ago

The remaining messenger-related leak with --show-reachable=yes:


</==27799== 304 bytes in 1 blocks are indirectly lost in loss record 9 of 13
==27799==    at 0x4C24DFA: operator new(unsigned long) (vg_replace_malloc.c:261)
==27799==    by 0x528BC87: SimpleMessenger::Pipe::Pipe(SimpleMessenger*, int) (SimpleMessenger.h:221)
==27799==    by 0x52C4559: SimpleMessenger::connect_rank(entity_addr_t const&, int) (SimpleMessenger.cc:2474)
==27799==    by 0x52C4EC9: SimpleMessenger::get_connection(entity_inst_t const&) (SimpleMessenger.cc:2584)
==27799==    by 0x52D66BD: Objecter::get_session(int) (Objecter.cc:340)
==27799==    by 0x52D8C09: Objecter::recalc_op_target(Objecter::Op*) (Objecter.cc:593)
==27799==    by 0x52D79A7: Objecter::op_submit(Objecter::Op*, Objecter::OSDSession*) (Objecter.cc:490)
==27799==    by 0x528F11F: Objecter::read(object_t const&, object_locator_t const&, unsigned long, unsigned long, snapid_t, ceph::buffer::list*, int, Context*, eversion_t*, ObjectOperation*) (Objecter.h:674)
==27799==    by 0x527E79D: librados::RadosClient::read(librados::IoCtxImpl&, object_t const&, ceph::buffer::list&, unsigned long, unsigned long) (librados.cc:1468)
==27799==    by 0x5281CD2: librados::IoCtx::read(std::string const&, ceph::buffer::list&, unsigned long, unsigned long) (librados.cc:2151)
==27799==    by 0x4E44540: librbd::list(librados::IoCtx&, std::vector<std::string, std::allocator<std::string> >&) (librbd.cc:520)
==27799==    by 0x4E4A784: rbd_list (librbd.cc:1519)
==27799== 
==27799== 364 bytes in 14 blocks are indirectly lost in loss record 10 of 13
==27799==    at 0x4C24A72: operator new[](unsigned long) (vg_replace_malloc.c:305)
==27799==    by 0x4E4C026: ceph::buffer::raw_char::raw_char(unsigned int) (buffer.h:174)
==27799==    by 0x4E4C512: ceph::buffer::create(unsigned int) (buffer.h:313)
==27799==    by 0x52C0A08: SimpleMessenger::Pipe::read_message(Message**) (SimpleMessenger.cc:1862)
==27799==    by 0x52BEE8B: SimpleMessenger::Pipe::reader() (SimpleMessenger.cc:1574)
==27799==    by 0x528B9D3: SimpleMessenger::Pipe::Reader::entry() (SimpleMessenger.h:196)
==27799==    by 0x52C8148: Thread::_entry_func(void*) (Thread.h:41)
==27799==    by 0x5C428B9: start_thread (pthread_create.c:300)
==27799==    by 0x57A502C: clone (clone.S:112)
==27799== 
==27799== 448 bytes in 14 blocks are indirectly lost in loss record 11 of 13
==27799==    at 0x4C24DFA: operator new(unsigned long) (vg_replace_malloc.c:261)
==27799==    by 0x4E4C4FF: ceph::buffer::create(unsigned int) (buffer.h:313)
==27799==    by 0x52C0A08: SimpleMessenger::Pipe::read_message(Message**) (SimpleMessenger.cc:1862)
==27799==    by 0x52BEE8B: SimpleMessenger::Pipe::reader() (SimpleMessenger.cc:1574)
==27799==    by 0x528B9D3: SimpleMessenger::Pipe::Reader::entry() (SimpleMessenger.h:196)
==27799==    by 0x52C8148: Thread::_entry_func(void*) (Thread.h:41)
==27799==    by 0x5C428B9: start_thread (pthread_create.c:300)
==27799==    by 0x57A502C: clone (clone.S:112)
==27799== 
==27799== 5,488 bytes in 14 blocks are indirectly lost in loss record 12 of 13
==27799==    at 0x4C24DFA: operator new(unsigned long) (vg_replace_malloc.c:261)
==27799==    by 0x5333E55: decode_message(ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&) (Message.cc:293)
==27799==    by 0x52C181F: SimpleMessenger::Pipe::read_message(Message**) (SimpleMessenger.cc:1952)
==27799==    by 0x52BEE8B: SimpleMessenger::Pipe::reader() (SimpleMessenger.cc:1574)
==27799==    by 0x528B9D3: SimpleMessenger::Pipe::Reader::entry() (SimpleMessenger.h:196)
==27799==    by 0x52C8148: Thread::_entry_func(void*) (Thread.h:41)
==27799==    by 0x5C428B9: start_thread (pthread_create.c:300)
==27799==    by 0x57A502C: clone (clone.S:112)
==27799== 
==27799== 7,052 (448 direct, 6,604 indirect) bytes in 14 blocks are definitely lost in loss record 13 of 13
==27799==    at 0x4C24DFA: operator new(unsigned long) (vg_replace_malloc.c:261)
==27799==    by 0x4E545E4: __gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >::allocate(unsigned long, void const*) (new_allocator.h:89)
==27799==    by 0x4E52D41: std::_List_base<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> >::_M_get_node() (stl_list.h:316)
==27799==    by 0x4E50CE2: std::list<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> >::_M_create_node(ceph::buffer::ptr const&) (stl_list.h:461)
==27799==    by 0x4E50BAC: std::list<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> >::_M_insert(std::_List_iterator<ceph::buffer::ptr>, ceph::buffer::ptr const&) (stl_list.h:1407)
==27799==    by 0x4E4F8D1: std::list<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> >::push_back(ceph::buffer::ptr const&) (stl_list.h:920)
==27799==    by 0x4E4D862: ceph::buffer::list::push_back(ceph::buffer::ptr const&) (buffer.h:804)
==27799==    by 0x52C0A85: SimpleMessenger::Pipe::read_message(Message**) (SimpleMessenger.cc:1865)
==27799==    by 0x52BEE8B: SimpleMessenger::Pipe::reader() (SimpleMessenger.cc:1574)
==27799==    by 0x528B9D3: SimpleMessenger::Pipe::Reader::entry() (SimpleMessenger.h:196)
==27799==    by 0x52C8148: Thread::_entry_func(void*) (Thread.h:41)
==27799==    by 0x5C428B9: start_thread (pthread_create.c:300)

Actions

Copy link

Updated by Greg Farnum about 13 years ago

I thought I saw something related to and QEMU on the lists -- am I making that up, or have we identified the source of these leaks elsewhere?

Actions

Copy link

Updated by Josh Durgin about 13 years ago

Assignee set to Josh Durgin

It looks like the remaining leaks are caused by Rados::notify. They show up in testradospp, but removing the notify calls removes the leaks coming from the messenger.

Actions

Copy link

Updated by Sage Weil about 13 years ago

Target version set to v0.27
Translation missing: en.field_position set to 324

Actions

Copy link

Updated by Sage Weil about 13 years ago

Subject changed from memory leaks in messenger to misc leaks in librados
Status changed from New to In Progress
Translation missing: en.field_story_points set to 3
Translation missing: en.field_position deleted (~~328~~)
Translation missing: en.field_position set to 328