Bug #9362
closed
librados, rados_read corrupts memory on timeout
Added by Matthias Kiefer over 9 years ago.
Updated over 9 years ago.
Description
If you configure librados with rados_osd_op_timeout, timeouts on rados_read will result in memory corruptions and segmentation faults.
If rados_read returns with a timeout (-110) the output buffer provided to rados_read is accessed after rados_read returned. This either results in the given buffer changing content after the call or, if the buffer is freed immediately, in a memory corruption.
During testing the timeout feature in a multithreading environment we experienced many crashes of the processes using librados and even worse had corrupted objects being written to ceph. These corrupted objects contained content of an object where rados_read resulted in a timeout at exactly the same time where the corrupted object was written to ceph.
Attached you can find a small program to reproduce the problem.
The problem was reproduced with librados 0.67.10 and 0.80.5
Files
I have also tried to reproduce this problem with a firewall dropping incoming packages from the primary osd the object is located on. In this case rados_read also returns the timeout error, but nothing is written into the buffer and no crash happens.
- Priority changed from High to Urgent
which version of librados is this? Thanks!
Sage pointed out elsewhere (and I'm with him) that it looks like the actual response is coming in and then the messenger is writing into the given data buffers. We'll need to clear them out.
I pushed the minimal patch to wip-dumpling-9362. (Obviously, it's based on the dumpling branch, which will eventually be .67.11.) It has absolutely no testing, but if you can test it (just on the client; this won't require any changes to the OSDs) and let us know...
- Status changed from New to Fix Under Review
- Source changed from other to Community (dev)
- Assignee set to Sage Weil
Thanks for your investigations! I have checked your patch and it indeed seem to solve the problem with the memory access after the call returns. However, while executing the attached radosreadtimeout.cpp many times, I got the following two errors:
- Failed assertion:
osdc/Objecter.cc: In function 'void Objecter::handle_osd_op_reply(MOSDOpReply*)' thread 7f6d487f4700 time 2014-09-09 08:53:30.384945
osdc/Objecter.cc: 2453: FAILED assert(initialized.read())
ceph version 0.85-684-g74f22fd (74f22fdf4f98f18c2632f213e2a3baeba6576d27)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f6d509e56ab]
2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x1a2c) [0x7f6d5096a1fc]
3: (Objecter::ms_dispatch(Message*)+0x283) [0x7f6d50973d43]
4: (DispatchQueue::fast_dispatch(Message*)+0x56) [0x7f6d50af1196]
5: (Pipe::reader()+0x1a54) [0x7f6d50b56f84]
6: (Pipe::Reader::entry()+0xd) [0x7f6d50b60b9d]
7: (()+0x8182) [0x7f6d4fa85182]
8: (clone()+0x6d) [0x7f6d501affbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion'
- Segmentation Fault: (stack trace from core dump (attached))
Program terminated with signal SIGSEGV, Segmentation fault.
#0 pthread_rwlock_wrlock () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S:41
41 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S: No such file or directory.
(gdb) where
#0 pthread_rwlock_wrlock () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S:41
#1 0x00007fd6aa5b02cf in RWLock::get_write (this=0x18, lockdep=<optimized out>) at ./common/RWLock.h:88
#2 0x00007fd6aa5964f0 in Objecter::op_cancel (this=0xa1dd10, s=0x0, tid=20535, r=-110)
at osdc/Objecter.cc:1851
#3 0x00007fd6aa55ff29 in Context::complete (this=0xa1b820, r=<optimized out>) at ./include/Context.h:64
#4 0x00007fd6aa61c65f in RWTimer::timer_thread (this=0xa1de30) at common/Timer.cc:268
#5 0x00007fd6aa61db2d in RWTimerThread::entry (this=<optimized out>) at common/Timer.cc:200
#6 0x00007fd6a96bf182 in start_thread (arg=0x7fd6a4e35700) at pthread_create.c:312
#7 0x00007fd6a9de9fbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thanks, Matthias. Those look like unrelated problems with the master branch.
- Status changed from Fix Under Review to Pending Backport
- Status changed from Pending Backport to Resolved
Also available in: Atom
PDF