Project

General

Profile

Bug #13208

small probability sigabrt when setting rados_osd_op_timeout

Added by Ruifeng Yang about 7 years ago. Updated almost 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
librados
Target version:
-
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
hammer
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ceph -v
ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)

uname -a
Linux cvk120 3.13.6 #1 SMP Sat Apr 25 14:42:46 CST 2015 x86_64 x86_64 x86_64 GNU/Linux

g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.6/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.3-1ubuntu5' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.6 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --disable-werror --with-arch-32=i686 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ff9affbf700 (LWP 8315)]
0x00007ffa378bf0d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffa378bf0d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffa378c283b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffa31b6f69d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffa31b6d846 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffa31b6d873 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffa31b6d96e in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffa35242509 in ceph::__ceph_assert_fail (assertion=<optimized out>, file=<optimized out>, line=146, func=0x7ffa354e2ae0 "void SafeTimer::add_event_at(utime_t, Context*)")
    at common/assert.cc:77
#7  0x00007ffa3523e56d in SafeTimer::add_event_at (this=0x21abe38, when=..., callback=0x0) at common/Timer.cc:146
#8  0x00007ffa3523e623 in SafeTimer::add_event_after (this=0x21abe38, seconds=120, callback=0x0) at common/Timer.cc:131
#9  0x00007ffa351c275a in Objecter::_op_submit_with_budget (this=0x21abcc0, op=0x7ff6940064f0, lc=..., ctx_budget=<optimized out>) at osdc/Objecter.cc:2017
#10 0x00007ffa351c2938 in Objecter::op_submit (this=0x21abcc0, op=0x7ff6940064f0, ctx_budget=0x0) at osdc/Objecter.cc:1984
#11 0x00007ffa35196588 in librados::IoCtxImpl::aio_operate_read (this=0x21bbd40, oid=..., o=0x7ff6940013e0, c=0x7ff694001250, flags=0, pbl=0x0) at librados/IoCtxImpl.cc:582
#12 0x00007ffa3516b409 in librados::IoCtx::aio_operate (this=0x21bb458, oid=..., c=0x7ff6940013c0, o=0x7ff9affbe8d0, flags=0, pbl=<optimized out>) at librados/librados.cc:1373
#13 0x00007ffa32b8edf8 in librbd::AioRead::send (this=0x7ff694001100) at librbd/AioRequest.cc:257
#14 0x00007ffa32bc64b2 in librbd::aio_read (ictx=0x21bb2e0, image_extents=..., buf=0x7ff9affbead8 "", pbl=0x0, c=0x7ff694006970, op_flags=0) at librbd/internal.cc:3639
#15 0x00007ffa32bc73c1 in librbd::read (ictx=0x21bb2e0, image_extents=..., buf=0x231e000 '!' <repeats 200 times>..., pbl=0x0, op_flags=0) at librbd/internal.cc:3017
#16 0x00007ffa32bc75f2 in librbd::read (ictx=0x21bb2e0, ofs=<optimized out>, len=<optimized out>, buf=0x231e000 '!' <repeats 200 times>..., op_flags=0) at librbd/internal.cc:3004
#17 0x00007ffa32b8b9ec in rbd_read (image=<optimized out>, ofs=<optimized out>, len=<optimized out>, buf=<optimized out>) at librbd/librbd.cc:1633
#18 0x00007ffa37685662 in ?? () from /usr/lib/tgt/backing-store/bs_rbd.so
#19 0x000000000042c512 in ?? ()
#20 0x00007ffa37e53e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007ffa3797d2ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
#22 0x0000000000000000 in ?? ()
(gdb) frame 9
#9  0x00007ffa351c275a in Objecter::_op_submit_with_budget (this=0x21abcc0, op=0x7ff6940064f0, lc=..., ctx_budget=<optimized out>) at osdc/Objecter.cc:2017
2017    osdc/Objecter.cc: No such file or directory.
(gdb) p cb
$1 = (C_CancelOp *) 0x7ff694006770
(gdb) p op->ontimeout
$2 = (Context *) 0x0
(gdb)

Related issues

Copied to Ceph - Backport #13340: small probability sigabrt when setting rados_osd_op_timeout Resolved

Associated revisions

Revision 0635b135 (diff)
Added by Ruifeng Yang about 7 years ago

Objecter: maybe access wild pointer(op) in _op_submit_with_budget.

look at "after giving up session lock it can be freed at any time by response handler" in _op_submit,
so the _op_submit_with_budget::op maybe is wild after call _op_submit.

Fixes: #13208
Signed-off-by: Ruifeng Yang <>

Revision f1d8a8f5 (diff)
Added by Ruifeng Yang about 7 years ago

Objecter: repeated free op->ontimeout.

repeated free op->ontimeout in SafeTimer::timer_thread::callback->complete

Fixes: #13208
Signed-off-by: Ruifeng Yang <>

Revision 84068f8c (diff)
Added by Ruifeng Yang about 7 years ago

Objecter: repeated free op->ontimeout.

repeated free op->ontimeout in SafeTimer::timer_thread::callback->complete

Fixes: #13208
Signed-off-by: Ruifeng Yang <>
(cherry picked from commit f1d8a8f577cee6d66f4dcffac667675f18145ebb)

Revision 394fbfcc (diff)
Added by Ruifeng Yang about 7 years ago

Objecter: maybe access wild pointer(op) in _op_submit_with_budget.

look at "after giving up session lock it can be freed at any time by response handler" in _op_submit,
so the _op_submit_with_budget::op maybe is wild after call _op_submit.

Fixes: #13208
Signed-off-by: Ruifeng Yang <>
(cherry picked from commit 0635b1358354b19ae44105576f730381f3b5b963)

History

#2 Updated by Kefu Chai about 7 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Ruifeng Yang

#3 Updated by Sage Weil about 7 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Source changed from other to Community (dev)
  • Backport set to hammer

#4 Updated by Loïc Dachary about 7 years ago

  • Description updated (diff)

#5 Updated by Loïc Dachary almost 7 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF