Ceph

The segfault only happens when enabling rados_tracing. Enabling all the others does not trigger the segfault.

I'm looking into this, but in the meantime I had a thought, if you'll forgive a little speculation. The short version is that it may be worthwhile to see if you can
upgrade tcmalloc and libunwind-- it looks like the versions being used in this report at least have some known bugs that aren't identical to what you're seeing but suspiciously similar...

Details:

In our bug report, just before the stack trace is gathered, we have:
32117 by 0xDD6D854: tcmalloc::PageHeap::GrowHeap(unsigned long) (in /usr/lib/libtcmalloc.so.4.1.2)
32117 by 0xDD6DB62: tcmalloc::PageHeap::New(unsigned long) (in /usr/lib/libtcmalloc.so.4.1.2)
32117 by 0xDD6C6D6: tcmalloc::CentralFreeList::Populate() (in /usr/lib/libtcmalloc.so.4.1.2)
32117 by 0xDD6C8A7: tcmalloc::CentralFreeList::FetchFromSpansSafe() (in /usr/lib/libtcmalloc.so.4.1.2)
32117 by 0xDD6C922: tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) (in /usr/lib/libtcmalloc.so.4.1.2)

...notice here there is a not-identical, but similar pattern (source: "https://github.com/gperftools/gperftools/issues/585"), especially the call FetchFromSpans() (vis-a-vis our
version's FetchFromSpansSafe()) after a call to RemoveRange():

#0 tcmalloc::CentralFreeList::FetchFromSpans (this=0x7f9898f1caa0)
at src/central_freelist.cc:298
#1 0x00007f9898cf1078 in tcmalloc::CentralFreeList::RemoveRange (
this=0x7f9898f1caa0, start=0x7f982d066480, end=0x7f982d066488, N=166)
at src/central_freelist.cc:269

...not identical, but suspiciously similar.

In that report, it sounds like the application code was patched to correct the bug.

This report blames libunwind:
https://github.com/gperftools/gperftools/issues/101

...which indeed appears further down in our stack trace. A little more digging found this interesting report from Seastar (a cool project that the async I/O core for ScyllaDB, BTW):
https://groups.google.com/forum/#!msg/seastar-dev/SiL4uUvvJAc/tsEsI87dAAAJ

...the tidbit of interest is:
"This manifests in a number of ways. One is that some operations appear
to hang due to worker thread completion notifications not being
delivered. Collectd statistics are not exported due to timers not
firing. It may also cause aborts on SIGTERM if it's delivered to
non-0th shard."

Other note:
I did find a Ceph report, but it has to do with performance rather than a segfault. For completeness: https://patchwork.kernel.org/patch/6107841/

Thanks for looking into the bug. I updated both tcmalloc and libunwind from yakkety sources and hit the same error, but with a small variation. Wonder if the packages need to be compiled for Ubuntu 14.04

Current packages:
vagrant@cephaio:/data/odisk/ceph/ceph/build/out$ sudo dpkg -l | grep tcmalloc
ii libtcmalloc-minimal4 2.4-0ubuntu5 amd64 efficient thread-caching malloc
vagrant@cephaio:/data/odisk/ceph/ceph/build/out$ sudo dpkg -l | grep google-perf
ii libgoogle-perftools-dev 2.4-0ubuntu5 amd64 libraries for CPU and heap analysis, plus an efficient thread-caching malloc
ii libgoogle-perftools4 2.4-0ubuntu5 amd64 libraries for CPU and heap analysis, plus an efficient thread-caching malloc
ii libgoogle-perftools4-dbg 2.4-0ubuntu5 amd64 libraries for CPU and heap analysis, plus an efficient thread-caching malloc
vagrant@cephaio:/data/odisk/ceph/ceph/build/out$ sudo dpkg -l | grep libgcc1
ii libgcc1:amd64 1:6.2.0-5ubuntu12 amd64 GCC support library
vagrant@cephaio:/data/odisk/ceph/ceph/build/out$ sudo dpkg -l | grep libunwind8
ii libunwind8 1.1-4.1ubuntu2 amd64 library to determine the call-chain of a program - runtime
ii libunwind8-dbg 1.1-4.1ubuntu2 amd64 library to determine the call-chain of a program - runtime
ii libunwind8-dev 1.1-4.1ubuntu2 amd64 library to determine the call-chain of a program - development

Logs:
2017-01-23 21:15:31.097217 7f526c2bc700 0 -- - >> 10.0.2.15:40000/0 pipe(0x7f5270094160 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7f527009c160).fault
--19333-- WARNING: Serious error when reading debug info
--19333-- When reading debug info from /usr/lib/libtcmalloc.so.4.2.6:
--19333-- Ignoring non-Dwarf2/3/4 block in .debug_info
--19333-- WARNING: Serious error when reading debug info
--19333-- When reading debug info from /usr/lib/libtcmalloc.so.4.2.6:
--19333-- Last block truncated in .debug_info; ignoring
--19333-- WARNING: Serious error when reading debug info
--19333-- When reading debug info from /usr/lib/libtcmalloc.so.4.2.6:
--19333-- parse_CU_Header: is neither DWARF2 nor DWARF3 nor DWARF4
--19333-- WARNING: Serious error when reading debug info
--19333-- When reading debug info from /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1:
--19333-- Ignoring non-Dwarf2/3/4 block in .debug_info
--19333-- WARNING: Serious error when reading debug info
--19333-- When reading debug info from /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1:
--19333-- Last block truncated in .debug_info; ignoring
--19333-- WARNING: Serious error when reading debug info
--19333-- When reading debug info from /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1:
--19333-- parse_CU_Header: is neither DWARF2 nor DWARF3 nor DWARF4
19333 Syscall param msync(start) points to uninitialised byte(s)
19333 at 0xDFCCB20: __msync_nocancel (syscall-template.S:81)
19333 by 0x10057266: access_mem (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0x1005AD03: apply_reg_state (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0x1005B24E: _ULx86_64_dwarf_find_save_locs (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0x1005B5C8: _ULx86_64_dwarf_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0x10057C70: _ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0xDD7C4DA: GetStackTrace_libunwind(void**, int, int) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD7CCED: GetStackTrace(void**, int, int) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD6E0BF: tcmalloc::PageHeap::GrowHeap(unsigned long) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD6E422: tcmalloc::PageHeap::New(unsigned long) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD6CD33: tcmalloc::CentralFreeList::Populate() (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD6CF27: tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 Address 0xfff000020 is on thread 1's stack
19333 in frame #7, created by GetStackTrace(void**, int, int) (?)
19333
2017-01-23 21:15:34.096121 7f526c1bb700 0 -- - >> 10.0.2.15:40000/0 pipe(0x7f5244000d90 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5244002090).fault
2017-01-23 21:15:34.120741 403e640 -1 WARNING: all dangerous and experimental features are enabled.
2017-01-23 21:15:34.242587 403e640 -1 WARNING: all dangerous and experimental features are enabled.
19333 Syscall param msync(start) points to uninitialised byte(s)
19333 at 0xDFCCB3D: ? (syscall-template.S:81)
19333 by 0x10057266: access_mem (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0x1005AD03: apply_reg_state (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0x1005B24E: _ULx86_64_dwarf_find_save_locs (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0x1005B5C8: _ULx86_64_dwarf_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0x10057C70: _ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0xDD7C4DA: GetStackTrace_libunwind(void**, int, int) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD7CCED: GetStackTrace(void**, int, int) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD6E0BF: tcmalloc::PageHeap::GrowHeap(unsigned long) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD6E422: tcmalloc::PageHeap::New(unsigned long) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD6CD33: tcmalloc::CentralFreeList::Populate() (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD6CF27: tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 Address 0xffeffe000 is on thread 1's stack
19333 in frame #6, created by GetStackTrace_libunwind(void**, int, int) (?)
19333
2017-01-23 21:15:34.869583 403e640 -1 WARNING: all dangerous and experimental features are enabled.
starting mon.a rank 0 at 10.0.2.15:40000/0 mon_data /data/ceph-disk//mon.a fsid a1433fb9-974b-4ca8-8081-d21b24103b30
19333 Thread 8 ms_dispatch:
19333 Syscall param msync(start) points to unaddressable byte(s)
19333 at 0xDFCCB3D: ? (syscall-template.S:81)
19333 by 0x10057266: access_mem (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0x1005AD03: apply_reg_state (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0x1005B24E: _ULx86_64_dwarf_find_save_locs (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0x1005B5C8: _ULx86_64_dwarf_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0x10057C70: _ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
19333 by 0xDD7C4DA: GetStackTrace_libunwind(void**, int, int) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD7CCED: GetStackTrace(void**, int, int) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD6E0BF: tcmalloc::PageHeap::GrowHeap(unsigned long) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD6E422: tcmalloc::PageHeap::New(unsigned long) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD6CD33: tcmalloc::CentralFreeList::Populate() (in /usr/lib/libtcmalloc.so.4.2.6)
19333 by 0xDD6CF27: tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) (in /usr/lib/libtcmalloc.so.4.2.6)
19333 Address 0x15df2088 is on thread 8's stack
19333 136 bytes below stack pointer
19333
0
Segmentation fault (core dumped)

I added -x to vstart and I am getting the below as logs.

2017-01-23 23:41:41.336351 403e640 -1 WARNING: all dangerous and experimental features are enabled.
starting mon.a rank 0 at 10.0.2.15:40000/0 mon_data /data/ceph-disk//mon.a fsid a1433fb9-974b-4ca8-8081-d21b24103b30
21867 Thread 8 ms_dispatch:
21867 Syscall param msync(start) points to unaddressable byte(s)
21867 at 0xDFCCB3D: ??? (syscall-template.S:81)
21867 by 0x10057266: access_mem (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
21867 by 0x1005AD03: apply_reg_state (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
21867 by 0x1005B24E: _ULx86_64_dwarf_find_save_locs (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
21867 by 0x1005B5C8: _ULx86_64_dwarf_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
21867 by 0x10057C70: _ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
21867 by 0xDD7C4DA: GetStackTrace_libunwind(void**, int, int) (in /usr/lib/libtcmalloc.so.4.2.6)
21867 by 0xDD7CCED: GetStackTrace(void**, int, int) (in /usr/lib/libtcmalloc.so.4.2.6)
21867 by 0xDD6E0BF: tcmalloc::PageHeap::GrowHeap(unsigned long) (in /usr/lib/libtcmalloc.so.4.2.6)
21867 by 0xDD6E422: tcmalloc::PageHeap::New(unsigned long) (in /usr/lib/libtcmalloc.so.4.2.6)
21867 by 0xDD6CD33: tcmalloc::CentralFreeList::Populate() (in /usr/lib/libtcmalloc.so.4.2.6)
21867 by 0xDD6CF27: tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) (in /usr/lib/libtcmalloc.so.4.2.6)
21867 Address 0x15df2088 is on thread 8's stack
21867 136 bytes below stack pointer
21867
0
+ ceph_adm osd crush add osd.0 1.0 host=cephaio root=default
+ [ 1 -eq 1 ]
+ prun /data/odisk/ceph/ceph/build/bin/ceph -c /data/odisk/ceph/ceph/build/ceph.conf -k /data/odisk/ceph/ceph/build/keyring osd crush add osd.0 1.0 host=cephaio root=default
+ echo /data/odisk/ceph/ceph/build/bin/ceph -c /data/odisk/ceph/ceph/build/ceph.conf -k /data/odisk/ceph/ceph/build/keyring osd crush add osd.0 1.0 host=cephaio root=default
/data/odisk/ceph/ceph/build/bin/ceph -c /data/odisk/ceph/ceph/build/ceph.conf -k /data/odisk/ceph/ceph/build/keyring osd crush add osd.0 1.0 host=cephaio root=default
+ /data/odisk/ceph/ceph/build/bin/ceph -c /data/odisk/ceph/ceph/build/ceph.conf -k /data/odisk/ceph/ceph/build/keyring osd crush add osd.0 1.0 host=cephaio root=default
2017-01-23 23:41:45.007479 7fe752d4f700 -1 WARNING: all dangerous and experimental features are enabled.
2017-01-23 23:41:45.072005 7fe752d4f700 -1 WARNING: all dangerous and experimental features are enabled.
add item id 0 name 'osd.0' weight 1 at location {host=cephaio,root=default} to crush map
Segmentation fault (core dumped)

Wonder if there is something else causing the segfault.

Sorry for the barrage of updates. Adding more logs on what i found when i ran 'ceph osd ls' via gdb.

• 4 Thread 0x7fffe8b43700 (LWP 23446) "python" uw_frame_state_for (
  context=context@entry=0x7fffe8b41ba0, fs=fs@entry=0x7fffe8b418f0)
  at ../../../src/libgcc/unwind-dw2.c:1249
  3 Thread 0x7fffe9a7a700 (LWP 23445) "log" pthread_cond_wait@@GLIBC_2.3.2 ()
  at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  1 Thread 0x7ffff7fe6740 (LWP 23434) "python" 0x00007ffff78e8c53 in select ()
  at ../sysdeps/unix/syscall-template.S:81
  (gdb)