Project

General

Profile

Actions
Copy link

Bug #3463

closed

Crash when adding monitor

Added by Jeff Mitchell over 11 years ago. Updated over 11 years ago.

Status:
Rejected
Priority:
High
Assignee:
-
Category:
Monitor
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I have three monitors now and wanted to add two more for extra reliability. I followed the instructions here http://ceph.com/docs/master/cluster-ops/add-or-rm-mons/ without incident, but when I launch I get a dump:

  1. /usr/bin/ceph-mon -i d --pid-file /var/run/ceph/mon.d.pid -c /etc/ceph/ceph.conf
    terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
    what(): buffer::end_of_buffer
    • Caught signal (Aborted)
      in thread 7f426f022780
      ceph version 0.53 (2528b5ee105b16352c91af064af5c0b5a7d45d7c)
      1: /usr/bin/ceph-mon() [0x548082]
      2: (()+0xf030) [0x7f426ebff030]
      3: (gsignal()+0x35) [0x7f426d4ea475]
      4: (abort()+0x180) [0x7f426d4ed6f0]
      5: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f426dd456dd]
      6: (()+0x637e6) [0x7f426dd437e6]
      7: (()+0x63813) [0x7f426dd43813]
      8: (()+0x63a3e) [0x7f426dd43a3e]
      9: /usr/bin/ceph-mon() [0x5f484f]
      10: (decode(ceph::buffer::list&, ceph::buffer::list::iterator&)+0xb9) [0x483e59]
      11: (main()+0x12fb) [0x4817fb]
      12: (__libc_start_main()+0xfd) [0x7f426d4d6ead]
      13: /usr/bin/ceph-mon() [0x483519]
      2012-11-08 13:23:49.893132 7f426f022780 -1       Caught signal (Aborted) *
      in thread 7f426f022780

      ceph version 0.53 (2528b5ee105b16352c91af064af5c0b5a7d45d7c)
      1: /usr/bin/ceph-mon() [0x548082]
      2: (()+0xf030) [0x7f426ebff030]
      3: (gsignal()+0x35) [0x7f426d4ea475]
      4: (abort()+0x180) [0x7f426d4ed6f0]
      5: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f426dd456dd]
      6: (()+0x637e6) [0x7f426dd437e6]
      7: (()+0x63813) [0x7f426dd43813]
      8: (()+0x63a3e) [0x7f426dd43a3e]
      9: /usr/bin/ceph-mon() [0x5f484f]
      10: (decode(ceph::buffer::list&, ceph::buffer::list::iterator&)+0xb9) [0x483e59]
      11: (main()+0x12fb) [0x4817fb]
      12: (__libc_start_main()+0xfd) [0x7f426d4d6ead]
      13: /usr/bin/ceph-mon() [0x483519]
      NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

      0> 2012-11-08 13:23:49.893132 7f426f022780 -1 ** Caught signal (Aborted) *
       in thread 7f426f022780

      ceph version 0.53 (2528b5ee105b16352c91af064af5c0b5a7d45d7c)
      1: /usr/bin/ceph-mon() [0x548082]
      2: (()+0xf030) [0x7f426ebff030]
      3: (gsignal()+0x35) [0x7f426d4ea475]
      4: (abort()+0x180) [0x7f426d4ed6f0]
      5: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f426dd456dd]
      6: (()+0x637e6) [0x7f426dd437e6]
      7: (()+0x63813) [0x7f426dd43813]
      8: (()+0x63a3e) [0x7f426dd43a3e]
      9: /usr/bin/ceph-mon() [0x5f484f]
      10: (decode(ceph::buffer::list&, ceph::buffer::list::iterator&)+0xb9) [0x483e59]
      11: (main()+0x12fb) [0x4817fb]
      12: (__libc_start_main()+0xfd) [0x7f426d4d6ead]
      13: /usr/bin/ceph-mon() [0x483519]
      NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Aborted

Running on Debian Wheezy; Ceph from the official debian-testing Ceph repo.

Actions
Copy link
 #1

Updated by Jeff Mitchell over 11 years ago

Backtrace:

Core was generated by `/usr/bin/ceph-mon -i d --pid-file /var/run/ceph/mon.d.pid -c /etc/ceph/ceph.con'.
Program terminated with signal 6, Aborted.
#0 0x00007fd9940c6efb in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt
#0 0x00007fd9940c6efb in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x0000000000548129 in reraise_fatal (signum=6) at global/signal_handler.cc:58
#2 handle_fatal_signal (signum=6) at global/signal_handler.cc:104
#3 <signal handler called>
#4 0x00007fd9929b2475 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007fd9929b56f0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007fd99320d6dd in _gnu_cxx::_verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007fd99320b7e6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007fd99320b813 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007fd99320ba3e in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00000000005f484f in ceph::buffer::list::iterator::copy (this=0x7fff4ec6e000, len=2704745901, dest=...) at common/buffer.cc:546
#11 0x0000000000483e59 in decode (s=..., p=...) at ./include/encoding.h:256
#12 0x00000000004817fb in main (argc=<optimized out>, argv=0x7fff4ec6e908) at ceph_mon.cc:292

Actions
Copy link
 #2

Updated by Jeff Mitchell over 11 years ago

It turns out that the issue was the documentation leading me to believe that I had to populate the kerying and monmap in the places in the mon dir (created in step 1), when in fact that's bad news. Getting the files in a totally different location fixed it.

Actions
Copy link
 #3

Updated by Joao Eduardo Luis over 11 years ago

  • Status changed from New to Rejected
Actions
Copy link

Also available in: Atom PDF