Bug #3460: ceph monitor crashed in the nightlies - Ceph - Ceph

Actions

Copy link

Bug #3460

closed

ceph monitor crashed in the nightlies

Added by Tamilarasi muthamizhan over 11 years ago. Updated over 11 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Sage Weil

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Log: ubuntu@teuthology:/a/teuthology-2012-11-06_02:00:02-regression-testing-master-basic/10579

2012-11-07T04:46:56.568 INFO:teuthology.task.ceph.mon.b.err:common/Thread.cc: 110: FAILED assert(ret == 0)
2012-11-07T04:46:56.568 INFO:teuthology.task.ceph.mon.b.err: ceph version 0.53 (commit:2528b5ee105b16352c91af064af5c0b5a7d45d7c)
2012-11-07T04:46:56.569 INFO:teuthology.task.ceph.mon.b.err: 1: (Thread::create(unsigned long)+0x8a) [0x5e274a]
2012-11-07T04:46:56.569 INFO:teuthology.task.ceph.mon.b.err: 2: (SimpleMessenger::add_accept_pipe(int)+0x6a) [0x5ddb7a]
2012-11-07T04:46:56.569 INFO:teuthology.task.ceph.mon.b.err: 3: (Accepter::entry()+0x279) [0x63b789]
2012-11-07T04:46:56.569 INFO:teuthology.task.ceph.mon.b.err: 4: (()+0x7e9a) [0x7f03b5aa3e9a]
2012-11-07T04:46:56.569 INFO:teuthology.task.ceph.mon.b.err: 5: (clone()+0x6d) [0x7f03b425c4bd]
2012-11-07T04:46:56.569 INFO:teuthology.task.ceph.mon.b.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2012-11-07T04:46:56.569 INFO:teuthology.task.ceph.mon.b.err:2012-11-07 04:46:39.921454 7f03b274d700 -1 common/Thread.cc: In function 'void Thread::create(size_t)' thread 7f03b274d700 time 2012-11-07 04:46:39.920768
2012-11-07T04:46:56.570 INFO:teuthology.task.ceph.mon.b.err:common/Thread.cc: 110: FAILED assert(ret == 0)
2012-11-07T04:46:56.570 INFO:teuthology.task.ceph.mon.b.err:
2012-11-07T04:46:56.570 INFO:teuthology.task.ceph.mon.b.err: ceph version 0.53 (commit:2528b5ee105b16352c91af064af5c0b5a7d45d7c)
2012-11-07T04:46:56.570 INFO:teuthology.task.ceph.mon.b.err: 1: (Thread::create(unsigned long)+0x8a) [0x5e274a]
2012-11-07T04:46:56.570 INFO:teuthology.task.ceph.mon.b.err: 2: (SimpleMessenger::add_accept_pipe(int)+0x6a) [0x5ddb7a]
2012-11-07T04:46:56.570 INFO:teuthology.task.ceph.mon.b.err: 3: (Accepter::entry()+0x279) [0x63b789]
2012-11-07T04:46:56.570 INFO:teuthology.task.ceph.mon.b.err: 4: (()+0x7e9a) [0x7f03b5aa3e9a]
2012-11-07T04:46:56.571 INFO:teuthology.task.ceph.mon.b.err: 5: (clone()+0x6d) [0x7f03b425c4bd]
2012-11-07T04:46:56.571 INFO:teuthology.task.ceph.mon.b.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2012-11-07T04:46:56.571 INFO:teuthology.task.ceph.mon.b.err:
2012-11-07T04:46:56.700 INFO:teuthology.task.ceph.mon.b.err:     0> 2012-11-07 04:46:39.921454 7f03b274d700 -1 common/Thread.cc: In function 'void Thread::create(size_t)' thread 7f03b274d700 time 2012-11-07 04:46:39.920768
2012-11-07T04:46:56.700 INFO:teuthology.task.ceph.mon.b.err:common/Thread.cc: 110: FAILED assert(ret == 0)
2012-11-07T04:46:56.700 INFO:teuthology.task.ceph.mon.b.err:
2012-11-07T04:46:56.700 INFO:teuthology.task.ceph.mon.b.err: ceph version 0.53 (commit:2528b5ee105b16352c91af064af5c0b5a7d45d7c)
2012-11-07T04:46:56.700 INFO:teuthology.task.ceph.mon.b.err: 1: (Thread::create(unsigned long)+0x8a) [0x5e274a]
2012-11-07T04:46:56.701 INFO:teuthology.task.ceph.mon.b.err: 2: (SimpleMessenger::add_accept_pipe(int)+0x6a) [0x5ddb7a]
2012-11-07T04:46:56.701 INFO:teuthology.task.ceph.mon.b.err: 3: (Accepter::entry()+0x279) [0x63b789]
2012-11-07T04:46:56.701 INFO:teuthology.task.ceph.mon.b.err: 4: (()+0x7e9a) [0x7f03b5aa3e9a]
2012-11-07T04:46:56.701 INFO:teuthology.task.ceph.mon.b.err: 5: (clone()+0x6d) [0x7f03b425c4bd]
2012-11-07T04:46:56.701 INFO:teuthology.task.ceph.mon.b.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2012-11-07T04:46:56.701 INFO:teuthology.task.ceph.mon.b.err:
2012-11-07T04:46:56.702 INFO:teuthology.task.ceph.mon.b.err:terminate called after throwing an instance of 'ceph::FailedAssertion'
2012-11-07T04:46:56.702 INFO:teuthology.task.ceph.mon.b.err:*** Caught signal (Aborted) **
2012-11-07T04:46:56.703 INFO:teuthology.task.ceph.mon.b.err: in thread 7f03b274d700
2012-11-07T04:46:56.703 INFO:teuthology.task.ceph.mon.b.err: ceph version 0.53 (commit:2528b5ee105b16352c91af064af5c0b5a7d45d7c)
2012-11-07T04:46:56.703 INFO:teuthology.task.ceph.mon.b.err: 1: /tmp/cephtest/binary/usr/local/bin/ceph-mon() [0x540a61]
2012-11-07T04:46:56.703 INFO:teuthology.task.ceph.mon.b.err: 2: (()+0xfcb0) [0x7f03b5aabcb0]
2012-11-07T04:46:56.703 INFO:teuthology.task.ceph.mon.b.err: 3: (gsignal()+0x35) [0x7f03b41a0445]
2012-11-07T04:46:56.703 INFO:teuthology.task.ceph.mon.b.err: 4: (abort()+0x17b) [0x7f03b41a3bab]
2012-11-07T04:46:56.703 INFO:teuthology.task.ceph.mon.b.err: 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f03b4aee69d]
2012-11-07T04:46:56.704 INFO:teuthology.task.ceph.mon.b.err: 6: (()+0xb5846) [0x7f03b4aec846]
2012-11-07T04:46:56.704 INFO:teuthology.task.ceph.mon.b.err: 7: (()+0xb5873) [0x7f03b4aec873]
2012-11-07T04:46:56.704 INFO:teuthology.task.ceph.mon.b.err: 8: (()+0xb596e) [0x7f03b4aec96e]
2012-11-07T04:46:56.704 INFO:teuthology.task.ceph.mon.b.err: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ea) [0x5f0caa]
2012-11-07T04:46:56.704 INFO:teuthology.task.ceph.mon.b.err: 10: (Thread::create(unsigned long)+0x8a) [0x5e274a]
2012-11-07T04:46:56.704 INFO:teuthology.task.ceph.mon.b.err: 11: (SimpleMessenger::add_accept_pipe(int)+0x6a) [0x5ddb7a]
2012-11-07T04:46:56.705 INFO:teuthology.task.ceph.mon.b.err: 12: (Accepter::entry()+0x279) [0x63b789]
2012-11-07T04:46:56.705 INFO:teuthology.task.ceph.mon.b.err: 13: (()+0x7e9a) [0x7f03b5aa3e9a]
2012-11-07T04:46:56.705 INFO:teuthology.task.ceph.mon.b.err: 14: (clone()+0x6d) [0x7f03b425c4bd]
2012-11-07T04:46:56.705 INFO:teuthology.task.ceph.mon.b.err:2012-11-07 04:46:40.055458 7f03b274d700 -1 *** Caught signal (Aborted) **
2012-11-07T04:46:56.706 INFO:teuthology.task.ceph.mon.b.err: in thread 7f03b274d700
2012-11-07T04:46:56.706 INFO:teuthology.task.ceph.mon.b.err:
2012-11-07T04:46:56.706 INFO:teuthology.task.ceph.mon.b.err: ceph version 0.53 (commit:2528b5ee105b16352c91af064af5c0b5a7d45d7c)
2012-11-07T04:46:56.706 INFO:teuthology.task.ceph.mon.b.err: 1: /tmp/cephtest/binary/usr/local/bin/ceph-mon() [0x540a61]
2012-11-07T04:46:56.706 INFO:teuthology.task.ceph.mon.b.err: 2: (()+0xfcb0) [0x7f03b5aabcb0]
2012-11-07T04:46:56.707 INFO:teuthology.task.ceph.mon.b.err: 3: (gsignal()+0x35) [0x7f03b41a0445]
2012-11-07T04:46:56.707 INFO:teuthology.task.ceph.mon.b.err: 4: (abort()+0x17b) [0x7f03b41a3bab]
2012-11-07T04:46:56.707 INFO:teuthology.task.ceph.mon.b.err: 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f03b4aee69d]
2012-11-07T04:46:56.707 INFO:teuthology.task.ceph.mon.b.err: 6: (()+0xb5846) [0x7f03b4aec846]
2012-11-07T04:46:56.707 INFO:teuthology.task.ceph.mon.b.err: 7: (()+0xb5873) [0x7f03b4aec873]
2012-11-07T04:46:56.707 INFO:teuthology.task.ceph.mon.b.err: 8: (()+0xb596e) [0x7f03b4aec96e]
2012-11-07T04:46:56.708 INFO:teuthology.task.ceph.mon.b.err: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ea) [0x5f0caa]
2012-11-07T04:46:56.708 INFO:teuthology.task.ceph.mon.b.err: 10: (Thread::create(unsigned long)+0x8a) [0x5e274a]
2012-11-07T04:46:56.708 INFO:teuthology.task.ceph.mon.b.err: 11: (SimpleMessenger::add_accept_pipe(int)+0x6a) [0x5ddb7a]
2012-11-07T04:46:56.708 INFO:teuthology.task.ceph.mon.b.err: 12: (Accepter::entry()+0x279) [0x63b789]
2012-11-07T04:46:56.708 INFO:teuthology.task.ceph.mon.b.err: 13: (()+0x7e9a) [0x7f03b5aa3e9a]
2012-11-07T04:46:56.708 INFO:teuthology.task.ceph.mon.b.err: 14: (clone()+0x6d) [0x7f03b425c4bd]
2012-11-07T04:46:56.709 INFO:teuthology.task.ceph.mon.b.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2012-11-07T04:46:56.709 INFO:teuthology.task.ceph.mon.b.err:
2012-11-07T04:46:56.709 INFO:teuthology.task.ceph.mon.b.err:     0> 2012-11-07 04:46:40.055458 7f03b274d700 -1 *** Caught signal (Aborted) **
2012-11-07T04:46:56.709 INFO:teuthology.task.ceph.mon.b.err: in thread 7f03b274d700
2012-11-07T04:46:56.709 INFO:teuthology.task.ceph.mon.b.err:
2012-11-07T04:46:56.709 INFO:teuthology.task.ceph.mon.b.err: ceph version 0.53 (commit:2528b5ee105b16352c91af064af5c0b5a7d45d7c)
2012-11-07T04:46:56.709 INFO:teuthology.task.ceph.mon.b.err: 1: /tmp/cephtest/binary/usr/local/bin/ceph-mon() [0x540a61]
2012-11-07T04:46:56.710 INFO:teuthology.task.ceph.mon.b.err: 2: (()+0xfcb0) [0x7f03b5aabcb0]
2012-11-07T04:46:56.710 INFO:teuthology.task.ceph.mon.b.err: 3: (gsignal()+0x35) [0x7f03b41a0445]
2012-11-07T04:46:56.710 INFO:teuthology.task.ceph.mon.b.err: 4: (abort()+0x17b) [0x7f03b41a3bab]
2012-11-07T04:46:56.710 INFO:teuthology.task.ceph.mon.b.err: 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f03b4aee69d]
2012-11-07T04:46:56.710 INFO:teuthology.task.ceph.mon.b.err: 6: (()+0xb5846) [0x7f03b4aec846]
2012-11-07T04:46:56.710 INFO:teuthology.task.ceph.mon.b.err: 7: (()+0xb5873) [0x7f03b4aec873]
2012-11-07T04:46:56.710 INFO:teuthology.task.ceph.mon.b.err: 8: (()+0xb596e) [0x7f03b4aec96e]
2012-11-07T04:46:56.711 INFO:teuthology.task.ceph.mon.b.err: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ea) [0x5f0caa]
2012-11-07T04:46:56.711 INFO:teuthology.task.ceph.mon.b.err: 10: (Thread::create(unsigned long)+0x8a) [0x5e274a]
2012-11-07T04:46:56.711 INFO:teuthology.task.ceph.mon.b.err: 11: (SimpleMessenger::add_accept_pipe(int)+0x6a) [0x5ddb7a]
2012-11-07T04:46:56.711 INFO:teuthology.task.ceph.mon.b.err: 12: (Accepter::entry()+0x279) [0x63b789]
2012-11-07T04:46:56.711 INFO:teuthology.task.ceph.mon.b.err: 13: (()+0x7e9a) [0x7f03b5aa3e9a]
2012-11-07T04:46:56.711 INFO:teuthology.task.ceph.mon.b.err: 14: (clone()+0x6d) [0x7f03b425c4bd]
2012-11-07T04:46:56.711 INFO:teuthology.task.ceph.mon.b.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@teuthology:/a/teuthology-2012-11-06_02:00:02-regression-testing-master-basic/10579$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 35152979e6181b1fbb4b61c3ff641c14df53ad66
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject socket failures: 500
    log-whitelist:
    - slow request
    sha1: 2528b5ee105b16352c91af064af5c0b5a7d45d7c
  s3tests:
    branch: testing
  workunit:
    sha1: 2528b5ee105b16352c91af064af5c0b5a7d45d7c
roles:
- - mon.a
  - mon.d
  - mon.g
  - mon.j
  - mon.m
  - mon.p
  - mon.s
  - osd.0
- - mon.b
  - mon.e
  - mon.h
  - mon.k
  - mon.n
  - mon.q
  - mon.t
  - mds.a
- - mon.c
  - mon.f
  - mon.i
  - mon.l
  - mon.o
  - mon.r
  - mon.u
  - osd.1
targets:
  ubuntu@plana54.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC60+3L8IN2WBpWHY94YAuCOMVdKs3xUqYQpO1ie127fBomk7fiEhR0RovhmDHWIzNr3qvvNkIp9Y+MHUcpZ7C4MFFMYsy2+zq026Ag3XLEOyZWDSyPfMapd5+nmuvxJqEvAx4wAWBhYVEB3aPFmDmz4mayZ9aSYoA1lhsClxfYpAHZ0zRWX3kY1KxXlk6UrZy0igYGvKIvmubkYcmFzOPsI3aWpgWU1rEXGWsFHOlwaor0KJPnpEsZYTrlPyLZqJcKbI/EcHgti0ak22vsDT7LVMKoyPXeUFL5ZGUEpuqQ+IMiECCMKa8X8vPG2MN9V6DK3gQezF+lo5CRCAu7DYdn
  ubuntu@plana73.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxoJnvRI1V0OJuQI9SosOedC7mj9O627LjoQWPKilJiBbHduPe1byBaKrgwTeEghl43VNf+EBs1+MwVH7zlDolnwN4tAlW9bRpC2SzURJfhZskp2CSQY3l8ca7a5f0J3hdOhx47oSSapN7O2cqmPzwlL/+MrFKGi+ITT613nUtzCjduZRPdhjyqZ0cQWeb0p1neDw5hbDBKd+HAH+ek/E6DK2PaqN6YAtmIgP76q0fQ85Omd0oDlmGXpKe3jlxlPT0W/5KD1+mpobPsh/EF2qar7IG/WqHHJ6NZAcXbdZ4KiMf9erP+Pk4KkD5SJ+e3GF7OEOwXtahKIIR1An4P2GD
  ubuntu@plana83.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPiffDjAR8Vr7B1C/WQ4ZiY0rnm4DSvknWpdrM1NMvHhxdrnceyfi6K8y0d+NdQKAkRXzCMbYFM/LU1gUEoIgT6k15wzz/KBUyWPbZ4BCNQNLng3z2aQ80FSA+CWYir0UVCOuj+glDN/MAqWmOwfJ7zHkej1/+hMH5mBTy9Nc8/VS7/yDIQgHLdHmeeMTlGVvBs+uc0b4rBd5kSzhzdwegxO+J86aR/HVsdL7/kEOVHeZQ0yWCT5mRa1+nTveZWYXBz+8ZIFzIx4e7QIhGw/XjDYqqhLD+ZjaUXuKNWG3GqLrQntt42LRvC6flIF11PoL1g0wsxW4DV5Vwdg+zkX71
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- mon_recovery: null

Actions

Copy link

Updated by Sage Weil over 11 years ago

Status changed from New to Resolved

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph

Custom queries

Bug #3460

ceph monitor crashed in the nightlies

Updated by Sage Weil over 11 years ago