Project

General

Profile

Actions

Bug #3292

closed

osd crash in handle_osd_ping

Added by Tamilarasi muthamizhan over 11 years ago. Updated over 11 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs:ubuntu@teuthology:/a/teuthology-2012-10-10_00:00:05-regression-next-testing-basic/5458

2012-10-10 01:54:05.661469 7f76f696f700 1 CephxAuthorizeHandler::verify_authorizer isvalid=0
2012-10-10 01:54:05.661450 7f76f9b8c700 -1 ** Caught signal (Segmentation fault) *
in thread 7f76f9b8c700

ceph version 0.52-838-gaed3612 (commit:aed3612f875a3aeb6463011cb630adc7c936adbd)
1: /tmp/cephtest/binary/usr/local/bin/ceph-osd() [0x74fa31]
2: (()+0xfcb0) [0x7f7708b16cb0]
3: (OSD::handle_osd_ping(MOSDPing*)+0x764) [0x5f7c54]
4: (OSD::heartbeat_dispatch(Message*)+0x25b) [0x5f85eb]
5: (DispatchQueue::entry()+0x711) [0x88eab1]
6: (DispatchQueue::DispatchThread::entry()+0xd) [0x7e74cd]
7: (()+0x7e9a) [0x7f7708b0ee9a]
8: (clone()+0x6d) [0x7f7706eb24bd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ubuntu@teuthology:/a/teuthology-2012-10-10_00:00:05-regression-next-testing-basic/5458$ cat summary.yaml 
ceph-sha1: aed3612f875a3aeb6463011cb630adc7c936adbd
client.0-kernel-sha1: 4282bca13961e22c10fb9f0d8f9c338dff11099c
description: collection:rbd-basic clusters:fixed-3.yaml fs:btrfs.yaml msgr-failures:few.yaml
  tasks:rbd_python_api_tests_old_format.yaml
duration: 804.11426186561584
failure_reason: 'Command failed with status 1: ''/tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage
  /tmp/cephtest/archive/coverage /tmp/cephtest/daemon-helper kill /tmp/cephtest/binary/usr/local/bin/ceph-osd
  -f -i 3 -c /tmp/cephtest/ceph.conf'''
flavor: basic
mon.a-kernel-sha1: 4282bca13961e22c10fb9f0d8f9c338dff11099c
mon.b-kernel-sha1: 4282bca13961e22c10fb9f0d8f9c338dff11099c
owner: scheduled_teuthology@teuthology
success: false
ubuntu@teuthology:/a/teuthology-2012-10-10_00:00:05-regression-next-testing-basic/5458$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 4282bca13961e22c10fb9f0d8f9c338dff11099c
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: aed3612f875a3aeb6463011cb630adc7c936adbd
  s3tests:
    branch: next
  workunit:
    sha1: aed3612f875a3aeb6463011cb630adc7c936adbd
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana36.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCe7CpJbnd7W2/n42TTTjDArnVkyZfbRANfmkdgfDM+6AYg6qd9wUhes6LP++eMvhuM96Sz5W4380o8OME0cguG1LkkADbm8pQbPAPZwF1Fj28YxgZKpc2PTPsF+sjOujC+AaXaQ82ffSkLL0oElKZgAiFEGCytSdUNFHZxjztDIOoWlt7kylQCy4sJCEbND8JFwFfeGyyePvMl3CNdbnR7H5GuyIx70iglLBO/XFwArjeOUZ/FboRZWOBivpZQf9IMy8k2rQetzxTyugd7cTVdq1G5N5NeHpbQfv286G2oDaZj1HT252jDF04UP083zMxH1W9gmOoUKzIhl+iXaNLZ
  ubuntu@plana37.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrxOb9f5/SfItd83HOnLVyJRnfji0fbdvL+3T82akjV6J4s/nyR8Bu+rpXbyUwu2BRDoxK4pT2dBqw86meq1qbU5Q1ypWBSH41MYGd213fy0g8YibFiYVGmXFCSwtY8X2Pet9vtLDoYvtnsgNI8djy5GPkQyZFKSszJHznZvQU10NWfM6RfxxtsBKXC/aot4QXb3GIym2/EmeuTAAef6p98dd15P9l9HQkpwXZLwiDZ53IbU79CTINo5HTD/6+1XHUcjb1OUKzQMx1jU485gW6IlsR0G0jJKSv+YEu4zSxxva7gWt1AYxGo2jhNDffEGLsNurzXFf9yeYshCTAszLf
  ubuntu@plana38.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDARPmUWw72IMuZaJozKLrN06DeIgQacSM4fOhaa9jLxqnt8VZRSynN2sMzbKfA+JjLgz69zBawXb0TiVu0cbdqzPV94FylkRduEcZYM9zeD3B74BKZTltZgmugaEPv20olaEfseYMV52VTDNMKSdKPbYmYOVCCpzDJAJIuWJ51UdngsBMDPwTqys49Dcj9Gul71L0FDUSa+pavNG5Ricao6tJyv6rgVrUIz8UfutVB/5xYjwYnR8yFDQuKhmteY+kk3ve8nqgNR9VjuRjfP5mg5jT5e5CtDi1OSCWUi6lJKepv+IVKCSs1vc/1WpHiNLYCLNh9PImzi5GlljuS7o7x
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- ceph-fuse: null
- workunit:
    clients:
      client.0:
      - rbd/test_librbd_python.sh
Actions #1

Updated by Tamilarasi muthamizhan over 11 years ago

Logs: ubuntu@teuthology:/a/teuthology-2012-10-10_00:00:05-regression-next-testing-basic/5460

2012-10-10 01:53:02.041414 7f6982382700 -1 *** Caught signal (Aborted) **
 in thread 7f6982382700

 ceph version 0.52-838-gaed3612 (commit:aed3612f875a3aeb6463011cb630adc7c936adbd)
 1: /tmp/cephtest/binary/usr/local/bin/ceph-osd() [0x74fa31]
 2: (()+0xfcb0) [0x7f699130ccb0]
 3: (gsignal()+0x35) [0x7f698f5ec445]
 4: (abort()+0x17b) [0x7f698f5efbab]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f698ff3a69d]
 6: (()+0xb5846) [0x7f698ff38846]
 7: (()+0xb5873) [0x7f698ff38873]
 8: (()+0xb596e) [0x7f698ff3896e]
 9: (std::__throw_length_error(char const*)+0x57) [0x7f698fee5907]
 10: (()+0x9eaa2) [0x7f698ff21aa2]
 11: (char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag)+0x35) [0x7f698ff23495]
 12: (std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, unsigned long, std::allocator<char> const&)+0x1d) [0x7f698ff2361d]
 13: (PrebufferedStreambuf::get_str() const+0xbd) [0x7dc15d]
 14: (ceph::log::Log::_flush(ceph::log::EntryQueue*, ceph::log::EntryQueue*, bool)+0x1d7) [0x770a77]
 15: (ceph::log::Log::dump_recent()+0x11d) [0x7715dd]
 16: /tmp/cephtest/binary/usr/local/bin/ceph-osd() [0x74fb78]
 17: (()+0xfcb0) [0x7f699130ccb0]
 18: (OSD::handle_osd_ping(MOSDPing*)+0x764) [0x5f7c54]
 19: (OSD::heartbeat_dispatch(Message*)+0x25b) [0x5f85eb]
 20: (DispatchQueue::entry()+0x711) [0x88eab1]
 21: (DispatchQueue::DispatchThread::entry()+0xd) [0x7e74cd]
 22: (()+0x7e9a) [0x7f6991304e9a]
 23: (clone()+0x6d) [0x7f698f6a84bd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@teuthology:/a/teuthology-2012-10-10_00:00:05-regression-next-testing-basic/5460$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 4282bca13961e22c10fb9f0d8f9c338dff11099c
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject socket failures: 500
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: aed3612f875a3aeb6463011cb630adc7c936adbd
  s3tests:
    branch: next
  workunit:
    sha1: aed3612f875a3aeb6463011cb630adc7c936adbd
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana19.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0IExVUuQEnwBzUbN6+jgEb/qLLsSDFLAe0OPv+R3Q3uUZn+4QjB+FQ3sMMEwaGpEhUfbpWn3xZbCmi4NngTsgPDjImYJPMeMecaxvVXqAJt4IM6gdBN0415lrKXtbXaGBJCmeFFDB+xN+JN6Qhk8DWN62DnJB8MS5vKn9u0S3HvtGzY/QrnutT3AX9I4097isbTepLFRC4n2CoAC9srXaxAprFgLgOIYHm386B3W7yK3yfIImWofvZxYpPCsr/7ws8DSgdRX9eDucGQfY2WcYKCEIgZclGPBhExm7q/ahHUqYKrzWY3RD93AuXJVgOJk5Yp8C1Ryx8vyJkAxBW4nb
  ubuntu@plana22.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6ZmsmnHcSY7O9viGtUzt5WebiPbwcXo9tg5qgWsaqn46DeegKbdsQ55ajysSUVVhvQA5hW6J9IYyZ5MjtlY2G/whyHYG85tNpAUiuedaQHmzARtL3URZmy2ZxwXgYyPHW3t1n0cu6KSb4pTv9vBjcaCouV2wgrinHAISzDOVuUeXdIhC8Tr3MB0nD1Gw6Xcak680XsQw6oYP6cM+yGCZ7sF15W8TR9IJGmphMIvtd8aTuBo9yet5rIxUfzpCM9Jiv+XgH2oT9h9WfacuL1uQ2C/dHUWoPynK36Uv2J785bfw/hVVtuSGu9Lb1n4o8p8Z88Ex4i8KaOxMiQAs3zqOx
  ubuntu@plana34.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc3URAyD772tnoWHzOPXosRcjoZsDTSs3bhFuCp8DZdkkphtXy9eAuhTz5+nCUCzcumDfpxcYsisl84204gw3lmwvEicW4Tf+3NRWuHu05s+VGSFe81fGKGxUviM64dEvA/A54KAXbg+hxPg/rYFhHgDSLDL61MKYYZWI8kMV+M93qShzApy73Z8tDdva3WF0DrXV37TnPwSxc4R5j08t/y+5/WDjIj7u21S+kxqfGNQ9ycx/yHT5lf//2HS60dvBnMfQda8NEsUAtS+/9lA85qFiBOI2MzrnCRdi1Z5htVvM/em7mQmT5ttDvF4R5Qn9MgeV438lVoagO0+lvKoMf
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- ceph-fuse: null
- workunit:
    clients:
      client.0:
      - rbd/copy.sh
    env:
      RBD_CREATE_ARGS: --new-format
ubuntu@teuthology:/a/teuthology-2012-10-10_00:00:05-regression-next-testing-basic/5460$ cat summary.yaml 
ceph-sha1: aed3612f875a3aeb6463011cb630adc7c936adbd
client.0-kernel-sha1: 4282bca13961e22c10fb9f0d8f9c338dff11099c
description: collection:rbd-basic clusters:fixed-3.yaml fs:btrfs.yaml msgr-failures:many.yaml
  tasks:rbd_cli_copy.yaml
duration: 663.96961379051208
failure_reason: 'Command failed with status 1: ''/tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage
  /tmp/cephtest/archive/coverage /tmp/cephtest/daemon-helper kill /tmp/cephtest/binary/usr/local/bin/ceph-osd
  -f -i 1 -c /tmp/cephtest/ceph.conf'''
flavor: basic
mon.a-kernel-sha1: 4282bca13961e22c10fb9f0d8f9c338dff11099c
mon.b-kernel-sha1: 4282bca13961e22c10fb9f0d8f9c338dff11099c
owner: scheduled_teuthology@teuthology
success: false

Actions #2

Updated by Sage Weil over 11 years ago

  • Priority changed from Normal to High
Actions #3

Updated by Tamilarasi muthamizhan over 11 years ago

Recent logs: ubuntu@teuthology:/a/teuthology-2012-10-25_02:00:04-regression-testing-master-basic/1516

Actions #4

Updated by Tamilarasi muthamizhan over 11 years ago

Recent log: ubuntu@teuthology:/a/teuthology-2012-10-25_02:00:04-regression-testing-master-basic/1520

Actions #5

Updated by Sage Weil over 11 years ago

  • Status changed from New to Resolved
Actions

Also available in: Atom PDF