Bug #5430: newfs makes ceph-mds segfault in suicide - CephFS - Ceph

Actions

Copy link

Bug #5430

closed

newfs makes ceph-mds segfault in suicide

Added by Sage Weil almost 11 years ago. Updated almost 8 years ago.

Status:

Duplicate

Priority:

Urgent

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

MDS

Labels (FS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2013-06-23T04:22:53.072 INFO:teuthology.task.workunit.client.0.err:+ '[' -s /tmp/mdsmap.30495 ']'
2013-06-23T04:22:53.072 INFO:teuthology.task.workunit.client.0.err:+ (( epoch = current_epoch + 1 ))
2013-06-23T04:22:53.073 INFO:teuthology.task.workunit.client.0.err:+ ceph mds setmap -i /tmp/mdsmap.30495 10
2013-06-23T04:22:53.399 INFO:teuthology.task.workunit.client.0.err:set mds map
2013-06-23T04:22:53.408 INFO:teuthology.task.workunit.client.0.err:+ rm /tmp/mdsmap.30495
2013-06-23T04:22:53.409 INFO:teuthology.task.workunit.client.0.err:+ ceph mds newfs 0 1 --yes-i-really-mean-it
2013-06-23T04:22:53.714 INFO:teuthology.task.ceph.mds.a.err:*** Caught signal (Segmentation fault) **
2013-06-23T04:22:53.715 INFO:teuthology.task.ceph.mds.a.err: in thread 7f1f3166d700
2013-06-23T04:22:53.717 INFO:teuthology.task.ceph.mds.a.err: ceph version 0.64-607-gb89d742 (b89d7420e3501247d6ed282d2253c95c758526b1)
2013-06-23T04:22:53.717 INFO:teuthology.task.ceph.mds.a.err: 1: ceph-mds() [0x97594a]
2013-06-23T04:22:53.718 INFO:teuthology.task.ceph.mds.a.err: 2: (()+0xfcb0) [0x7f1f35beacb0]
2013-06-23T04:22:53.718 INFO:teuthology.task.ceph.mds.a.err: 3: (SimpleMessenger::mark_down_all()+0x355) [0x86d0b5]
2013-06-23T04:22:53.718 INFO:teuthology.task.ceph.mds.a.err: 4: (SimpleMessenger::shutdown()+0x69) [0x8641b9]
2013-06-23T04:22:53.718 INFO:teuthology.task.ceph.mds.a.err: 5: (MDS::suicide()+0xdd) [0x5743ed]
2013-06-23T04:22:53.718 INFO:teuthology.task.ceph.mds.a.err: 6: (MDS::respawn()+0x633) [0x575083]
2013-06-23T04:22:53.719 INFO:teuthology.task.ceph.mds.a.err: 7: (MDS::handle_mds_map(MMDSMap*)+0x2974) [0x586d74]
2013-06-23T04:22:53.719 INFO:teuthology.task.ceph.mds.a.err: 8: (MDS::handle_core_message(Message*)+0x5bb) [0x58909b]
2013-06-23T04:22:53.719 INFO:teuthology.task.ceph.mds.a.err: 9: (MDS::_dispatch(Message*)+0x2f) [0x5897cf]
2013-06-23T04:22:53.719 INFO:teuthology.task.ceph.mds.a.err: 10: (MDS::ms_dispatch(Message*)+0x1d3) [0x58b253]
2013-06-23T04:22:53.719 INFO:teuthology.task.ceph.mds.a.err: 11: (DispatchQueue::entry()+0x549) [0x943ab9]
2013-06-23T04:22:53.720 INFO:teuthology.task.ceph.mds.a.err: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x86da5d]
2013-06-23T04:22:53.720 INFO:teuthology.task.ceph.mds.a.err: 13: (()+0x7e9a) [0x7f1f35be2e9a]
2013-06-23T04:22:53.720 INFO:teuthology.task.ceph.mds.a.err: 14: (clone()+0x6d) [0x7f1f3417fccd]
2013-06-23T04:22:53.720 INFO:teuthology.task.ceph.mds.a.err:2013-06-23 04:22:56.418031 7f1f3166d700 -1 *** Caught signal (Segmentation fault) **
2013-06-23T04:22:53.720 INFO:teuthology.task.ceph.mds.a.err: in thread 7f1f3166d700
2013-06-23T04:22:53.721 INFO:teuthology.task.ceph.mds.a.err:

triggered by cephtool tests,

ubuntu@teuthology:/a/teuthology-2013-06-23_01:00:12-rados-master-testing-basic/43199$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: 2dd322b42d608a37f3e5beed57a8fbc673da6e32
machine_type: plana
nuke-on-error: true
overrides:
  admin_socket:
    branch: master
  ceph:
    conf:
      global:
        ms inject socket failures: 500
      mon:
        debug mon: 20
        debug ms: 20
        debug paxos: 20
      osd:
        osd op thread timeout: 60
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: b89d7420e3501247d6ed282d2253c95c758526b1
  install:
    ceph:
      sha1: b89d7420e3501247d6ed282d2253c95c758526b1
  s3tests:
    branch: master
  workunit:
    sha1: b89d7420e3501247d6ed282d2253c95c758526b1
roles:
- - mon.0
  - mon.1
  - mon.2
  - mds.a
  - osd.0
  - osd.1
  - osd.2
  - client.0
tasks:
- chef: null
- clock.check: null
- install: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - had wrong client addr
- workunit:
    clients:
      all:
      - cephtool
      - mon/pool_ops.sh

Actions

Copy link

Updated by Sage Weil almost 11 years ago

2013-06-23 04:22:56.414770 7f1f3166d700  1 mds.-1.-1 handle_mds_map i (10.214.131.23:6812/29684) dne in the mdsmap, respawning myself
2013-06-23 04:22:56.414780 7f1f3166d700  1 mds.-1.-1 respawn
2013-06-23 04:22:56.414783 7f1f3166d700  1 mds.-1.-1  e: 'ceph-mds'
2013-06-23 04:22:56.414786 7f1f3166d700  1 mds.-1.-1  0: 'ceph-mds'
2013-06-23 04:22:56.414789 7f1f3166d700  1 mds.-1.-1  1: '-f'
2013-06-23 04:22:56.414791 7f1f3166d700  1 mds.-1.-1  2: '-i'
2013-06-23 04:22:56.414793 7f1f3166d700  1 mds.-1.-1  3: 'a'
2013-06-23 04:22:56.414809 7f1f3166d700  1 mds.-1.-1  cwd /home/ubuntu
2013-06-23 04:22:56.414849 7f1f3166d700  0 mds.-1.-1 respawn execv ceph-mds failed with (2) No such file or directory
2013-06-23 04:22:56.414864 7f1f3166d700  1 mds.-1.-1 suicide.  wanted down:dne, now down:dne

so i suspect respawn is broken with teuthology, and also suicide is broken in general?

Actions

Copy link