Project

General

Profile

Actions
Copy link

Bug #9657

closed

MMDSBeacon: failure to decode; compat_version = 3 on Firefly monitor

Added by Yuri Weinstein over 9 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Greg Farnum
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-10-01_19:20:01-upgrade:firefly-x-giant-distro-basic-multi/525516/

2014-10-03T10:28:14.138 DEBUG:teuthology.misc:Ceph health: HEALTH_OK
2014-10-03T10:28:14.138 INFO:teuthology.task.sequential:In sequential, running task sleep...
2014-10-03T10:28:14.138 INFO:teuthology.task.sleep:Sleeping for 60
2014-10-03T10:28:21.723 INFO:tasks.ceph.mon.b.plana25.stderr:*** Caught signal (Segmentation fault) **
2014-10-03T10:28:21.723 INFO:tasks.ceph.mon.b.plana25.stderr: in thread 7fee597ae700
2014-10-03T10:28:21.740 INFO:tasks.ceph.mon.b.plana25.stderr: ceph version 0.80.6-27-g711a7e6 (711a7e6f81983ff2091caa0f232af914a04a041c)
2014-10-03T10:28:21.740 INFO:tasks.ceph.mon.b.plana25.stderr: 1: ceph-mon() [0x86933f]
2014-10-03T10:28:21.741 INFO:tasks.ceph.mon.b.plana25.stderr: 2: (()+0x10340) [0x7fee5f449340]
2014-10-03T10:28:21.741 INFO:tasks.ceph.mon.b.plana25.stderr: 3: (Message::get_source_inst() const+0x20) [0x57c2e0]
2014-10-03T10:28:21.741 INFO:tasks.ceph.mon.b.plana25.stderr: 4: (Monitor::handle_forward(MForward*)+0x2fe) [0x571d6e]
2014-10-03T10:28:21.741 INFO:tasks.ceph.mon.b.plana25.stderr: 5: (Monitor::dispatch(MonSession*, Message*, bool)+0x360) [0x572a60]
2014-10-03T10:28:21.742 INFO:tasks.ceph.mon.b.plana25.stderr: 6: (Monitor::_ms_dispatch(Message*)+0x215) [0x573045]
2014-10-03T10:28:21.742 INFO:tasks.ceph.mon.b.plana25.stderr: 7: (Monitor::ms_dispatch(Message*)+0x20) [0x590da0]
2014-10-03T10:28:21.742 INFO:tasks.ceph.mon.b.plana25.stderr: 8: (DispatchQueue::entry()+0x57a) [0x836bfa]
2014-10-03T10:28:21.742 INFO:tasks.ceph.mon.b.plana25.stderr: 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x74e67d]
2014-10-03T10:28:21.743 INFO:tasks.ceph.mon.b.plana25.stderr: 10: (()+0x8182) [0x7fee5f441182]
2014-10-03T10:28:21.743 INFO:tasks.ceph.mon.b.plana25.stderr: 11: (clone()+0x6d) [0x7fee5dbb538d]
2014-10-03T10:28:21.743 INFO:tasks.ceph.mon.b.plana25.stderr:2014-10-03 10:28:21.739553 7fee597ae700 -1 *** Caught signal (Segmentation fault) **
2014-10-03T10:28:21.743 INFO:tasks.ceph.mon.b.plana25.stderr: in thread 7fee597ae700
2014-10-03T10:28:21.744 INFO:tasks.ceph.mon.b.plana25.stderr:
2014-10-03T10:28:21.744 INFO:tasks.ceph.mon.b.plana25.stderr: ceph version 0.80.6-27-g711a7e6 (711a7e6f81983ff2091caa0f232af914a04a041c)
2014-10-03T10:28:21.744 INFO:tasks.ceph.mon.b.plana25.stderr: 1: ceph-mon() [0x86933f]
2014-10-03T10:28:21.744 INFO:tasks.ceph.mon.b.plana25.stderr: 2: (()+0x10340) [0x7fee5f449340]
2014-10-03T10:28:21.745 INFO:tasks.ceph.mon.b.plana25.stderr: 3: (Message::get_source_inst() const+0x20) [0x57c2e0]
2014-10-03T10:28:21.745 INFO:tasks.ceph.mon.b.plana25.stderr: 4: (Monitor::handle_forward(MForward*)+0x2fe) [0x571d6e]
2014-10-03T10:28:21.745 INFO:tasks.ceph.mon.b.plana25.stderr: 5: (Monitor::dispatch(MonSession*, Message*, bool)+0x360) [0x572a60]
2014-10-03T10:28:21.745 INFO:tasks.ceph.mon.b.plana25.stderr: 6: (Monitor::_ms_dispatch(Message*)+0x215) [0x573045]
2014-10-03T10:28:21.746 INFO:tasks.ceph.mon.b.plana25.stderr: 7: (Monitor::ms_dispatch(Message*)+0x20) [0x590da0]
2014-10-03T10:28:21.746 INFO:tasks.ceph.mon.b.plana25.stderr: 8: (DispatchQueue::entry()+0x57a) [0x836bfa]
2014-10-03T10:28:21.746 INFO:tasks.ceph.mon.b.plana25.stderr: 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x74e67d]
2014-10-03T10:28:21.746 INFO:tasks.ceph.mon.b.plana25.stderr: 10: (()+0x8182) [0x7fee5f441182]
2014-10-03T10:28:21.747 INFO:tasks.ceph.mon.b.plana25.stderr: 11: (clone()+0x6d) [0x7fee5dbb538d]
2014-10-03T10:28:21.747 INFO:tasks.ceph.mon.b.plana25.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-10-03T10:28:21.747 INFO:tasks.ceph.mon.b.plana25.stderr:
2014-10-03T10:28:21.836 INFO:tasks.ceph.mon.b.plana25.stderr:     0> 2014-10-03 10:28:21.739553 7fee597ae700 -1 *** Caught signal (Segmentation fault) **
2014-10-03T10:28:21.837 INFO:tasks.ceph.mon.b.plana25.stderr: in thread 7fee597ae700
2014-10-03T10:28:21.837 INFO:tasks.ceph.mon.b.plana25.stderr:
2014-10-03T10:28:21.837 INFO:tasks.ceph.mon.b.plana25.stderr: ceph version 0.80.6-27-g711a7e6 (711a7e6f81983ff2091caa0f232af914a04a041c)
2014-10-03T10:28:21.837 INFO:tasks.ceph.mon.b.plana25.stderr: 1: ceph-mon() [0x86933f]
2014-10-03T10:28:21.837 INFO:tasks.ceph.mon.b.plana25.stderr: 2: (()+0x10340) [0x7fee5f449340]
2014-10-03T10:28:21.838 INFO:tasks.ceph.mon.b.plana25.stderr: 3: (Message::get_source_inst() const+0x20) [0x57c2e0]
2014-10-03T10:28:21.838 INFO:tasks.ceph.mon.b.plana25.stderr: 4: (Monitor::handle_forward(MForward*)+0x2fe) [0x571d6e]
2014-10-03T10:28:21.838 INFO:tasks.ceph.mon.b.plana25.stderr: 5: (Monitor::dispatch(MonSession*, Message*, bool)+0x360) [0x572a60]
2014-10-03T10:28:21.838 INFO:tasks.ceph.mon.b.plana25.stderr: 6: (Monitor::_ms_dispatch(Message*)+0x215) [0x573045]
2014-10-03T10:28:21.839 INFO:tasks.ceph.mon.b.plana25.stderr: 7: (Monitor::ms_dispatch(Message*)+0x20) [0x590da0]
2014-10-03T10:28:21.839 INFO:tasks.ceph.mon.b.plana25.stderr: 8: (DispatchQueue::entry()+0x57a) [0x836bfa]
2014-10-03T10:28:21.839 INFO:tasks.ceph.mon.b.plana25.stderr: 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x74e67d]
2014-10-03T10:28:21.839 INFO:tasks.ceph.mon.b.plana25.stderr: 10: (()+0x8182) [0x7fee5f441182]
2014-10-03T10:28:21.840 INFO:tasks.ceph.mon.b.plana25.stderr: 11: (clone()+0x6d) [0x7fee5dbb538d]
2014-10-03T10:28:21.840 INFO:tasks.ceph.mon.b.plana25.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

archive_path: /var/lib/teuthworker/archive/teuthology-2014-10-01_19:20:01-upgrade:firefly-x-giant-distro-basic-multi/525516
branch: giant
description: upgrade:firefly-x/parallel/{0-cluster/start.yaml 1-firefly-install/firefly.yaml
  2-workload/test_rbd_python.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final-upgrade/client.yaml
  5-final-workload/rgw_s3tests.yaml distros/ubuntu_14.04.yaml}
email: ceph-qa@ceph.com
job_id: '525516'
kernel: &id001
  kdb: true
  sha1: distro
last_in_suite: false
machine_type: plana,burnupi,mira
name: teuthology-2014-10-01_19:20:01-upgrade:firefly-x-giant-distro-basic-multi
nuke-on-error: true
os_type: ubuntu
os_version: '14.04'
overrides:
  admin_socket:
    branch: giant
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
        mon warn on legacy crush tunables: false
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    - scrub mismatch
    - ScrubResult
    sha1: b1ca1f23ff47857a27f6196c5a050f83f3acc9fc
  ceph-deploy:
    branch:
      dev: giant
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: b1ca1f23ff47857a27f6196c5a050f83f3acc9fc
  s3tests:
    branch: giant
  workunit:
    sha1: b1ca1f23ff47857a27f6196c5a050f83f3acc9fc
owner: scheduled_teuthology@teuthology
priority: 1000
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
- - mon.b
  - mon.c
  - osd.2
  - osd.3
- - client.0
  - client.1
suite: upgrade:firefly-x
suite_branch: master
suite_path: /var/lib/teuthworker/src/ceph-qa-suite_master
targets:
  ubuntu@burnupi22.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvcs33PS2MQMxrCXm6jPOz1tz2i0bGA+8uVmTWe7Iu2W+fsHonGTMHJxPVYRAMe8WDulEYtkwK64s9D/Ph38kRK0o62SA599NKVIPvh1LzZadkWCX6aKlLv6cQYvQxOaUBuAlOIKQ5h0IKxu2lBPCHg6CIRHLUCYmTcR2PzXahMS9ToGMq+NS36+/4HmYPQ80lJcf1D9J+m6ETVMZ+DDcV4B6DWyImczwfNvXJY/Mj10bh4ZyEt5MSTVqfFKqNa3K1fWDzWUfsx8G3QGnyXNwuhUfRBslkIn4bFM2oJvGpFqZOSPGEskjM3IZaqhcoydnYDTTIWHG/8K5WJ70ZBhdx
  ubuntu@burnupi34.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVHu1u8/oxlx4Gs/CzuGsF6R5obvz8zBIZJ2oW6ZlWn3da3ybaWDEY3rmRtCEpmFIXK5UKFRFEqlKcbDVbl3OB53a4SUcgLgH0YcVgab3zy4rp7SDdBXzGJK7aM7hhGiKY73O7pKpFLX8thRxNIzRBR1Rr49Re41WXfb/45fDl2tiGNMX0QgorKUtMCkeKv4C/NhG4g+pk0j2kur4QCUfFGGzcYJNlpGzmyBoe0g8UYtLAPKOBjpUHY4iDwe2hB36ifiW1T9WvJ3f7/axcZpFuFosdMEJJ3mrIOAeko4CpcV7lJVCT3S/Kj9KsyklLt682ni999dQ/RRHDQkqd0Qth
  ubuntu@plana25.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC85wgMM7frtcFeCfatlKcc0Ru1HB4X/557M//6iIT2hQExLRtPADyOpfZdZmhP4Nh/mP6C9oB5gYH82sHnuVbCboq9J9OzBK0STFo4OIToRgbLJCTRfNuKt0VX0WCpvneQfA4SKmAsO527HgDcY/yyhzg67rWIel4LilQpFbPTe+rB9wBjO/DpbhxoF7d8vQUwtt2dYv6BXOvYPHCvgydTCAMIOgHIP/UqQceJgj/I3u85851yllYnBNE7LaJRXB96FlRtO25ZV7F7pFYLxyCsm+vGfRmp5YqdP+Qw72UaXuMpan+dQDwzpfRklLvolrq9jOLLYIvwnzd+GQgbRR87
tasks:
- internal.lock_machines:
  - 3
  - plana,burnupi,mira
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.push_inventory: null
- internal.serialize_remote_roles: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    branch: firefly
- print: '**** done installing firefly'
- ceph:
    fs: xfs
- print: '**** done ceph'
- parallel:
  - workload
  - upgrade-sequence
- print: '**** done parallel'
- install.upgrade:
    client.0: null
- print: '**** done install.upgrade'
- rgw:
  - client.1
- s3tests:
    client.1:
      branch: dumpling
      rgw_server: client.1
teuthology_branch: master
tube: multi
upgrade-sequence:
  sequential:
  - install.upgrade:
      mon.a: null
  - print: '**** done install.upgrade mon.a to the version from teuthology-suite arg'
  - ceph.restart:
      daemons:
      - mon.a
      wait-for-healthy: true
  - sleep:
      duration: 60
  - ceph.restart:
      daemons:
      - osd.0
      - osd.1
      wait-for-healthy: true
  - sleep:
      duration: 60
  - ceph.restart:
    - mds.a
  - sleep:
      duration: 60
  - print: '**** running mixed versions of osds and mons'
  - exec:
      mon.b:
      - ceph osd crush tunables firefly
  - install.upgrade:
      mon.b: null
  - print: '**** done install.upgrade mon.b to the version from teuthology-suite arg'
  - ceph.restart:
      daemons:
      - mon.b
      - mon.c
      wait-for-healthy: true
  - sleep:
      duration: 60
  - ceph.restart:
      daemons:
      - osd.2
      - osd.3
      wait-for-healthy: true
  - sleep:
      duration: 60
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.multi.3171
workload:
  sequential:
  - workunit:
      branch: firefly
      clients:
        client.0:
        - rbd/test_librbd_python.sh

description: upgrade:firefly-x/parallel/{0-cluster/start.yaml 1-firefly-install/firefly.yaml
  2-workload/test_rbd_python.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final-upgrade/client.yaml
  5-final-workload/rgw_s3tests.yaml distros/ubuntu_14.04.yaml}
duration: 754.9805121421814
failure_reason: 'Command failed on plana25 with status 1: ''sudo adjust-ulimits ceph-coverage
  /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f -i b'''
flavor: basic
owner: scheduled_teuthology@teuthology
success: false
Actions
Copy link

Also available in: Atom PDF