Project

General

Profile

Bug #10911

mon: osd featuers aren't updated correctly

Added by Sage Weil about 9 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
firefly
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ubuntu@plana41:/var/log/ceph$ ceph osd dump -f json-pretty | grep feature
2015-02-18 09:05:45.559405 7fd9f8261700  0 -- :/1020897 >> 10.214.132.37:6790/0 pipe(0x7fd9f4065010 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd9f4026900).fault
            "features": 1125899906842623,
            "features": 70368744177663,
            "features": 1125899906842623,
            "features": 1125899906842623,
ubuntu@plana41:/var/log/ceph$ ceph tell osd.\* version
2015-02-18 09:05:57.659672 7f2cec263700  0 -- :/1020935 >> 10.214.132.37:6790/0 pipe(0x7f2ce8065010 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x7f2ce8026900).fault
osd.0: {
    "version": "ceph version 0.92-1117-g90a9ae4 (90a9ae450634a3f2281b93fbfadbc62ad703a0ec)" 
}
osd.1: {
    "version": "ceph version 0.92-1117-g90a9ae4 (90a9ae450634a3f2281b93fbfadbc62ad703a0ec)" 
}
osd.2: {
    "version": "ceph version 0.92-1117-g90a9ae4 (90a9ae450634a3f2281b93fbfadbc62ad703a0ec)" 
}
osd.3: {
    "version": "ceph version 0.92-1117-g90a9ae4 (90a9ae450634a3f2281b93fbfadbc62ad703a0ec)" 
}

running cluster is on plana41, plana58

reliably reproduced with the below:

interactive-on-error: true
overrides:
  admin_socket:
    branch: hammer
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
        mon warn on legacy crush tunables: false
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    - scrub mismatch
    - ScrubResult
    - must scrub before tier agent can activate
  install:
    ceph:
      sha1: 90a9ae450634a3f2281b93fbfadbc62ad703a0ec
  workunit:
    sha1: 26dfad798d5193295cbe86902a6bbcf773b03974
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
- - mon.b
  - mon.c
  - osd.2
  - osd.3
- - client.0
  - client.1
  - client.2
  - client.3
  - client.4
suite_branch: wip-10704
targets:
  ubuntu@plana41.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8wmSjYkK3No1H5yQZL9Yzl3M1nVtiLgScELyN3Duk71HaLRP8wr+Nq98Ylp8N3fK9oSdEtH5I92dEg9MhrEf+FAxyFDIffqpTSdokcJxMrlugALgz2/1OgONIpg15wHQXByJRE1gsecauIJmz+NcucGmfZubWex+/PsyDsDq6o26iXTvl7Bf/cEZ/ya5
k7YNXF5LFx/ekcMkeF1bWHqk4eyQgJLAYPCewBhAZ2EQxBSUQoUJQtokXhe873yiS+Wtzuoo/Z8CX4tzrn+PdK56czySdw9lBTQIAB5V3bX+4aTTTpv9QfvgHKy/5frd9k2Y/LBSn2NIq7jutToRx1qGl
  ubuntu@plana58.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQingRKteHV641XBd2nFr9XehgercdQI5pXSigO0VqKUckA8HyCNOy62K6W+l3eGGF0A54M3YJGULd+cWNH2YeA4uK/O2VDLYsBbLKL5/2tX0WPp0a2BTRwHBWz+mTNBKJfB4YFb/27SoZKLHp9bij1LQW6tjDGkRCUoJUJ1hkOcVpmYVMH7XZXT/t55a
YHoOCl7iyXEjOpB30XhpIzsYLISumDssk/hbVR03L5kiw9YKqA9Rr16sTN+ooTowRql5ktZU0lqCzxyRKs0mVv/rq5yorx91ZU6XUXvR63LfX/1sLwv5u/G2ix5BPd4ZlP5TevfABdUYB2hkwQNv5qe7D
  ubuntu@plana78.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8m+86JHGSyRkSWj9p/K6JUbRcPjB7TtLZ9OBudXAGZNgReiOJoCU5kkpwejl0uXXCOHe/DB/bH81JCQbqY3XCJjU5JZ1wBsL/owaErPSfbbaouNV2k1FQjiSXYtPzx+qwEOeOZtEBPQ4p04npai6NzPLX43OGx/UiAwpyEGfVxZedmci0VBtC7QdCQkP
3sNJqSxFYdoVGjU5jv6BarPqV8LM4v00f8TmD1GdP51bfLGSKii6UU1IKXXR78ifb+9QUX4p/Clkl6Qgz8CJ70Iu+mcBZclJaGoAyuoKBhXE2oi2W1cQVquPqloxbN+VbbjoOL5OHbGg2euxyohZhgJaF
tasks:
- chef: null
- clock.check: null
- install:
    branch: wip-ms-dump-on-send-giant
- print: '**** done installing giant'
- ceph:
    fs: xfs
- print: '**** done ceph'
- parallel:
  - workload
  - upgrade-sequence
- print: '**** done parallel'
- rados:
    clients:
    - client.1
    objects: 50
    op_weights:
      delete: 50
      read: 100
      rollback: 50
      snap_create: 50
      snap_remove: 50
      write: 100
    ops: 4000
- print: '**** done rados 4-final-workload'
- workunit:
    clients:
      client.1:
      - rados/load-gen-mix.sh
- print: '**** done rados/load-gen-mix.sh 4-final-workload'
- mon_thrash:
    revive_delay: 20
    thrash_delay: 1
- print: '**** done mon_thrash 4-final-workload'
- workunit:
    clients:
      client.1:
      - rados/test.sh
- print: '**** done rados/test.sh 4-final-workload'
- workunit:
    clients:
      client.1:
      - cls/test_cls_rbd.sh
- print: '**** done cls/test_cls_rbd.sh 4-final-workload'
- workunit:
    clients:
      client.1:
      - rbd/import_export.sh
    env:
      RBD_CREATE_ARGS: --new-format
- print: '**** done rbd/import_export.sh 4-final-workload'
- rgw:
  - client.1
- print: '**** done rgw 4-final-workload'
- swift:
    client.1:
      rgw_server: client.1
- print: '**** done swift 4-final-workload'
upgrade-sequence:
  sequential:
  - install.upgrade:
      mon.a: null
  - print: '**** done install.upgrade mon.a to the version from teuthology-suite arg'
  - ceph.restart:
      daemons:
      - mon.a
      wait-for-healthy: true
  - sleep:
      duration: 60
  - ceph.restart:
      daemons:
      - osd.0
      - osd.1
      wait-for-healthy: true
  - sleep:
      duration: 60
  - ceph.restart:
    - mds.a
  - sleep:
      duration: 60
  - print: '**** running mixed versions of osds and mons'
  - exec:
      mon.b:
      - ceph osd crush tunables firefly
  - print: '**** done ceph osd crush tunables firefly'
  - install.upgrade:
      mon.b: null
  - print: '**** done install.upgrade mon.b to the version from teuthology-suite arg'
  - ceph.restart:
      daemons:
      - mon.b
      - mon.c
      wait-for-healthy: true
  - sleep:
      duration: 60
  - ceph.restart:
      daemons:
      - osd.2
      - osd.3
      wait-for-healthy: true
  - sleep:
      duration: 60
verbose: true
workload:
  parallel:
  - rados:
      clients:
      - client.0
      ec_pool: true
      objects: 50
      op_weights:
        append: 100
        copy_from: 50
        delete: 50
        read: 100
        rmattr: 25
        rollback: 50
        setattr: 25
        snap_create: 50
        snap_remove: 50
        write: 0
      ops: 4000
  - print: '**** done rados ec parallel'
  - workunit:
      branch: giant
      clients:
        client.1:
        - cls
  - print: '**** done cls 2-workload parallel'
  - workunit:
      branch: giant
      clients:
        client.2:
        - rados/load-gen-big.sh
  - print: '**** done rados/load-gen-big.sh 2-workload parallel'
  - sequential:
    - exec:
        client.0:
        - ceph osd pool create base 4
        - ceph osd pool create cache 4
        - ceph osd tier add base cache
        - ceph osd tier cache-mode cache writeback
        - ceph osd tier set-overlay base cache
        - ceph osd pool set cache hit_set_type bloom
        - ceph osd pool set cache hit_set_count 8
        - ceph osd pool set cache hit_set_period 3600
        - ceph osd pool set cache target_max_objects 250
    - rados:
        clients:
        - client.0
        objects: 500
        op_weights:
          copy_from: 50
          delete: 50
          evict: 50
          flush: 50
          read: 100
          rollback: 50
          snap_create: 50
          snap_remove: 50
          try_flush: 50
          write: 100
        ops: 4000
        pool_snaps: true
        pools:
        - base
    - print: '**** done test_cache-pool-snaps 2-workload parallel_run'
  - workunit:
      branch: giant
      clients:
        client.3:
        - rbd/test_librbd.sh
  - print: '**** done rbd/test_librbd.sh 2-workload parallel'
  - workunit:
      branch: giant
      clients:
        client.4:
        - rbd/test_librbd_python.sh
  - print: '**** done rbd/test_librbd_python.sh 2-workload parallel'


Related issues

Copied to Ceph - Backport #12027: mon: osd featuers aren't updated correctly Resolved 02/18/2015

Associated revisions

Revision bee91548 (diff)
Added by Sage Weil about 9 years ago

osd,mon: explicitly specify OSD features in MOSDBoot

We are using the connection features to populate the features field in the
OSDMap, but this is the intersection of mon and osd features, not the
osd features. Fix this by explicitly specifying the features in
MOSDBoot.

Fixes: #10911
Backport: giant, firefly
Signed-off-by: Sage Weil <>

Revision 254676b6 (diff)
Added by Sage Weil almost 9 years ago

osd,mon: explicitly specify OSD features in MOSDBoot

We are using the connection features to populate the features field in the
OSDMap, but this is the intersection of mon and osd features, not the
osd features. Fix this by explicitly specifying the features in
MOSDBoot.

Fixes: #10911
Backport: giant, firefly
Signed-off-by: Sage Weil <>
(cherry picked from commit bee91548b07c6a28314dddc50fef8b6a2677e774)

Conflicts:
src/osd/OSD.cc
Minor difference in the MOSDBoot constructor

History

#1 Updated by Sage Weil about 9 years ago

  • Status changed from 12 to 7

#2 Updated by Sage Weil about 9 years ago

  • Status changed from 7 to Resolved

#3 Updated by David Zafman almost 9 years ago

  • Status changed from Resolved to Pending Backport
  • Regression set to No

This was never backported. I've create wip-10911-firefly for testing.

#4 Updated by Loïc Dachary almost 9 years ago

  • Backport set to firefly

#5 Updated by Nathan Cutler over 8 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF