Project

General

Profile

Actions
Copy link

Bug #10911

closed

mon: osd featuers aren't updated correctly

Added by Sage Weil about 9 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Sage Weil
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
firefly
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ubuntu@plana41:/var/log/ceph$ ceph osd dump -f json-pretty | grep feature
2015-02-18 09:05:45.559405 7fd9f8261700  0 -- :/1020897 >> 10.214.132.37:6790/0 pipe(0x7fd9f4065010 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd9f4026900).fault
            "features": 1125899906842623,
            "features": 70368744177663,
            "features": 1125899906842623,
            "features": 1125899906842623,
ubuntu@plana41:/var/log/ceph$ ceph tell osd.\* version
2015-02-18 09:05:57.659672 7f2cec263700  0 -- :/1020935 >> 10.214.132.37:6790/0 pipe(0x7f2ce8065010 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x7f2ce8026900).fault
osd.0: {
    "version": "ceph version 0.92-1117-g90a9ae4 (90a9ae450634a3f2281b93fbfadbc62ad703a0ec)" 
}
osd.1: {
    "version": "ceph version 0.92-1117-g90a9ae4 (90a9ae450634a3f2281b93fbfadbc62ad703a0ec)" 
}
osd.2: {
    "version": "ceph version 0.92-1117-g90a9ae4 (90a9ae450634a3f2281b93fbfadbc62ad703a0ec)" 
}
osd.3: {
    "version": "ceph version 0.92-1117-g90a9ae4 (90a9ae450634a3f2281b93fbfadbc62ad703a0ec)" 
}

running cluster is on plana41, plana58

reliably reproduced with the below:

interactive-on-error: true
overrides:
  admin_socket:
    branch: hammer
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
        mon warn on legacy crush tunables: false
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    - scrub mismatch
    - ScrubResult
    - must scrub before tier agent can activate
  install:
    ceph:
      sha1: 90a9ae450634a3f2281b93fbfadbc62ad703a0ec
  workunit:
    sha1: 26dfad798d5193295cbe86902a6bbcf773b03974
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
- - mon.b
  - mon.c
  - osd.2
  - osd.3
- - client.0
  - client.1
  - client.2
  - client.3
  - client.4
suite_branch: wip-10704
targets:
  ubuntu@plana41.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8wmSjYkK3No1H5yQZL9Yzl3M1nVtiLgScELyN3Duk71HaLRP8wr+Nq98Ylp8N3fK9oSdEtH5I92dEg9MhrEf+FAxyFDIffqpTSdokcJxMrlugALgz2/1OgONIpg15wHQXByJRE1gsecauIJmz+NcucGmfZubWex+/PsyDsDq6o26iXTvl7Bf/cEZ/ya5
k7YNXF5LFx/ekcMkeF1bWHqk4eyQgJLAYPCewBhAZ2EQxBSUQoUJQtokXhe873yiS+Wtzuoo/Z8CX4tzrn+PdK56czySdw9lBTQIAB5V3bX+4aTTTpv9QfvgHKy/5frd9k2Y/LBSn2NIq7jutToRx1qGl
  ubuntu@plana58.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQingRKteHV641XBd2nFr9XehgercdQI5pXSigO0VqKUckA8HyCNOy62K6W+l3eGGF0A54M3YJGULd+cWNH2YeA4uK/O2VDLYsBbLKL5/2tX0WPp0a2BTRwHBWz+mTNBKJfB4YFb/27SoZKLHp9bij1LQW6tjDGkRCUoJUJ1hkOcVpmYVMH7XZXT/t55a
YHoOCl7iyXEjOpB30XhpIzsYLISumDssk/hbVR03L5kiw9YKqA9Rr16sTN+ooTowRql5ktZU0lqCzxyRKs0mVv/rq5yorx91ZU6XUXvR63LfX/1sLwv5u/G2ix5BPd4ZlP5TevfABdUYB2hkwQNv5qe7D
  ubuntu@plana78.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8m+86JHGSyRkSWj9p/K6JUbRcPjB7TtLZ9OBudXAGZNgReiOJoCU5kkpwejl0uXXCOHe/DB/bH81JCQbqY3XCJjU5JZ1wBsL/owaErPSfbbaouNV2k1FQjiSXYtPzx+qwEOeOZtEBPQ4p04npai6NzPLX43OGx/UiAwpyEGfVxZedmci0VBtC7QdCQkP
3sNJqSxFYdoVGjU5jv6BarPqV8LM4v00f8TmD1GdP51bfLGSKii6UU1IKXXR78ifb+9QUX4p/Clkl6Qgz8CJ70Iu+mcBZclJaGoAyuoKBhXE2oi2W1cQVquPqloxbN+VbbjoOL5OHbGg2euxyohZhgJaF
tasks:
- chef: null
- clock.check: null
- install:
    branch: wip-ms-dump-on-send-giant
- print: '**** done installing giant'
- ceph:
    fs: xfs
- print: '**** done ceph'
- parallel:
  - workload
  - upgrade-sequence
- print: '**** done parallel'
- rados:
    clients:
    - client.1
    objects: 50
    op_weights:
      delete: 50
      read: 100
      rollback: 50
      snap_create: 50
      snap_remove: 50
      write: 100
    ops: 4000
- print: '**** done rados 4-final-workload'
- workunit:
    clients:
      client.1:
      - rados/load-gen-mix.sh
- print: '**** done rados/load-gen-mix.sh 4-final-workload'
- mon_thrash:
    revive_delay: 20
    thrash_delay: 1
- print: '**** done mon_thrash 4-final-workload'
- workunit:
    clients:
      client.1:
      - rados/test.sh
- print: '**** done rados/test.sh 4-final-workload'
- workunit:
    clients:
      client.1:
      - cls/test_cls_rbd.sh
- print: '**** done cls/test_cls_rbd.sh 4-final-workload'
- workunit:
    clients:
      client.1:
      - rbd/import_export.sh
    env:
      RBD_CREATE_ARGS: --new-format
- print: '**** done rbd/import_export.sh 4-final-workload'
- rgw:
  - client.1
- print: '**** done rgw 4-final-workload'
- swift:
    client.1:
      rgw_server: client.1
- print: '**** done swift 4-final-workload'
upgrade-sequence:
  sequential:
  - install.upgrade:
      mon.a: null
  - print: '**** done install.upgrade mon.a to the version from teuthology-suite arg'
  - ceph.restart:
      daemons:
      - mon.a
      wait-for-healthy: true
  - sleep:
      duration: 60
  - ceph.restart:
      daemons:
      - osd.0
      - osd.1
      wait-for-healthy: true
  - sleep:
      duration: 60
  - ceph.restart:
    - mds.a
  - sleep:
      duration: 60
  - print: '**** running mixed versions of osds and mons'
  - exec:
      mon.b:
      - ceph osd crush tunables firefly
  - print: '**** done ceph osd crush tunables firefly'
  - install.upgrade:
      mon.b: null
  - print: '**** done install.upgrade mon.b to the version from teuthology-suite arg'
  - ceph.restart:
      daemons:
      - mon.b
      - mon.c
      wait-for-healthy: true
  - sleep:
      duration: 60
  - ceph.restart:
      daemons:
      - osd.2
      - osd.3
      wait-for-healthy: true
  - sleep:
      duration: 60
verbose: true
workload:
  parallel:
  - rados:
      clients:
      - client.0
      ec_pool: true
      objects: 50
      op_weights:
        append: 100
        copy_from: 50
        delete: 50
        read: 100
        rmattr: 25
        rollback: 50
        setattr: 25
        snap_create: 50
        snap_remove: 50
        write: 0
      ops: 4000
  - print: '**** done rados ec parallel'
  - workunit:
      branch: giant
      clients:
        client.1:
        - cls
  - print: '**** done cls 2-workload parallel'
  - workunit:
      branch: giant
      clients:
        client.2:
        - rados/load-gen-big.sh
  - print: '**** done rados/load-gen-big.sh 2-workload parallel'
  - sequential:
    - exec:
        client.0:
        - ceph osd pool create base 4
        - ceph osd pool create cache 4
        - ceph osd tier add base cache
        - ceph osd tier cache-mode cache writeback
        - ceph osd tier set-overlay base cache
        - ceph osd pool set cache hit_set_type bloom
        - ceph osd pool set cache hit_set_count 8
        - ceph osd pool set cache hit_set_period 3600
        - ceph osd pool set cache target_max_objects 250
    - rados:
        clients:
        - client.0
        objects: 500
        op_weights:
          copy_from: 50
          delete: 50
          evict: 50
          flush: 50
          read: 100
          rollback: 50
          snap_create: 50
          snap_remove: 50
          try_flush: 50
          write: 100
        ops: 4000
        pool_snaps: true
        pools:
        - base
    - print: '**** done test_cache-pool-snaps 2-workload parallel_run'
  - workunit:
      branch: giant
      clients:
        client.3:
        - rbd/test_librbd.sh
  - print: '**** done rbd/test_librbd.sh 2-workload parallel'
  - workunit:
      branch: giant
      clients:
        client.4:
        - rbd/test_librbd_python.sh
  - print: '**** done rbd/test_librbd_python.sh 2-workload parallel'

Related issues 1 (0 open — 1 closed)

Copied to Ceph - Backport #12027: mon: osd featuers aren't updated correctlyResolvedNathan Cutler02/18/2015Actions
Actions
Copy link
 #1

Updated by Sage Weil about 9 years ago

  • Status changed from 12 to 7
Actions
Copy link
 #2

Updated by Sage Weil about 9 years ago

  • Status changed from 7 to Resolved
Actions
Copy link
 #3

Updated by David Zafman almost 9 years ago

  • Status changed from Resolved to Pending Backport
  • Regression set to No

This was never backported. I've create wip-10911-firefly for testing.

Actions
Copy link
 #4

Updated by Loïc Dachary almost 9 years ago

  • Backport set to firefly
Actions
Copy link
 #5

Updated by Nathan Cutler over 8 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF