Project

General

Profile

Bug #15411

Added new OSDs with 0.94.6 disabled RBD access

Added by Bosse Klykken almost 8 years ago. Updated almost 7 years ago.

Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We've added two new nodes with 0.94.6, with a total of 60 OSDs, to a running cluster on 0.94.3, accidentally without upgrading the cluster to the newest version before adding the nodes, as is suggested in the documentation. After a short time, the RBD mounts stopped working, and couldn't be remounted. The RBD client logs mentioned stuff like:

libceph: osd44 10.xx.xx.xx:6820 feature set mismatch, my 2b84a842a42 < server's 102b84a842a42, missing 1000000000000
libceph: osd44 10.xx.xx.xx:6820 socket error on read
libceph: corrupt inc osdmap (-22) epoch 76574 off 60 (ffffc9002e58a058 of ffffc9002e58a01c-ffffc9002e58c1bc)
<and a lot of hex dumps filling the logs>

As far as we've been able to dig up, the feature set mismatch refers to CEPH_FEATURE_CRUSH_V4. When looking at the crush map, we noticed that the new OSDs were in buckets set with alg straw2, while the remainder were straw.

We've stopped the newly added OSDs, but the problem still persists:

#rbd map rbd-volume
rbd: sysfs write failed
rbd: map failed: (5) Input/output error

Attached is the compressed ceph report. After we generated this report, we tried removing the newly added OSDs from the crush map, in order to upgrade the cluster before adding them back, but the problem persists - we can't mount RBD volumes, and we get log messages with feature set mismatch as mentioned above.

History

#1 Updated by Loïc Dachary almost 8 years ago

RBD mounts

Could you add more details about how RBD volumes are mounted ? client and version ?

#2 Updated by Bosse Klykken almost 8 years ago

Issue was resolved by also removing the hosts properly from the crushmap.

ceph osd getcrushmap -o /tmp/crushmap
crushtool -d /tmp/crushmap -o /tmp/crushmap.txt
vim /tmp/crushmap.txt # removing references to hosts with straw2
crushtool -c /tmp/crushmap.txt -o /tmp/crushmap.new
ceph osd setcrushmap -i /tmp/crushmap.new

Our RBD mounts are now working again

#3 Updated by Bosse Klykken almost 8 years ago

Loic Dachary wrote:

Could you add more details about how RBD volumes are mounted ? client and version ?

I tried with several versions, the main one being 0.94.5. I tried upgrading my test system to 0.94.6 to see if that would help. I didn't make a note of what version I had before that, but I believe it was 0.94.3.

Example of volume:
size 3072 GB in 786432 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1dffe522ae8944a
format: 2
features: layering
flags:

#4 Updated by Bosse Klykken almost 8 years ago

Bosse Klykken wrote:

Our RBD mounts are now working again

Actually, no. We were able to mount the RBD, but it seems to lock after some time. I also tried to delete a test volume with 'rbd rm volumename' and that hangs as well.

Our plan going forward is to allow the rebalancing on the original cluster to finish, and then do an upgrade of clients, MON and OSDs, before reintroducing the new OSDs. The question is why the RBD client is acting up, while normal object store operations seems to be unaffected.

#5 Updated by Bosse Klykken almost 8 years ago

Bosse Klykken wrote:

Bosse Klykken wrote:

Our RBD mounts are now working again

Actually, no. We were able to mount the RBD, but it seems to lock after some time.

Update: We regained full access to the RBDs after the cluster had completely finished the backfills after removing the 0.94.6 OSDs. We haven't previously experienced locks like this during normal rebalance operations.

#6 Updated by Bosse Klykken almost 8 years ago

Attempting to add new OSDs reintroduced the problem. Reason seems to be straw2. Problem disappeared when I changed the new hosts' CRUSH algorithm from straw2 to straw manually.

How can it be that new hosts get set up with straw2 with these tunables?

# ceph osd crush show-tunables
{
    "choose_local_tries": 0,
    "choose_local_fallback_tries": 0,
    "choose_total_tries": 50,
    "chooseleaf_descend_once": 1,
    "chooseleaf_vary_r": 1,
    "straw_calc_version": 1,
    "allowed_bucket_algs": 54,
    "profile": "hammer",
    "optimal_tunables": 0,
    "legacy_tunables": 0,
    "require_feature_tunables": 1,
    "require_feature_tunables2": 1,
    "require_feature_tunables3": 1,
    "has_v2_rules": 1,
    "has_v3_rules": 0,
    "has_v4_buckets": 0
}

#7 Updated by Greg Farnum almost 7 years ago

  • Status changed from New to Won't Fix

Hammer's done.

Also available in: Atom PDF