Bug #15411
Added new OSDs with 0.94.6 disabled RBD access
0%
Description
We've added two new nodes with 0.94.6, with a total of 60 OSDs, to a running cluster on 0.94.3, accidentally without upgrading the cluster to the newest version before adding the nodes, as is suggested in the documentation. After a short time, the RBD mounts stopped working, and couldn't be remounted. The RBD client logs mentioned stuff like:
libceph: osd44 10.xx.xx.xx:6820 feature set mismatch, my 2b84a842a42 < server's 102b84a842a42, missing 1000000000000
libceph: osd44 10.xx.xx.xx:6820 socket error on read
libceph: corrupt inc osdmap (-22) epoch 76574 off 60 (ffffc9002e58a058 of ffffc9002e58a01c-ffffc9002e58c1bc)
<and a lot of hex dumps filling the logs>
As far as we've been able to dig up, the feature set mismatch refers to CEPH_FEATURE_CRUSH_V4. When looking at the crush map, we noticed that the new OSDs were in buckets set with alg straw2, while the remainder were straw.
We've stopped the newly added OSDs, but the problem still persists:
#rbd map rbd-volume
rbd: sysfs write failed
rbd: map failed: (5) Input/output error
Attached is the compressed ceph report. After we generated this report, we tried removing the newly added OSDs from the crush map, in order to upgrade the cluster before adding them back, but the problem persists - we can't mount RBD volumes, and we get log messages with feature set mismatch as mentioned above.
History
#1 Updated by Loïc Dachary almost 8 years ago
RBD mounts
Could you add more details about how RBD volumes are mounted ? client and version ?
#2 Updated by Bosse Klykken almost 8 years ago
Issue was resolved by also removing the hosts properly from the crushmap.
ceph osd getcrushmap -o /tmp/crushmap crushtool -d /tmp/crushmap -o /tmp/crushmap.txt vim /tmp/crushmap.txt # removing references to hosts with straw2 crushtool -c /tmp/crushmap.txt -o /tmp/crushmap.new ceph osd setcrushmap -i /tmp/crushmap.new
Our RBD mounts are now working again
#3 Updated by Bosse Klykken almost 8 years ago
Loic Dachary wrote:
Could you add more details about how RBD volumes are mounted ? client and version ?
I tried with several versions, the main one being 0.94.5. I tried upgrading my test system to 0.94.6 to see if that would help. I didn't make a note of what version I had before that, but I believe it was 0.94.3.
Example of volume:
size 3072 GB in 786432 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1dffe522ae8944a
format: 2
features: layering
flags:
#4 Updated by Bosse Klykken almost 8 years ago
Bosse Klykken wrote:
Our RBD mounts are now working again
Actually, no. We were able to mount the RBD, but it seems to lock after some time. I also tried to delete a test volume with 'rbd rm volumename' and that hangs as well.
Our plan going forward is to allow the rebalancing on the original cluster to finish, and then do an upgrade of clients, MON and OSDs, before reintroducing the new OSDs. The question is why the RBD client is acting up, while normal object store operations seems to be unaffected.
#5 Updated by Bosse Klykken almost 8 years ago
Bosse Klykken wrote:
Bosse Klykken wrote:
Our RBD mounts are now working again
Actually, no. We were able to mount the RBD, but it seems to lock after some time.
Update: We regained full access to the RBDs after the cluster had completely finished the backfills after removing the 0.94.6 OSDs. We haven't previously experienced locks like this during normal rebalance operations.
#6 Updated by Bosse Klykken almost 8 years ago
Attempting to add new OSDs reintroduced the problem. Reason seems to be straw2. Problem disappeared when I changed the new hosts' CRUSH algorithm from straw2 to straw manually.
How can it be that new hosts get set up with straw2 with these tunables?
# ceph osd crush show-tunables { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 1, "straw_calc_version": 1, "allowed_bucket_algs": 54, "profile": "hammer", "optimal_tunables": 0, "legacy_tunables": 0, "require_feature_tunables": 1, "require_feature_tunables2": 1, "require_feature_tunables3": 1, "has_v2_rules": 1, "has_v3_rules": 0, "has_v4_buckets": 0 }