Ceph

client rbd (firefly) --with setallochint--> primary (firefly) --with setallochint--> replica (dumpling) crash

The problem here appears to be that the user upgraded the clients before the osds were fully upgraded. librbd sends the setallochint unconditionally, old osds will respond with ENOTSUPP. The bug here would be that the primary supported the op and the replicas didn't. It probably should have returned ENOTSUPP.

What happens if

• all OSDs in a PG support setallochint
• one secondary OSD goes down
• the secondary is replaced by an OSD that does not support setallochint

On any change of pg configuration peering happens, so a new collection of feature bits from the peers is collected. If not all peers support the feature, EOPNOTSUPP is returned to client and no messages are sent to any secondaries.

Thanks for explaining. Since alloc hint is optional it does not matter if it is activated and deactivate later.

Notes on using feature bits already present. The problem is that CEPH_FEATURE_MSGR_KEEPALIVE2 was back ported, so we'd have to check CEPH_FEATURE_OSD_POOLRESEND but that is over 2 months worth of changes later. For maintainability I'd rather have a feature bit dedicated to the feature being checked for.

f825624f (Sage Weil 2014-01-29 19:47:21 -0800 125) CEPH_FEATURE_OSD_PRIMARY_AFFINITY
64568023 (Ilya Dryomov 2014-02-21 16:34:13 +0200 Added HINT CODE (v0.78)
d747d79f (Sage Weil 2014-03-27 21:09:13 -0700 126) CEPH_FEATURE_MSGR_KEEPALIVE2 (v0.79)
45e79a17 (Sage Weil 2014-05-08 10:50:51 -0700 54) CEPH_FEATURE_OSD_POOLRESEND (v0.81)

next step is to add a tests for this to the upgrade suties.