Ceph » RADOS

Background:

The need for fencing in a kubernetes multicluster scenario is presented here: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/3FTOBHPZQGTEY3RHMO2EXKOLSP3SJGNW/

During the discussion, a more generic form of blocklisting clients en-masse was also discussed. This would enable multiple entity names being blocklisted based on an identifying label.

Also, in the ceph-mgr case, the ability to unblock a blocked entity or set of entities by label was discussed in the Ceph CDM Dec, 2nd, 2020 (https://tracker.ceph.com/projects/ceph/wiki/CDM_02-DEC-2020). This was conceptualized as blocklisting entities prior to a generation number, and hence a newer connection by the same entity with a higher generation number is not blocked.

Summary:

The requirement is broken into 3 parts, with part (1) ideally available first/sooner for use, and parts (2) and (3) as extensions in the future.

1) Require ability to blocklist/unblocklist an entity name and its currently valid tickets

This would ensure that the entity name blocked cannot perform IO operations, and is effectively fenced from the cluster till it is unblocked.

It would also be desirable for the block/unblock operation to accept a glob, to enable operating against a pattern based entity naming scheme and hence would support blocklisting/unblocklisting for a group of entities using the same construct.

2) Require additional entity labels that can help group a set of entities for related operations. In particular to blocklist/unblocklist a group of entities by label.

This would help block entities en-masse when there is more than one entity in use that needs fencing from a client cluster. An example would be a client cluster using different entities for CephFS and RBD operations, but can be grouped by a client cluster label and fenced in one go.

Labels and grouping, would enable going beyond a glob to identify a set of entities that are not named similarly.

3) Require the ability to block the current and older sessions for an entity, thus allowing the entity to unblock itself by opening a newer session

This need comes from the fact that a process (the ceph-mgr failover scenario in this case), can recover from stale sessions when it is no longer the primary. IOW, an entity can unblocklist itself.

As discussed in the CDM call, this can be done creating newer tickets and with a newer generation number (or even reuse the "global_id" as the generation), thus enabling a special blocklist that would either blocklist an entity (and it's outstanding valid tickets) or a group of entities based on label till a defined generation.