Ceph » CephFS

When using Ceph (python-cephfs-10.2.3+git.1475228057.755cf99 , SLE12SP2) together with OpenStack Manila and trying to allow share access, the manila-share service fails with:

2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server [req-ea32cd1e-ceb8-4c04-a237-ad0df3a3885f 7804d3df369a40d3a8d60acc1b970f91 f648a37bd19b4a9c9b32a28f3a2d2452 - - -] Exception during message handling
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 163, in wrapped
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/utils.py", line 617, in wrapper
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 2572, in allow_access
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     share_server=share_server
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 60, in update_access_rules
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     share_server=share_server,
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 53, in _update_access_rules_locked
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     return self._update_access_rules(*args, **kwargs)
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 112, in _update_access_rules
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     share_server=share_server
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/cephfs_native.py", line 288, in update_access
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     access_key = self._allow_access(context, share, rule)
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/cephfs_native.py", line 237, in _allow_access
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     tenant_id=share['project_id'])
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 974, in authorize
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     key = self._authorize_volume(volume_path, auth_id, readonly)
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 1010, in _authorize_volume
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     key = self._authorize_ceph(volume_path, auth_id, readonly)
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 1087, in _authorize_ceph
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     'mon', cap['caps'].get('mon')]
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 1278, in _rados_command
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server     raise rados.Error(outs)
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server Error: Can't handle arrays of non-strings
2016-11-04 13:29:03.514 16317 ERROR oslo_messaging.rpc.server 

Printing the sent json with some debug messages added to the code:

{'format': 'json', 'entity': 'client.alice', 'caps': ['mds', ',allow rw path=/volumes/_nogroup/dba70486-fc31-4
f83-a7dd-e391a00d759c', 'osd', ',allow rw pool=cephfs_data namespace=fsvolumens_dba70486-fc31-4f83-a7dd-e391a00d759c', 'mon', None]}

So the problem is that the mon caps are None. The problematic code is: https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L1087