Ceph

Change default filestore 'omap' backend to 'rocksdb' from 'leveldb'

This is the option: "filestore_omap_backend":

Version-Release number of selected component (if applicable):
Red Hat Ceph Storage 2.3/3.0
Upstream Jewel and Luminous

We have seen a lot of issues when OMAP directories become very large(40G+) and then leveldb compaction takes a lot of time and this cause OSD's to hit suicide timeout because they won't respond when compaction is running.

Rocksdb will help because it uses multi-threading in compaction and it has other benefits also.

This change does not need code work because the option is already implemented. A simple Pull request would be necessary to change the default we can do that but before that, we need to do upstream and downstream QA testing on an average scale so we can check it is fixing this issue.

For this, we would also need package/release engineering team help to package rocksdb library downstream.

Downstream Features:

RHCS 2.y: https://bugzilla.redhat.com/show_bug.cgi?id=1462011
RHCS 3.0: https://bugzilla.redhat.com/show_bug.cgi?id=1462012