Ceph

I see from the code that there are a couple of scenarios that need to be handled or at least documented:

1. Export of a replicated pg from an OSD that had transitioned to shard handling, then import onto an OSD that hadn't

This should just be allowed. It doesn't matter if source compat_features includes SHARDS feature

2. Export of a sharded pg (OSD must be sharded), and then attempt to import onto an OSD that hadn't transitioned

The tool would have to properly transition the OSD by updating its super block and doing a set_allow_sharded_objects() on the objectstore

3. Export of a sharded pg (OSD must be sharded), and then import onto an earlier release that didn't support sharded objects.

The import should fail because the feature bit is set. The tool added feature bit checking and was backported to earlier releases that didn't supported sharded pgs.

4. Export of a replicated pg where OSD happens have transitioned to sharded, and then import onto an earlier release that didn't support sharded objects

This should be made to work.

We can fix #1 and #4 by removing the CEPH_OSD_FEATURE_INCOMPAT_SHARDS feature bit from the compat_features in the export when exporting a replicated pg.

For #3 leaving the feature bit will cause earlier release to fail an import as long as it doesn't have sharded pools

For #2 maybe now is the time to add the capability to transition to sharded mode in the tool.

How do we tell the difference between (2) and (3)? In both cases, ceph_objectstore_tool will see there is no SHARDS in the sb. If the difference is really the version of ceph-osd that was last running (and not anything encoded on disk), maybe it could look at the store's newest OSDMap to see if the ERASURE_CODE feature is set?

I'm not sure if that amount of smarts is needed, though. It might be simpler to add an command line argument --set-allow-sharded-objects, perhaps accompanied by a warning (on the usage or man page), and be done with it. Then the user decides explicitly whether to make the transition or not.