Ceph » CephFS

would checking the nature of pools during 'fs new' on the monitor and failing if any of the specified pools (data or metadata) are cache pools work? Or did you have something else in mind?

I mean something like this (although I'm not positive I got all the requirements right):

diff --git a/src/mon/MDSMonitor.cc b/src/mon/MDSMonitor.cc
index fea198c..14fb003 100644
--- a/src/mon/MDSMonitor.cc
+++ b/src/mon/MDSMonitor.cc
@@ -985,6 +985,15 @@ bool MDSMonitor::management_command(
       return true;
     }

+    const pg_pool_t *p = mon->osdmon()->osdmap.get_pg_pool(metadata);
+    assert(p != NULL);
+    if (p->is_tier() && p->cache_mode != pg_pool_t::CACHEMODE_NONE) {
+      r = -EINVAL;
+      ss << "pool '" << metadata_name << "' cannot be a cache pool";
+      return true;
+    }
+
+

Yeah, that's the simple solution. I was also wondering though if we wanted to do something more sophisticated trying to have the OSDMap or similar structures just transparently map from the cache pool to the base pools.

My vote is NAK on this. THis is exactly what I want to do on my cluster and I this is the only way EC can be used for CephFS currently. Why do we want to prevent it?

This conversation is getting split across several mediums, but this shouldn't prevent specifying the use of a base pool which is EC with a cache pool that is replicated. Just the hooking of the filesystem's metadata to a pool which is sort of "ephemeral" like cache pools are.

I lean toward setting data pool to the base pool too. I worry about having to stand up so many guard rails, though. Hopefully we can consolidate the checks into "is_safe_to_remove" and "is_safe_to_remove_tier" type functions?

Yes, that's what I'm hoping as well. That's what _check_remove_pool() is; we'd need to add an equivalent for tiering. And, maybe, have it check when changing the tier type as well... :/

First half here: https://github.com/ceph/ceph/tree/wip-9435 (no handling of tiering updates yet)