Ceph » RADOS

My plan current script to setup a vstart to test out the above hypothesis:

MON=3 MDS=2 MGR=3 OSD=6 ../src/vstart.sh -n -x -l -d --without-dashboard

NUM_OSDS_UP=$(./bin/ceph osd df | grep "up" | wc -l)

if [ $NUM_OSDS_UP -lt 6 ]; then
    echo "test requires at least 6 OSDs up and running" 
    exit 1
fi

for dc in dc1 dc2 dc3
    do
      ./bin/ceph osd crush add-bucket $dc datacenter
      ./bin/ceph osd crush move $dc root=default
    done

./bin/ceph osd crush add-bucket node-1 host
./bin/ceph osd crush add-bucket node-2 host
./bin/ceph osd crush add-bucket node-3 host
./bin/ceph osd crush add-bucket node-4 host
./bin/ceph osd crush add-bucket node-5 host
./bin/ceph osd crush add-bucket node-6 host

./bin/ceph osd crush move node-1 datacenter=dc1
./bin/ceph osd crush move node-2 datacenter=dc1
./bin/ceph osd crush move node-3 datacenter=dc2
./bin/ceph osd crush move node-4 datacenter=dc2
./bin/ceph osd crush move node-5 datacenter=dc3
./bin/ceph osd crush move node-6 datacenter=dc3

./bin/ceph osd crush move osd.0 host=node-1
./bin/ceph osd crush move osd.1 host=node-2
./bin/ceph osd crush move osd.2 host=node-3
./bin/ceph osd crush move osd.3 host=node-4
./bin/ceph osd crush move osd.4 host=node-5
./bin/ceph osd crush move osd.5 host=node-6

./bin/ceph mon set_location a datacenter=dc1 host=node-1
./bin/ceph mon set_location b datacenter=dc2 host=node-3
./bin/ceph mon set_location c datacenter=dc3 host=node-5

./bin/ceph osd getcrushmap > crushmap || return 1
./bin/crushtool --decompile crushmap > crushmap.txt || return 1
sed 's/^# end crush map$//' crushmap.txt > crushmap_modified.txt || return 1
cat >> crushmap_modified.txt << EOF
rule replicated_rule_custom {
        id 1
        type replicated
        step take dc1
        step chooseleaf firstn 2 type host
        step emit
        step take dc2
        step chooseleaf firstn 2 type host
        step emit
        step take dc3
        step chooseleaf firstn 2 type host
        step emit
}
# end crush map
EOF

./bin/crushtool --compile crushmap_modified.txt -o crushmap.bin || return 1
./bin/ceph osd setcrushmap -i crushmap.bin  || return 1

poolname=test_pool
./bin/ceph osd pool create $poolname 32 32 replicated_rule_custom  || return 1
./bin/ceph osd pool set $poolname size 6 || return 1
./bin/ceph osd pool set $poolname min_size 2 || return 1

Don't forget that there is also pg_pool_t::peering_crush_bucket_count that directly requires a minimum number of higher-level CRUSH buckets (eg, data centers) in order to go active. We need this available as well.

peering_crush_bucket_[count|target|barrier]

WIP PR: https://github.com/ceph/ceph/pull/56233

I recently created a draft PR https://github.com/ceph/ceph/pull/56233/, adding the additional arguments peering_bucket_count and peering_crush_bucket_barrier to the pool creation command.
The reason I added only 2 arguments is because after auditing the code I found that peering_crush_bucket_count and peering_crush_bucket_target shares the same value which is stretch_bucket_count.

Here are the two places:

OSDMonitor::prepare_new_pool()
  if (osdmap.stretch_mode_enabled) {
    pi->peering_crush_bucket_count = osdmap.stretch_bucket_count;
    pi->peering_crush_bucket_target = osdmap.stretch_bucket_count;
    pi->peering_crush_bucket_barrier = osdmap.stretch_mode_bucket;
    pi->peering_crush_mandatory_member = CRUSH_ITEM_NONE;
    if (osdmap.degraded_stretch_mode) {
      pi->peering_crush_bucket_count = osdmap.degraded_stretch_mode;
      pi->peering_crush_bucket_target = osdmap.degraded_stretch_mode;
      // pi->peering_crush_bucket_mandatory_member = CRUSH_ITEM_NONE;
      // TODO: drat, we don't record this ^ anywhere, though given that it
      // necessarily won't exist elsewhere it likely doesn't matter
      pi->min_size = pi->min_size / 2;
      pi->size = pi->size / 2; // only support 2 zones now
    }
  }

OSDMonitor::try_enable_stretch_mode()
  if (commit) {
    for (auto pool : pools) {
      pool->crush_rule = new_rule;
      pool->peering_crush_bucket_count = bucket_count;
      pool->peering_crush_bucket_target = bucket_count;
      pool->peering_crush_bucket_barrier = dividing_id;
      pool->peering_crush_mandatory_member = CRUSH_ITEM_NONE;
      pool->size = g_conf().get_val<uint64_t>("mon_stretch_pool_size");
      pool->min_size = g_conf().get_val<uint64_t>("mon_stretch_pool_min_size");
    }
    pending_inc.change_stretch_mode = true;
    pending_inc.stretch_mode_enabled = true;
    pending_inc.new_stretch_bucket_count = bucket_count;
    pending_inc.new_degraded_stretch_mode = 0;
    pending_inc.new_stretch_mode_bucket = dividing_id;

I’m in the process of testing locally to see if there are other areas I need to tweak in order to decouple the capability of handling pg_temp. Will have more results on Monday

Okay so the latest change that I added will have two commands:

ceph osd pool stretch set <pool> <peering_crush_bucket_count> <peering_crush_bucket_target> <peering_crush_bucket_barrier> <crush_rule> <size> <min_size>
and
ceph osd pool stretch show <pool>

example output:

pool: cephfs.a.data
pool_id: 3
is_stretch_pool: 1
peering_crush_bucket_count: 3
peering_crush_bucket_target: 2
peering_crush_bucket_barrier: datacenter
crush_rule: replicated_rule_custom
size: 6
min_size: 3

1.All arguments must be present
2.All arguments must have the correct type (Mostly handled in MonCommands.h)
3.Non existence pool should return appropriate error
4.count|target must be > 0
5.Can’t have a count|target > whatever actual count of the type barrier found in the crush map.
4.Non existence barrier should return appropriate error
5.If the specified crush_rule doesn’t have the same type as the pool, e.g., replicated != ec_pool, should return appropriate error.

I tested and verified that all the cases are handled correctly.
Also, the reason I did not include <peering_crush_bucket_count> <peering_crush_bucket_target> <peering_crush_bucket_barrier> to ceph osd pool create is because the command doesn’t require all arguments to be present, also there is no argument formin_size which is required, so it’s harder to enforce the integrity of the feature. Therefore, for now we rely on setting these values after pool creation.

Moreover, I also simulate the case Sam described in https://tracker.ceph.com/issues/64802
So I set up MON=9 MDS=1 MGR=1 OSD=9

rule replicated_rule_custom {
       id 1
       type replicated
       step take default
       step choose firstn 3 type datacenter
       step chooseleaf firstn 2 type host
       step emit
}

ID   CLASS  WEIGHT   TYPE NAME            STATUS  REWEIGHT  PRI-AFF
-1         0.78868  root default
-5         0.29575      datacenter dc1
-11         0.09859          host node-1
 0    hdd  0.09859              osd.0        up   1.00000  1.00000
-12         0.09859          host node-2
 1    hdd  0.09859              osd.1        up   1.00000  1.00000
-13         0.09859          host node-3
 2    hdd  0.09859              osd.2        up   1.00000  1.00000
-7         0.29576      datacenter dc2
-14         0.09859          host node-4
 3    hdd  0.09859              osd.3        up   1.00000  1.00000
-15         0.09859          host node-5
 4    hdd  0.09859              osd.4        up   1.00000  1.00000
-16         0.09859          host node-6
 5    hdd  0.09859              osd.5        up   1.00000  1.00000
-9         0.19717      datacenter dc3
-17   hdd  0.09859          host node-7
6           0.09859              osd.6       up   1.00000  1.00000
-18         0.09859         host node-8
 7    hdd  0.09859              osd.7        up   1.00000  1.00000
-19        0.09859          host node-9
 8    hdd  0.09859              osd.8        up   1.00000  1.00000

Pools:
cephfs.a.meta
cephfs.a.meta
.mgr

./bin/ceph osd pool stretch set cephfs.a.meta 2 3 datacenter replicated_rule_custom 6 3
./bin/ceph osd pool stretch set cephfs.a.data 2 3 datacenter replicated_rule_custom 6 3
./bin/ceph osd pool stretch set .mgr 2 3 datacenter replicated_rule_custom 6 3

Just created final revision: https://github.com/ceph/ceph/pull/56233

waiting for review