CRUSH: override 'crush location' in the ceph config so that modifications to OSD position in the CRUSH map stick
#4 Updated by Christina Meno over 6 years ago
I cannot figure out what was intended when I created this ticket.
osd host = osd-host-a
osd host = osd-host-b
That doesn't make sense because it seems like osd id is chosen at creation time. ceph-deploy/ceph-disk don't seem to honor it.
#5 Updated by John Spray over 6 years ago
In normal operation, there are two sources of modifications to the crush map:
- A when the admin modifies the crush map using the ceph CLI
- B when each OSD starts up, it invokes a script hook to update its location http://ceph.com/docs/master/rados/operations/crush-map/#ceph-crush-location-hook
A is what Calamari currently does. B is designed to make OSDs somewhat hot pluggable between hosts.
The trouble is that if A is doing something clever like assigning an SSD into a different CRUSH root than the default, then B will come and trample all over it the next time the OSD starts. This ticket is about solving that problem.
The crush_location setting is a way of telling the B path (ceph-crush-location) "Instead of inferring your location from $hostname, set it to this".
#6 Updated by Christina Meno over 6 years ago
So the way I see it there are several ways to change this:
A. set the value to false.
B. set the value to a calamari-specific script
1. in the package post-hook type scripts:
- on install of calamari-server set A or B
- on removal of calamari-server restore to previous value, or do nothing
2. when cthulhu detects that the api is making a change to the crush map that requires this and the setting is not false
- it sets A or B
3. we helpfully explain all the details and allow the user to choose.
What would the calamari-specific script do?
It could just read the crush map and offer a startup stasis
I recommend going with 1A it is the simple solution, it is least surprising, and requires the least effort.
#7 Updated by Christina Meno over 6 years ago
- Status changed from New to Need More Info
after talking to Dan the solution that seems best is:
Have calamari set "osd crush location hook" to a script that asks either calamari or the cluster for the OSDs last known location in the CRUSH map if this is a new OSD fallback to a sensible default e.g. the behavior as it "osd update on start" were true
The thing I like most about this approach is that we edit the config file one time.