make PG state names intelligible
A question came up on the mailing list about whether PGs go degraded on CRUSH map changes, and how to distinguish that from actually loss of redundancy. A reply came in:
The PG just counts as degraded in both scenarios, but if you look at the objects in the degraded PGs (visible in ceph status) some of them are degraded objects and others are misplaced objects. Degraded objects have less than your replica size of copies, like what happens when you lose an OSD. When you change the CRUSH map you get misplaced objects. The PG will have which OSDs it will end up on and the acting OSDs that it's currently on. The acting OSDs will remain the authority for the PG until the new OSDs have a full up to date copy of the PG. At that point the old OSDs will clean up their copy. You should always be serving from your full replica size of OSDs when adding storage and making changes to your CRUSH map.
I did not think it was correct. It is. At a quick skim, we go PG_STATE_DEGRADED whenever it is not located in the correct places, or any of the replicas have missing objects. We additionally go PG_STATE_UNDERSIZED if there are not enough replicas. There is no explicit relationship between those and PG_STATE_REMAPPED.
So, not only is there no clear relationship between the possible states, they are different than the representations we use for reporting degraded versus misplaced objects! We should clear that up so it makes some kind of coherent sense.