Ceph

That's a good summary; see PG::may_need_replay(). I glanced over this and it looks like the PG is placed on the OSD's "replay_queue" when it activates (which is when it gets put in the REPLAY state), and the replay_queue is processed every time the OSD ticks by calling replay_queued_ops() on any PGs which have finished that state.
And replay_queued_ops() calls publish_stats_to_osd(), which sets the stats up to get published to the Monitor. Which nixes my first thought, that the replay PGs simply weren't exiting that state and telling the monitors about it if there were no incoming ops. :/
But there still could be something more subtle going on.

1. will do
2. my concern is that the replay bit appears to be lost because of a restart. say, osds 0, 1 and 2 (all holding PG 0.0) are restarted. they repeer, and the status of 0.0 gets stuck in active+replay+degraded. Then, I restart osd2, and after repeering, it gets all the way to active+degraded, without having ever given any indication that the replay required after the prior recovery was done. Indeed, it matters little whether it actually did or did not do the replay: it doesn't look like the complex conditions that trigger a replay would carry over an earlier and still pending need for replay onto the subsequent recovery.

Hummm, it looks like you're right: we start a replay period (which is time-based, 30s by default) as we go active and fill in info.history.last_epoch_started...which means that if the primary goes away during the replay period, the next one (possibly the same) won't go back to replay when it finishes the next peering round thanks to that check.
At least, if I'm reading all the interval checks correctly in this function. Which I might not be.

ceph pg <PGid> query shows PGs stuck in active+replay+degraded state have that set in their primary state. Replicas, OTOH, are shown as either peering (still?), or as active+degraded. Some PGs had one replica in each state. Is this the info you were looking for? What should I look for otherwise?

As for the issue of losing replay states upon member osd restarts... Could the fix be as simple as not setting interval.maybe_went_rw when the state of the PG has the REPLAY bit set? Or do we need something far more elaborate, to account for the fact that we clear the REPLAY bit before we even get started replaying the messages, and we might propagate this state before all messages are fully processed?

Hi

Is this Fix merged to the master code base? why I still found the active+replay+clean state in my cluster with the version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)?And I found the fix is also not backported to the firefly and dumpling version.

Thanks!