Ceph » RADOS

Do you have a reproducer for this?

I get nervous when people want to remove mark_down calls, as they are generally quite important to maintain correctness. In this case, the replica should not be getting any incoming messages from the primary, but it also shouldn't be allowing a new incoming Connection — that's probably the part of the process that is breaking. (And maybe something should be sharing the missing osdmap? I think we try and do that as well.)

Greg Farnum wrote:

Do you have a reproducer for this?

I get nervous when people want to remove mark_down calls, as they are generally quite important to maintain correctness. In this case, the replica should not be getting any incoming messages from the primary, but it also shouldn't be allowing a new incoming Connection — that's probably the part of the process that is breaking. (And maybe something should be sharing the missing osdmap? I think we try and do that as well.)

yes，i have reproducer this in my test env many times.
what i think like you before, Primary should not sending to Replica during this period. but
1. Primary does not konw if Replica have know he is down, connection shutdown not mean that Replica konw Primary is shutdown, and Primary need drain remained op.
2. Replica pg has not peering, even if Replica OSD has committed new osdmap, but pg do not kown.
3. i agree with you that maybe something should be sharing the missing osdmap, but i think we don't need to mark_down an inuse connection also.
4. maybe i missunderstand mark_down effect, could you piont out, thankyou.

Sage Weil wrote:

So in step 5, the primary hasn't seen osdmap 20, right? Only the replica has? The part I don't understand is that in step 6, the connection will still be in standby state on the primary, and when the connection get's rebuilt, it should fail to connect/deliver to the replica. That is the intention, at least.. I think that is where the problem is.

The problem with not marking down the peer connection is that we will never clean up the state for these connections and we'll leak memory over the long term. Maybe this could be paired with a mark_down() call when the osd comes back up so that then we replace the old connection, at a minimum?

1. yes, Primary hasn't seen osdmap 20. Replica has seen osdmap 20.
2. why should standby connection rebuild failed? even if Replica has seen osdmap 20, but pg hasn't seen, he has not peering. means that pg has not known Primary is going down.
3. yep, i guessed that why we need to mark down when committed_osdmap found down osd is that there is no other way to reclaim socket。so why not add epoll event EPOLLRDHUP to capture peer socket close event, and then reclaim socket.

Ok, I think I understand. This would noramlly trigger a RESETSESSION in the v1 protocol because the primary's connect_seq > 0 and the replica has no record of the session (it did mark_down). In v2 the reconnects are more explicit and generally safer. However, I'm not sure we have the correct behavior there yet... what we really want is for the replica to never accept that (re)connection, but I suspect we might be retrying.

In any case, can you add a reproducer for this scenario to ceph_test_msgr? see src/test/msgr/test_msgr.cc. That will make it easy for us to verify we are resolving the problem... especially in the v2 protocol. I have a feeling v1 will be harder to fix, but I think it can be done too.

Thanks!