Ceph » Messengers

Probably a bug I and Guang introduced in commit:bdee119076dd0eb65334840d141ccdf06091e3c9, but perhaps not?

Okay, so the crashed OSD (osd.5) was marked down by the other endpoint (osd.4). Then osd.5 reconnected to osd.4, which for some reason allowed it. (osd.4 marked the connection down again as soon as it got a message delivered.)
osd.4 sent back an "existing_seq" of 0 as part of the final connection reply, which means that it had already been set to 0 in the existing Pipe's in_seq member. Which only happens if accept() detects "is_reset_from_peer" by comparing the provided and existing connect_seq (here, we get the connect_seq value we expect from a not-reset peer).

So it looks like osd.4 called mark_down on the addr-based interface and nuked the Pipe from handle_osd_map()...but then it sent a pg_info out along the Connection interface from a PG. Since it wasn't a lazy send_message(), that starts a new Pipe in connect().
Meanwhile, osd.5 tried to reconnect, and so osd.4's accept() races with its connect(), resulting in the behavior we're witnessing. That's definitely not supposed to happen; get_con_osd_cluster() uses get_nextmap_reserved() to make sure it's up-to-date. It could happen if we get that Connection (thereby dropping the map reservation), and then the Connection is marked down, and then we use it to send a Message — so that may be something we need to fix. But in this case, it doesn't look like that's what happened, because an earlier Message in the same thread appears to have been dropped from send_message_osd_cluster() due to the map marking the OSD out.
So I'm still trying to understand what's actually happened here; I'm going to check out the other failure and see if I can find anything else there.

The one I originally reported is much simpler:
osd.0 got marked down
osd.1 called mark_down and then forgot about it
osd.0 sent another message, opening a session
osd.1 said "oh, a new connection" and opened it up (with CEPH_MSGR_TAG_READY)
osd.0 kept around its old state, because it didn't get a RESET of any kind.

So I think we have two issues:
1) The OSD is (somehow) going backwards in time in terms of who it wants to connect to
2) the use of (mid-2012) policy.resetcheck is preventing us from exposing necessary information about when connections are actually going away. I want to discuss with Sam and Sage on these, respectively.

Greg Farnum wrote:

So it looks like osd.4 called mark_down on the addr-based interface and nuked the Pipe from handle_osd_map()...but then it sent a pg_info out along the Connection interface from a PG. Since it wasn't a lazy send_message(), that starts a new Pipe in connect().
Meanwhile, osd.5 tried to reconnect, and so osd.4's accept() races with its connect(), resulting in the behavior we're witnessing. That's definitely not supposed to happen; get_con_osd_cluster() uses get_nextmap_reserved() to make sure it's up-to-date. It could happen if we get that Connection (thereby dropping the map reservation), and then the Connection is marked down, and then we use it to send a Message — so that may be something we need to fix.

Realized in discussion with Sage that this can't happen — if the Connection still exists, it's marked as "failed" by the mark_down() call.

OSD bug:
1) In handle_osd_map(), we need to wait until nobody is using a (previously-)reserved map prior to doing any mark_down() calls. I think we can just insert a call to await_reserved_maps() here, but need to make sure.
2) The interface around OSDService::get_con_osd_cluster() is basically broken; we need a reserved map to stick around for as long as the ConnectionRef does. We can probably pull an RAII trick here...?

Messenger bug:
There is still an issue, illustrated in my original set of logs:
osd.a marks down osd.b and forgets about it (by not having any outstanding refs to the Connection)
osd.b sends a new message to osd.a, and attempts to reopen the Connection
osd.a doesn't have anything about osd.b, but happily echoes back the global and connect seqs it is given
osd.b crashes on the message seq values from osd.a, never having had a way to find out its Connection was reset on the other side :(

get_con_osd_cluster might be okay, since a marked_down Connection won't send any outgoing messages, which is what we want. The problem is doing the Connection lookups and having them race with a mark_down. (send_message_osd_cluster is broken in the same way.)

Of course the lookup is supposed to be protected by the map reservation system, but since we're doing mark_down prior to waiting for map reservations we don't have the guarantees we want. (That's just problem 1, making the rest of the stuff look busted.)