Ceph

In talking with Sam about firewall rules the other day, he mentioned that the OSD TCP port selection is not really deterministic. If you restart an OSD service soon enough, it could choose a different port.

< ktdreyer> : it sounds like you're saying we should just adjust the documentation to say "open TCP 6800-7300 on each of your OSDs"?
< ktdreyer>: I always wondered what the "right" way to handle this would be, if we were to submit a "ceph-osd" rule definition to the firewalld project.
< sjust>: ktdreyer: I don't really know, that's what sage kind of said
< sjust>: it certainly seems to be the case that the osd can choose any of those ports
< ktdreyer>: so the logic isn't "find the lowest available port after 6800", it's more like "choose a random port in that range" ?
< sjust>: no, it's actually choose the lowest one
< ktdreyer>: oh
< sjust>: but it tries to avoid recently used ports
< sjust>: so it isn't really deterministic or meant to be
< ktdreyer>: how does it know which ports were recently used?
< ktdreyer>: oh ok
< sjust>: osdmap
< sjust>: I think
< sjust>: or maybe that's only when it's marked down
< ktdreyer>: so you could have a single OSD, and it could bind to many different ports, if you restarted it quickly enough
< sjust>: it might be that it rebinds if marked down
< sjust>: one sec
< sjust>: yeah, looks like if it's marked down, it tries to avoid the ports it was on just before
< sjust>: in any case, it's not guarranteed to stick to the lowest N ports
< sjust>: so we probably have to open whatever range is allowed by the default config
< ktdreyer>: yeah, I think that's fair
< ktdreyer>: thanks!
< ktdreyer>: I think we should fix this in the docs upstream and downstream, and get a rule submitted to firewalld for the OSDs that just opens the whole range
< sjust>: yeah
< sjust>: we should confirm with sage and greg

Based on this conversation, it sounds like http://ceph.com/docs/master/rados/configuration/network-config-ref/ needs to change?