Ceph

Recently there has been a big problem in our production ceph
cluster.It has been running very well for one and a half years.

RBD client network and ceph public network are different,
communicating through a router.

Our ceph version is 0.94.5. Our IO transport is using Simple Messanger.

Yesterday some of our VM (using qemu librbd) can not send IO to ceph cluster.

Ceph status is healthy and no osd up/down and no pg inactive and down.

When we export an rbd image through rbd export ,we find the rbd client
can not connect to one osd just to say osd.34.

We find thant osd.34 up and running ,but in the log we find some
errors as follows:
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.

We find that our max open files is set to 200000, but filestore fd
cache size is too big like 500000.
I think we have some wrong fd configurations.But when there are some
errors in Accepter::entry() ,it's better to assert the osd process so
that new rbd client can connect to the ceph cluster and when there
are some network probem, the old rbd client can also reconnect to the
cluster.