Project

General

Profile

Bug #23649

[simple/msg]Add heartbeat timeout beforeAccepter::entry break out for osd thread

Added by 相洋 于 8 months ago. Updated 16 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
common
Target version:
-
Start date:
04/11/2018
Due date:
04/11/2018
% Done:

0%

Source:
Community (user)
Tags:
Backport:
luminous, mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:

Description

Recently there has been a big problem in our production ceph
cluster.It has been running very well for one and a half years.

RBD client network and ceph public network are different,
communicating through a router.

Our ceph version is 0.94.5. Our IO transport is using Simple Messanger.

Yesterday some of our VM (using qemu librbd) can not send IO to ceph cluster.

Ceph status is healthy and no osd up/down and no pg inactive and down.

When we export an rbd image through rbd export ,we find the rbd client
can not connect to one osd just to say osd.34.

We find thant osd.34 up and running ,but in the log we find some
errors as follows:
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.
accepter no incoming connection? sd =-1 ,errer 24, too many open files.

We find that our max open files is set to 200000, but filestore fd
cache size is too big like 500000.
I think we have some wrong fd configurations.But when there are some
errors in Accepter::entry() ,it's better to assert the osd process so
that new rbd client can connect to the ceph cluster and when there
are some network probem, the old rbd client can also reconnect to the
cluster.


Related issues

Copied to Ceph - Backport #36157: luminous: [simple/msg]Add heartbeat timeout beforeAccepter::entry break out for osd thread Resolved
Copied to Ceph - Backport #36219: mimic: [simple/msg]Add heartbeat timeout beforeAccepter::entry break out for osd thread Resolved

History

#1 Updated by 相洋 于 8 months ago

#2 Updated by Kefu Chai 8 months ago

  • Status changed from New to Need Review
  • Assignee set to 相洋 于
  • Target version deleted (v12.2.5)
  • Backport set to jewel,luminous

#4 Updated by 相洋 于 7 months ago

相洋 于 wrote:

New PR:

https://github.com/ceph/ceph/pull/22056/

Ignore this message.

#6 Updated by Kefu Chai 3 months ago

  • Status changed from Need Review to Pending Backport

#7 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #36157: luminous: [simple/msg]Add heartbeat timeout beforeAccepter::entry break out for osd thread added

#9 Updated by Greg Farnum 3 months ago

  • Backport changed from jewel,luminous to jewel,luminous, mimic

We certainly need to put this in Mimic if we're going to backport it at all!

Actually we might want to put it there first and let it bake a bit before backporting to luminous or jewel in case we discover any issues.

#10 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #36219: mimic: [simple/msg]Add heartbeat timeout beforeAccepter::entry break out for osd thread added

#11 Updated by Nathan Cutler about 1 month ago

  • Backport changed from jewel,luminous, mimic to luminous, mimic

#12 Updated by Nathan Cutler about 1 month ago

Jewel is EOL

#13 Updated by Nathan Cutler 16 days ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF