Actions
Bug #22114
closedmon: ops get stuck in "resend forwarded message to leader"
% Done:
0%
Source:
Tags:
low-hanging-fruit
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Monitor
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
some ops get stuck in "resend forwarded message to leader" for days.
{ "description": "osd_pg_created(1.7d2)", "initiated_at": "2017-11-07 16:51:41.018915", "age": 494016.079968, "duration": 494016.100409, "type_data": { "events": [ { "time": "2017-11-07 16:51:41.018915", "event": "initiated" }, { "time": "2017-11-07 16:51:41.018915", "event": "mon:_ms_dispatch" }, { "time": "2017-11-07 16:51:41.018916", "event": "mon:dispatch_op" }, { "time": "2017-11-07 16:51:41.018916", "event": "psvc:dispatch" }, { "time": "2017-11-07 16:51:41.018916", "event": "osdmap:preprocess_query" }, { "time": "2017-11-07 16:51:41.018916", "event": "osdmap:preprocess_pg_created" }, { "time": "2017-11-07 16:51:41.018916", "event": "forward_request_leader" }, { "time": "2017-11-07 16:51:41.018920", "event": "forwarded" }, { "time": "2017-11-10 14:40:38.489281", "event": "resend forwarded message to leader" }, { "time": "2017-11-10 15:25:34.712647", "event": "resend forwarded message to leader" }, { "time": "2017-11-10 15:49:44.912601", "event": "resend forwarded message to leader" }, { "time": "2017-11-10 15:49:47.615925", "event": "resend forwarded message to leader" }, { "time": "2017-11-10 17:01:26.958438", "event": "resend forwarded message to leader" }, { "time": "2017-11-10 17:43:39.204785", "event": "resend forwarded message to leader" }, { "time": "2017-11-10 17:46:24.955425", "event": "resend forwarded message to leader" }, { "time": "2017-11-10 18:08:14.914648", "event": "resend forwarded message to leader" }, { "time": "2017-11-10 18:19:58.752120", "event": "resend forwarded message to leader" }, { "time": "2017-11-10 18:23:31.995804", "event": "resend forwarded message to leader" }, { "time": "2017-11-10 18:47:32.106938",
And the memory of mon daemon grew huge
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3731 root 20 0 13.194g *9.998g* 8204 S 0.7 34.9 194:05.72 ceph-mon
Actions