Project

General

Profile

Actions
Copy link

Bug #19643

closed

mgr: stops sending beacon, stops handling incoming messages

Added by Sage Weil about 7 years ago. Updated almost 7 years ago.

Status:
Can't reproduce
Priority:
Immediate
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

the last thing from the dispatch thread

2017-04-18 00:54:47.558139 7f365e5b4700  1 -- 172.21.15.47:6813/24275 <== osd.0 172.21.15.137:6800/20183 1 ==== mgropen(0) v1 ==== 5+0+0 (150087197 0 0) 0x557d30e0ba40 con 0x557d307a1a00
2017-04-18 00:54:47.558149 7f365e5b4700  4 mgr.server handle_open from 0x557d307a1a00 name 0
2017-04-18 00:54:47.558150 7f365e5b4700  1 -- 172.21.15.47:6813/24275 --> 172.21.15.137:6800/20183 -- mgrconfigure() v1 -- ?+0 0x557d30e0d640 con 0x557d307a1a00
2017-04-18 00:54:47.558161 7f365e5b4700 20 mgr.server handle_open updating existing DaemonState for 0

and shortly after,
2017-04-18 00:55:11.251309 7f36605b8700  1 -- 172.21.15.47:0/2261402931 --> 172.21.15.137:6793/0 -- mgrbeacon mgr.x(d526337e-d534-43ec-becf-405adc860759,4110, 172.21.15.47:6813/24275, 1) v2 -- 0x557d30b9ef40 con 0
2017-04-18 00:55:16.249359 7f3662dbd700 10 client.4110.objecter tick
2017-04-18 00:55:16.251449 7f36605b8700  1 mgr send_beacon active
2017-04-18 00:55:16.251466 7f36605b8700 10 mgr send_beacon sending beacon as gid 4110
2017-04-18 00:55:16.251504 7f36605b8700  1 -- 172.21.15.47:0/2261402931 --> 172.21.15.137:6793/0 -- mgrbeacon mgr.x(d526337e-d534-43ec-becf-405adc860759,4110, 172.21.15.47:6813/24275, 1) v2 -- 0x557d3065cac0 con 0
2017-04-18 00:55:17.700781 7f36615ba700 10 cephx: validate_tickets want 55 have 55 need 0
2017-04-18 00:55:17.700785 7f36615ba700 20 cephx client: need_tickets: want=55 have=55 need=0
2017-04-18 00:55:17.700796 7f36615ba700 10 auth: dump_rotating:
2017-04-18 00:55:17.700797 7f36615ba700 10 auth:  id 1 AQBDYfVYih1MJhAAa6ePNdFRE4EsoCFkDwexXg== expires 2017-04-18 01:43:47.642517
2017-04-18 00:55:17.700814 7f36615ba700 10 auth:  id 2 AQBDYfVYd2dMJhAAjsd0CX+umDf71TDV95BCIA== expires 2017-04-18 02:43:47.642517
2017-04-18 00:55:17.700820 7f36615ba700 10 auth:  id 3 AQBDYfVYBI9MJhAATnyHi/uzSjl043iWm7R/nw== expires 2017-04-18 03:43:47.642517
2017-04-18 00:55:17.700835 7f36615ba700  1 -- 172.21.15.47:0/2261402931 >> 172.21.15.137:6793/0 conn(0x557d30879000 :-1 s=STATE_OPEN pgs=283 cs=1 l=1).mark_down
2017-04-18 00:55:21.249447 7f3662dbd700 10 client.4110.objecter tick
2017-04-18 00:55:21.251638 7f36605b8700  1 mgr send_beacon active
2017-04-18 00:55:26.249582 7f3662dbd700 10 client.4110.objecter tick
2017-04-18 00:55:31.249727 7f3662dbd700 10 client.4110.objecter tick
2017-04-18 00:55:36.249898 7f3662dbd700 10 client.4110.objecter tick
2017-04-18 00:55:41.250068 7f3662dbd700 10 client.4110.objecter tick
2017-04-18 00:55:46.250220 7f3662dbd700 10 client.4110.objecter tick
2017-04-18 00:55:51.250335 7f3662dbd700 10 client.4110.objecter tick
...

/a/sage-2017-04-17_18:48:43-rados-wip-past-intervals---basic-smithi/1036638

the mon removes the active mgr, and then a teuthology 'pg dump' hangs and times out.

no core file unfortunately.

Actions
Copy link
 #1

Updated by Sage Weil almost 7 years ago

  • Status changed from New to Can't reproduce

haven't seen this in 2 weeks; am guessing it was resolved.

Actions
Copy link

Also available in: Atom PDF