Project

General

Profile

Actions
Copy link

Bug #19199

open

Odd OSD failure path; ERROR: osd init failed: (110) Connection timed out

Added by J. Eric Ivancich about 7 years ago. Updated almost 7 years ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Correctness/Safety
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v11.1.0
ceph-qa-suite:
Component(RADOS):
MonClient, OSD
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

See attached OSD log for more details.
commit 6f8e4b38103d6f519e6661acc97a47ceccf5e5fc was the latest master
Intermittent

Call stack:
ceph version 11.1.0-6031-g205bb7e (205bb7ea94e49bb92f22727404c5de3c01ab77ff)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80)
[0x558fdb0640f0]
2: (ceph::HeartbeatMap::~HeartbeatMap()+0xd1) [0x558fdb14df01]
3: (CephContext::~CephContext()+0x428) [0x558fdb134e48]
4: (CephContext::put()+0x173) [0x558fdb1351c3]
5: (main()+0xdf8) [0x558fda7de0a8]
6: (__libc_start_main()+0xf1) [0x7f4b

Files

FAILED.osd.0.log-tail (48.1 KB) FAILED.osd.0.log-tail osd log J. Eric Ivancich, 03/06/2017 06:19 PM
Actions
Copy link
 #1

Updated by J. Eric Ivancich about 7 years ago

  • Affected Versions v11.1.0 added
  • Affected Versions deleted (v10.2.7)
Actions
Copy link
 #2

Updated by Josh Durgin about 7 years ago

  • Priority changed from Normal to Low

Earlier in the log the root cause appears:

-39> 2017-03-05 21:07:12.493773 7f4be25e0a80  0 monclient: wait_auth_rotating timed out after 30
-38> 2017-03-05 21:07:12.493795 7f4be25e0a80 -1 osd.0 6982 unable to obtain rotating service keys; retrying
-37> 2017-03-05 21:07:12.493864 7f4be25e0a80 10 monclient: shutdown

The osd wasn't able to authenticate with the monitor in 30 seconds. The heartbeatmap assert shouldn't be hit, but this isn't that common an error case.

Actions
Copy link
 #3

Updated by Greg Farnum almost 7 years ago

  • Project changed from Ceph to RADOS
  • Category set to Correctness/Safety
  • Component(RADOS) MonClient, OSD added
Actions
Copy link

Also available in: Atom PDF