Project

General

Profile

Actions
Copy link

Bug #17170

closed

mon/monclient: update "unable to obtain rotating service keys when osd init" to suggest clock sync

Added by huanwen ren over 7 years ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Greg Farnum
Category:
Security
Target version:
-
% Done:

100%

Source:
other
Tags:
backport_processed
Backport:
pacific,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v10.2.7
ceph-qa-suite:
Component(RADOS):
Pull request ID:
48318
Crash signature (v1):
Crash signature (v2):

Description

1.cluster info:
a> have 3 servers
b> have a mon each server

2.Constructing scenarios(only one times)
I changed the time of server, when I found wrong time

3.osd error(some osds, not all)
I restart osd after the modification time found the following error osd：

2016-08-26 10:13:14.556691 7fbac89de700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2016-08-26 09:13:14.556670)
2016-08-26 10:13:14.557430 7fbad09ee700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2016-08-26 09:13:14.557419)
2016-08-26 10:13:15.766918 7fbad8c2d700 10 journal commit_start max_applied_seq 55383725, open_ops 0
2016-08-26 10:13:15.766942 7fbad8c2d700 10 journal commit_start blocked, all open_ops have completed
2016-08-26 10:13:15.766945 7fbad8c2d700 10 journal commit_start nothing to do
2016-08-26 10:13:15.766949 7fbad8c2d700 10 journal commit_start
2016-08-26 10:13:20.767043 7fbad8c2d700 10 journal commit_start max_applied_seq 55383725, open_ops 0
2016-08-26 10:13:20.767085 7fbad8c2d700 10 journal commit_start blocked, all open_ops have completed
2016-08-26 10:13:20.767087 7fbad8c2d700 10 journal commit_start nothing to do
2016-08-26 10:13:20.767091 7fbad8c2d700 10 journal commit_start
2016-08-26 10:13:21.553471 7fbae466b880  0 monclient: wait_auth_rotating timed out after 30
2016-08-26 10:13:21.553499 7fbae466b880 -1 osd.2 9877 unable to obtain rotating service keys; retrying
2016-08-26 10:13:21.559833 7fbad8c2d700 10 journal commit_start max_applied_seq 55383725, open_ops 0
2016-08-26 10:13:21.559853 7fbad8c2d700 10 journal commit_start blocked, all open_ops have completed
2016-08-26 10:13:21.559865 7fbad8c2d700 10 journal commit_start nothing to do
2016-08-26 10:13:21.559869 7fbad8c2d700 10 journal commit_start
2016-08-26 10:13:21.560085 7fbae466b880 10 journal journal_stop
2016-08-26 10:13:21.560142 7fbae466b880  1 journal close /var/lib/ceph/osd/ceph-2/journal
2016-08-26 10:13:21.560156 7fbad842c700 20 journal write_thread_entry woke up
2016-08-26 10:13:21.560165 7fbad842c700 10 journal write_thread_entry finish
2016-08-26 10:13:21.560223 7fbae466b880 15 journal do_write writing 16200609792~0 + header
2016-08-26 10:13:21.560367 7fbae466b880 20 journal do_write latency 0.000139
2016-08-26 10:13:21.560372 7fbae466b880 20 journal do_write queueing finishers through seq 0
2016-08-26 10:13:21.560376 7fbae466b880 20 journal write_header_sync finish
2016-08-26 10:13:21.560391 7fbad7c2b700 10 journal write_finish_thread_entry exit
2016-08-26 10:13:21.624241 7fbae466b880 -1 ^[[0;31m ** ERROR: osd init failed: (110) Connection timed out^[[0m

Files

Download all files
ceph-osd.2_back.log (41.9 KB) ceph-osd.2_back.log huanwen ren, 08/30/2016 09:18 AM
osd.log.txt (329 KB) osd.log.txt tangwenjun tang, 03/29/2018 01:19 AM
rotating_service_keys.png (141 KB) rotating_service_keys.png liqun zhang, 03/25/2022 02:35 AM

Related issues 2 (0 open — 2 closed)

Copied to RADOS - Backport #58333: pacific: mon/monclient: update "unable to obtain rotating service keys when osd init" to suggest clock syncRejectedKonstantin ShalyginActions
Copied to RADOS - Backport #58334: quincy: mon/monclient: update "unable to obtain rotating service keys when osd init" to suggest clock syncResolvedRadoslaw ZarzynskiActions
Actions
Copy link

Also available in: Atom PDF